Commit graph

4 commits

Author SHA1 Message Date
Seebs
54f5cc799c performance hackery: add intersectCallback for use in running callbacks
In BitmapBitmapFilter.ConsiderData, we intersect things solely in order
to perform callbacks on them. Creating these intermediate arrays is
actually somewhat expensive, and all we're going to do with them is
make callbacks anyway.

So, we add a new `intersectCallback`, which behaves similarly to
`intersectionCount`, but which dramatically reduces the amount of memory
allocation associated with doing the callbacks; in some test cases
on mutex data, this code was >90% of all memory allocations, and
getting rid of that helps a lot.

At that point, we no longer need the separate intersectAny check,
because it doesn't save us any time anymore.
2021-05-10 11:58:18 -05:00
Seebs
4ddbadcea7 review issues: fix unclearSets (now sliceDifference) and prune/fullPrune
unclearSets was completely broken and I have no idea why the test I thought
was testing it didn't actually catch that problem. Added unit tests and fixed
the logic. Improved/clarified prune and fullPrune, and unexported their
names because why export methods on an unexported type.

Also improve some comments and rename a variable or two to improve clarity.
2020-12-16 13:18:18 -06:00
Seebs
dec0a00155 add container N to ConsiderKey 2020-12-16 13:16:46 -06:00
Seebs
59d89dda99 Allow arbitrary and potentially more efficient filtering of bitmaps
This is a partial solution to a nasty performance problem, which is that
a ContainerIterator has to *generate* all the containers. With roaring, this
was cheap because they already exist in memory; with transactional backends,
it's an allocation per container, *even for the containers we don't use*.

This design admits filters which can distinguish between answers they
can give just based on keys and times when they actually need containers
instantiated, and can also give hints as to future answers -- saying "yes"
or "no" to entire rows at a time, or indicating when they're done.

This is only part of the solution; we also need a Tx API hook for
doing scans like this which doesn't rely on ContainerIterator.
2020-12-16 13:16:46 -06:00