The anti-entropy feature has never actually worked. We've been
talking about removing it or replacing it for ages, but haven't
had a concrete motivation.
But the anti-entropy interface is the sole user of several components
of the Tx interface, and now that we're trying to replace that
interface, being able to drop those components has some appeal, so
let's remove the one thing that used them, in the hopes that this
will simplify life.
This also lets us drop ForEach and ForEachRange, which were
barely used at all. The one surviving usage (CSV export) can be
handled by using the container iterator we already have, and
making ContainerCallback exported so we can use it to just call
things for every bit.
This is fairly experimental, but basically, we make a fragment-level
op which, given a []uint64, can produce a union of all those rows in
the fragment, with a single scan through the fragment and not needing
a ton of additional space to reify all the rows at once.
Now with the Repair calls happening in the Results assembly rather than
on the intermediate data.
There are 16 containers in a row in a shard with the default shard
width of 2^20, but since you can change the shard width at compile
time, everything should be computed off that.
This adds a shard-based import endpoint which takes bitmap data for
all field types and imports data for the whole shard transactionally.
It uses the BitmapRewriter interface to try to intelligently allow for
setting and clearing bits simultaneously without multiple writes which
is especially helpful when ingesting into int-like fields, but also
allows clear-and-then-set behavior for set fields.
The intent of these lines was "if there's no filter, return
immediately rather than doing operations".
But actually we didn't write that, so we were calling intersectionCallback
on empty filters, which didn't matter since it failed out quickly, but
it's still a waste of effort.
Except we shouldn't get to these anyway because ConsiderKey already
correctly rejected these cases. I think. But still.
In fact, we *do* want to skip ahead sometimes to the next thing, and
only call our callback for things that match our filter. I was thinking
that we needed to call the callback for all data regardless, because
what if you're writing to a mutex and adding new data.
But even if you're adding new data, it's still in the filter, because
it has to be, because we don't start out knowing there's no existing
data. So the mutex actually works fine.
So the rule for BitmapBitmapTrimmer is that your filter doesn't have
any meaning other than (1) it tells us which containers you need
to see, (2) we provide it to your callback function. Maybe you want
to subtract those. Maybe you want to add them. That's up to you to
decide.
This in a parallel to ApplyFilter/BitmapFilter which allows writebacks
while it's running. It's a write operation, so it needs a write lock
on the Tx, and needs to create bitmaps if they don't already exist.
The semantics are a bit messy and need better documentation still.
We don't really need to fully extract every row, we just need counts.
This naive approach uses logic similar to BitmapBitmapFilter, but tweaks
it so that we can intercept the existence and sign bit rows, work with
an optional filter, and yield a sum. We accumulate the statistics
internally, rather than using a callback, because I tried to make it
work with a callback and it was a complete mess.
Note the fancy check for container reuse in the BSI Count filter.
This is because intersection(full container, X) is just the original
X, *not* a copy, but in this case we need a copy because RBF
ApplyFilter will in fact reuse a single container's storage for
each consecutive container.
it was somewhat difficult to avoid ripping this out without also
touching some of the stuff that supports roaring backend. That's going
soon too, so no worries :)
i used this script, a little clunky but it got the job done
```bash
for file in `find . -type f -print | grep '\.go'`; do
sed '1,/^\/\/ limitations under the License.$/d' $file > $file.tmp;
result=`cat $file.tmp`
if [[ result != "" ]]; then
gofmt $file.tmp &> /dev/null;
if [[ $? == 0 ]]; then
mv $file.tmp $file && gofmt -w $file;
else
rm $file.tmp;
fi
else
rm $file.tmp;
fi
done
```
We support query parameters for details (default false) which
request additional data, and for a limit (default 0/MaxInt32)
on number of results returned to limit the amount of spam
produced if there's a lot of results. The simpler default
output should reduce load and runtime significantly, and the
ability to specify limits makes it easier to get reasonably
small responses.
There's some context support here, but the underlying filters
don't take contexts or check for them, which is probably
a flaw but might be a bit large to correct for this.
Despite being large, this set of changes is actually
fairly well contained within the mutex-checking code.
This implements a fairly straightforward sanity-check for mutexes,
implemented as a bitmapfilter at the fragment level, and with higher
levels combining results. There's two endpoints, an internal endpoint
which only checks the local node's shards, and an external one which
forwards requests (using the internal endpoint) to all the other nodes.
The internal endpoint does not do key translation, the external one
does.
The transmission format is a probably-inefficient JSON blob, and
returns data separated per-shard so we don't have as much merging
work to do.
This introduces a horrifying monstrosity function which tries to
sneakily corrupt mutex fields and which has to be exported (EWWWWW)
but which is only present in _test code (!??!! THIS WORKS WHY).
Also one typo fix in unrelated code caused by not wanting to keep
fighting with gofmt about this.
In BitmapBitmapFilter.ConsiderData, we intersect things solely in order
to perform callbacks on them. Creating these intermediate arrays is
actually somewhat expensive, and all we're going to do with them is
make callbacks anyway.
So, we add a new `intersectCallback`, which behaves similarly to
`intersectionCount`, but which dramatically reduces the amount of memory
allocation associated with doing the callbacks; in some test cases
on mutex data, this code was >90% of all memory allocations, and
getting rid of that helps a lot.
At that point, we no longer need the separate intersectAny check,
because it doesn't save us any time anymore.
unclearSets was completely broken and I have no idea why the test I thought
was testing it didn't actually catch that problem. Added unit tests and fixed
the logic. Improved/clarified prune and fullPrune, and unexported their
names because why export methods on an unexported type.
Also improve some comments and rename a variable or two to improve clarity.
This is a partial solution to a nasty performance problem, which is that
a ContainerIterator has to *generate* all the containers. With roaring, this
was cheap because they already exist in memory; with transactional backends,
it's an allocation per container, *even for the containers we don't use*.
This design admits filters which can distinguish between answers they
can give just based on keys and times when they actually need containers
instantiated, and can also give hints as to future answers -- saying "yes"
or "no" to entire rows at a time, or indicating when they're done.
This is only part of the solution; we also need a Tx API hook for
doing scans like this which doesn't rely on ContainerIterator.
