This works around an issue where unreplicated keys will not be matched everywhere.
This also avoids the cost of creating millions of bolt read transactions and allocating strings.
Since we don't always have "snapshots" anymore, the arguable benefit of
avoiding the snapshot is reduced, and the primary expense of
importPositions has been dramatically reduced as well, so let's
just use that all the time, and simplify life.
We also want to make it faster. We don't know how many bits there
are to set or clear in the input set, but we do know exactly how
many bits there are to set AND clear. We can subdivide these into
batches by rows, then process each batch by storing sets at the
bottom and clears at the top. We can also do batches by columns,
reducing the memory overhead of unpacking all the bits at once.
(For extra credit, we could alternate set/clear settings, and
thus do batches of "the clears from row 0, followed by the clears
from row 1" and "the sets from row 1, followed by the sets from
row 2", and so on, but this is too fancy.)
Every caller of importValue is in fact already providing values
with column IDs sorted. As such, we don't need a map for checking
the previously-set columns; we just need to check against the
previous value.
Attributes are unmaintained and unused.
They have become more of a liability than a benefit.
This change eliminates them from the codebase.
The only user-visible change (assuming that attrs are not used) is that the attrs field will no longer appear in row JSON.
When writing things that cause additions to the cache, mark it dirty and
flag it for recomputing, but only sometimes actually do the recalculation,
currently implying a 10-second window. We still mark the cache dirty,
so if a request comes in, we'll get fresh data, but the query will be
slowed down because the recomputation will happen then. But that's better
than doing thousands of recalculations which are never used...
The "batched" flag creates a complexity which is that the return value of Add
might or might not be meaningful, but it doesn't really buy us very much.
If we are concerned about the ops log size of writing single ops as 21-byte
arrays of 1 op rather than as 13-byte ops, we can make the AddN code smarter
about how it writes ops. And probably should.
Along with this, change Remove to use the batched operation form, which
writes a more meaningful ops log, and return a meaningful value for changes
made. Otherwise, it ends up writing potentially thousands of ops to the
ops log without reporting any OpN, because the number of ops written isn't
the same as the number of changes those ops made. This could result in
files growing by megabytes without OpN changing.
There was a comment here about a test failing with RemoveN. I can't prove
it, but I strongly suspect that this was actually a result of that test
case hitting a particular bug that we eventually fixed, and which we might
have fixed sooner if we'd realized why using RemoveN made that test
fail.
When *fragment.openStorage is invoked in both f.importValue and
f.importValueSmallWrite and it returns an error, this means there's
some underlying error with the storage device and at the point of this
commit, the sane thing to do is to close the process, otherwise the
operation of Pilosa might proceed in an inconsistent state thus
precipiatting other silent but hairy errors along the way such as
dereferencing *fragment.gen later on which is set to nil once
openStorage fails.
unclearSets was completely broken and I have no idea why the test I thought
was testing it didn't actually catch that problem. Added unit tests and fixed
the logic. Improved/clarified prune and fullPrune, and unexported their
names because why export methods on an unexported type.
Also improve some comments and rename a variable or two to improve clarity.
In some cases, ApplyFilter can be significantly faster. On the other hand, it doesn't
matter as much as you might think on the mutex imports, because we've already sucked
most of the time out of those.
Added additional mutex sample data and batches of it so we can
confirm that overwrite works. It didn't work, so that needed to be fixed.
Couple of things:
(1) Wasn't updating "last value seen" so the check for an unsorted list
didn't work.
(2) Also didn't handle the case where there were to-clear values higher
than any to-set value.
This could result in bits not getting cleared, which could result in
there being more than N bits to clear for N new bits. And that could cause
really strange problems when the input slices were parts of a single
larger slice, because bit positions to clear could get shoved in as
possible columns in a future batch.
This is a partial solution to a nasty performance problem, which is that
a ContainerIterator has to *generate* all the containers. With roaring, this
was cheap because they already exist in memory; with transactional backends,
it's an allocation per container, *even for the containers we don't use*.
This design admits filters which can distinguish between answers they
can give just based on keys and times when they actually need containers
instantiated, and can also give hints as to future answers -- saying "yes"
or "no" to entire rows at a time, or indicating when they're done.
This is only part of the solution; we also need a Tx API hook for
doing scans like this which doesn't rely on ContainerIterator.
This replaces the former TopK BSI building algorithm, as the row cache was too expensive.
Additionally, BSI addition has been optimized with specialized adders inside of roaring.
- only allocate the rowcache if it is in use (avoid allocation per fragment)
- when the rowcache is use, fragment.go intRowIterator must write lock the
fragment because the f.rowCache will be updated.
- eliminate unused bitmapCache interface to keep the linter happy.
- fixes#1035
The logic assumes that the lack of a corresponding rowSegment means that
there's no changes, but that's not true -- we just deleted all the
existing data! Update to match clearRow behavior better.
Also, add a corresponding test case for this.
Also, change references to 'defaultSnapshotQueue' to use
[fragment].holder.SnapshotQueue, because defaultSnapshotQueue was
the queueless queue, but holders were getting a snapshot queue,
meaning that "awaiting" a snapshot could result in moving on
and closing the holder before the actual snapshot queue finished
snapshotting.
- log Debugf when we repair a fragment block
- better run-run roaring testing for over-sized containers
- add which fragment path to panic on container too big
- include container contents in roaring hash for pilosa-chk/pilosa-check-backup
- blue_green for doing migration. Called before Holder.Open finishes.
- holdbkg.go added for index lookup. Less wedging between a deadlock and a race.
- fix fault under read-only map under lmdb at
TestExecutor_Execute_Row_Range/RowIDColumnID by doing cow in roaring.
- roaring -tags gofuzz builds again
- roaringparanoia build tag added to make test targets in Makefile
- add rbf.NewDBWithAllocZero for out-of-bounds memory checks
- .circleci/config.yml test-shardwidth-22 with large run container, kept OOM-ing we suspect.
Fixes#819
