This adds a shard-based import endpoint which takes bitmap data for
all field types and imports data for the whole shard transactionally.
It uses the BitmapRewriter interface to try to intelligently allow for
setting and clearing bits simultaneously without multiple writes which
is especially helpful when ingesting into int-like fields, but also
allows clear-and-then-set behavior for set fields.
We've been seeing weird retention of Tx that shouldn't still be open, and
one possible explanation is that, until a Tx actually uses the freelist
cursor (either to allocate a page or to release it back to the freelist),
the freelistCursor statically stored in the Db object continues to have a
pointer to the previous Tx which used it, which allows a Tx, and thus its
dirty page map, to be retained forever.
I previously thought this should also nil out the page maps in the Tx, but
the more I think about it, the less I think that's a good idea. The actual
lifespan of a committed Tx should be quite short. If it *does* stick around,
it's beneficial to us as debuggers to see those large maps of dirty pages
sticking around. So after thinking about it a lot I decided not to do
that.
Similarly, when closing out a container filter (whether a filter or
a rewriter), zero out the Cursor, Tx, and filter and rewriter functions.
(We don't have to worry about the cursor's Tx, because the cursor gets
closed, which zeros its Tx and returns the cursor to the cursor pool,
too.) This likely matters a lot less, as the filters in the pool
get garbage collected "soon", but it still reduces the amount of
stuff being retained.
We have code to correctly fill in cell.BitN when a leaf cell already
exists but isn't of the correct sort, but not to handle the case where
it already exists and *is* a BitmapPtr, but doesn't necessarily have
the right BitN value.
Several changes. One is, we don't provide a `New` for pagePool, which
allows allocPage to check whether a page was returned, and thus, zero
pages which were found in the pool, or make new pages, but never zero
pages it just created with make. We then also make many more things
which were making pages use the pool.
Reuse the same page allocation for multiple header pages dumped into
the WAL; the bitmap header pages aren't stashed in our page map,
they're only written to the disk, so we don't need to make a new page
each time, we can just make one new page for the whole batch.
Internally in the pool, we pool pointers to [PageSize]byte, rather
than slices. sync.Pool needs pointer-like things. To store a pointer
to a slice, you have to heap-allocate the slice, also. So, instead
of heap-allocating copies of these slices, we just use pointers to
the raw data.
This commit fixes a bug in RBF where deleting all the elements in
a bitmap that has a depth greater than 2 will cause the root bitmap
to be a branch page with a cell count of zero. This breaks an
assertion in `readBranchCell()` which causes a panic post-commit.
A new assertion has been added to prevent a branch page from being
written with a zero count in the future.
i used this script, a little clunky but it got the job done
```bash
for file in `find . -type f -print | grep '\.go'`; do
sed '1,/^\/\/ limitations under the License.$/d' $file > $file.tmp;
result=`cat $file.tmp`
if [[ result != "" ]]; then
gofmt $file.tmp &> /dev/null;
if [[ $? == 0 ]]; then
mv $file.tmp $file && gofmt -w $file;
else
rm $file.tmp;
fi
else
rm $file.tmp;
fi
done
```
- the sync.Pool default uses little memory under CI.
- arena approach provides ability to control the maximum memory
used by rbf Cursors.
- cursor caching is adjustable with --rbf-cursor-cache
currently 0 by default (meaning use sync.Pool), and
larger than 0 meaning use an arena of this size.
With the arena, 20 or less is needed to pass CI.
- rbf test suite runs ~ 4x faster
- kitchen sink ingest test runs 16% faster.
- report TotalAlloc in CALLSTATs
fixes#1105
- fix a bug in computing leafCell.BitN in a run after a bit Remove
- shrink bitmaps on remove
- util_test.go has Cursor.DebugSlowCheckAllPages to verify;
used by cursor_test.go
In rare cases, RBF can produce containers which have a recorded N value which
is incorrect. This rarely affects anything, but on some particular queries,
this can result in very strange outcomes, like array containers with more
than 1<<16 entries.
To fix this, we have toContainer specify that it doesn't know the correct
N for the bitmap containers it's creating, which costs extra time for counting,
and should be considered a temporary workaround.
Also, we add a CheckN() function which is controlled by the
roaringparanoia flag, and add a number of calls to it, for instance, as
deferred calls after every container operation when roaringparanoia is
enabled. This means that we get improved confidence that we've caught
the relevant errors, but is not suitable for production use.
This commit fixes an issue where direct writes would overwrite the
source page where data was being copied from because writes are
immediate (instead of going to the WAL first).
- rbf had races around the new rootRecords cache in tx
- rbf tx needed a write lock on the db now that rootRecords are written
- added a global registry for rbfDB to correctly dedup instances
- implement DeleteFragment, DeleteIndex for rbf
- use badger style keys for rbf to allow content checksumming to be list
containers in the same order
- lots of other integration of rbf into pilosa layer.
