This commit fixes an issue where direct writes would overwrite the
source page where data was being copied from because writes are
immediate (instead of going to the WAL first).
- blue_green for doing migration. Called before Holder.Open finishes.
- holdbkg.go added for index lookup. Less wedging between a deadlock and a race.
- fix fault under read-only map under lmdb at
TestExecutor_Execute_Row_Range/RowIDColumnID by doing cow in roaring.
- roaring -tags gofuzz builds again
- roaringparanoia build tag added to make test targets in Makefile
- add rbf.NewDBWithAllocZero for out-of-bounds memory checks
- .circleci/config.yml test-shardwidth-22 with large run container, kept OOM-ing we suspect.
Fixes#819
blake3 code is used in several places on the code. The file was
duplicated on root and rbf package.
To avoid cyclic dependencies, I moved it to hash package. Some methods
must be public to use them in different places.
HashOfDir method was removed. Not used.
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
This commit adds the ability to start a transaction with an exclusive
lock for the entire database. This ensures no other read or write
transactions can run at the same time. Writes in this mode write
directly to the database and skip the WAL entirely.
- introduce Query Context (Qcx) for managing database-per-shard.
- replaces the MultiTx, so mtx.go is retired and removed.
- introduces the HolderConfig struct and all Holders now have
a path from birth.
- rbf speedups on bitwise writes
- badgerdb is removed due to unresolvable write conflicts.
fixes#703#676
- rbf had races around the new rootRecords cache in tx
- rbf tx needed a write lock on the db now that rootRecords are written
- added a global registry for rbfDB to correctly dedup instances
- implement DeleteFragment, DeleteIndex for rbf
- use badger style keys for rbf to allow content checksumming to be list
containers in the same order
- lots of other integration of rbf into pilosa layer.
Previously, the `checkpoint()` function determined the segments to drop
based on the current active transactions' WAL ID references. However, if
no transactions are active then the checkpoint would drop segments too
aggressively.
This changes the determination by using the highest WAL ID that is
actually checkpointed to disk to determine the high water mark. If no
page are checkpointed then no segments can be dropped.
green:
TestFragment_RowsIteration/combinations
TestFragment_RoaringImportTopN
red: (needs Ben's attention)
PILOSA_TXSRC=rbf go test -v -run TestFragment_TopN_IDs -tags=' shardwidth20' "-gcflags=all=-d=checkptr=0"
also red: (one for Ben)
TestCursor_FirstNext_Quick/9 is throwing
panic: cannot find segment containing WAL page: 1
as we check the error back from checkpoint() in Rollback().
