Commit graph

37 commits

Author SHA1 Message Date
Seebs
dc67149326 read leaf cells through a pointer
This reduces noticably the cost of reading leaf cells, by passing
a single pointer down the stack instead of the entire data structure
up the stack. It's only a few percent overall, but it's noticeable.
2020-12-16 13:16:46 -06:00
Jason E. Aten
82d07bc123 debugstats and rbf tooling for enhanced debugging/diagnostics 2020-12-08 22:47:40 +00:00
Ben Johnson
79e6156003 Add sync.Pool for RBF pages 2020-12-03 07:32:36 -07:00
Ben Johnson
df958f40dd Optimize putLeafCellFast() 2020-12-01 12:12:54 -07:00
Jason E. Aten
e1bb6e303a rbf: use an inlined immutable.Map<uint32, int64> for the PageMap
- goes 7% faster on kitchen sink import test

- reduces total allocations by 5% on same test.
2020-11-20 00:49:41 +00:00
Ben Johnson
a9831ad5a1 Replace literals with leafCellHeaderSize 2020-11-19 14:18:53 -07:00
Ben Johnson
33334511c4
Merge branch 'master' into fast-write 2020-11-19 13:44:09 -07:00
Ben Johnson
a31487b873 Fix rbf write root record iterator reset 2020-11-19 11:15:24 -07:00
Ben Johnson
0175c66756 Use immutable.SortedMap for root records
This commit fixes a bug where the root record cache was being
updated in-place causing a race condition with other transactions
using it. The cache implementation has been changed from `rbtree`
to an `immutable.SortedMap`.
2020-11-18 11:22:52 -07:00
Ben Johnson
519dc5069d Implement optimized fast leaf write.
This commit adds an optimized implementation for `putLeafCell()` if
the insert/update will not cause the page to overflow.
2020-11-16 08:13:35 -07:00
Jason E. Aten
4120024b10 rbf: BitN needs int32 to hold its maximum value.
- decrease ArrayMaxSize and RLEMaxSize by 1
  to make space
- add TestForEachRange, fix bugs found in ForEach,
  where the call f() logic was backwards.
2020-11-13 22:59:18 +00:00
Jason E. Aten
d46b603cd6 rbf: keep ElemN and BitN up to date.
- fix a bug in computing leafCell.BitN in a run after a bit Remove
- shrink bitmaps on remove
- util_test.go has Cursor.DebugSlowCheckAllPages to verify;
  used by cursor_test.go
2020-11-13 21:18:09 +00:00
Jason E. Aten
eae82b72c2 rbf: use a red-black tree to manage the root records list
- 40% faster on ingest_test when putting 10K roots/containers.
 - 15% fewer bytes allocated total
 - clarifying renames leafCell.N -> ElemN, allocate -> allocatePgno,
   deallocate -> freePgno
2020-11-13 16:49:26 +00:00
Ben Johnson
9eba299d35 Restrict max RBF transaction size 2020-11-10 08:14:14 -07:00
Ben Johnson
78eb9e0711 Fix linter 2020-11-10 07:07:06 -07:00
Ben Johnson
3554048877 Refactor RBF to use a single WAL file 2020-11-09 08:03:22 -07:00
jaten-molecula
4543237e09
Merge branch 'master' into rbf-cli 2020-10-27 10:09:24 -05:00
Ben Johnson
7fa1a5edd2 Add RBF CLI commands 2020-10-27 08:12:22 -06:00
Jason E. Aten
957cba1768 performance tuning: rbfcfg package, binary search for wal segment
- rbfcfg package holds Config for --rbf- command line flags
- wal.go: replace linear search with bisection for wal segment
2020-10-27 00:49:56 +00:00
J
3ad8ec7f0f rbf: add runtime options to DB struct
- FsyncEnabled and DoAllocZero moved to DB struct.
 - deletes unused xrbrsupport.go and cmd/convert
 - fixes #941
2020-10-12 18:35:00 -05:00
Ben Johnson
265452cf3d Add rbf.SyncEnabled 2020-10-06 09:36:31 -06:00
Ben Johnson
c51ba69c09 Fix RBF sync calls 2020-10-06 09:36:31 -06:00
Ben Johnson
627b50d89d misc import optimizations 2020-08-31 08:46:53 -06:00
Todd Gruben
547ee14f5b all test green on rbf. WOOT.
- rbf had races around the new rootRecords cache in tx
 - rbf tx needed a write lock on the db now that rootRecords are written
 - added a global registry for rbfDB to correctly dedup instances
 - implement DeleteFragment, DeleteIndex for rbf
 - use badger style keys for rbf to allow content checksumming to be list
    containers in the same order
 - lots of other integration of rbf into pilosa layer.
2020-08-12 21:08:30 +00:00
Ben Johnson
f8cacd8081 Multiple RBF test fixes 2020-08-05 08:20:34 -06:00
Jason Aten
a3d802f8a3 rbf: OffsetRange, ImportRoaringBits, CountRange work
green:
TestFragment_RowsIteration/combinations
TestFragment_RoaringImportTopN

red: (needs Ben's attention)
PILOSA_TXSRC=rbf go test -v -run TestFragment_TopN_IDs  -tags=' shardwidth20'  "-gcflags=all=-d=checkptr=0"

also red: (one for Ben)
TestCursor_FirstNext_Quick/9 is throwing
  panic: cannot find segment containing WAL page: 1
  as we check the error back from checkpoint() in Rollback().
2020-07-30 20:20:38 -04:00
Ben Johnson
64de208170 Implement pilosa.Tx for RBF 2020-07-29 11:25:41 -06:00
Todd Gruben
7c5c693fcb linter fix 2020-07-17 09:03:32 -05:00
Todd Gruben
59d2d89a5c removed leafArg and added conatinertypebitmapptr; lint fixes 2020-07-17 02:16:35 -05:00
Todd Gruben
bbd83e4618 . 2020-07-16 08:31:09 -05:00
Todd Gruben
e68f0a0d8f missing license 2020-07-14 09:00:09 -05:00
Todd Gruben
b61bdbd7a0 fix linter errors 2020-07-14 08:54:30 -05:00
Todd Gruben
374a4ec9ce fixed missing refactor test;refactor GetBitmap 2020-07-14 07:57:02 -05:00
Todd Gruben
d6986b78a5 . 2020-07-13 20:08:22 -05:00
Todd Gruben
b2b614a686 refactor cursor.GetBitmap->tx.GetBitmap;rename to WalkRootRecordPages;err check 2020-07-13 19:32:26 -05:00
Todd Gruben
b11b43af43 refactor putleaf 2020-07-13 10:48:25 -05:00
Ben Johnson
de51b538f3 rbf: roaring bitmap format
Co-authored-by: Todd Gruben <todd@pilosa.com>
2020-07-08 13:27:18 -06:00