scriptoriumadmin/featurebase
1
0
Fork 0
mirror of https://github.com/featurebasedb/featurebase.git synced 2026-09-05 08:10:50 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
6501 commits 45 branches 174 tags 85 MiB
Commit graph

15 commits

Author SHA1 Message Date
Jason E. Aten
eae82b72c2 rbf: use a red-black tree to manage the root records list 
- 40% faster on ingest_test when putting 10K roots/containers.
 - 15% fewer bytes allocated total
 - clarifying renames leafCell.N -> ElemN, allocate -> allocatePgno,
   deallocate -> freePgno
 2020-11-13 16:49:26 +00:00
Ben Johnson
8de1959938 Remove RBF exclusive/direct write. 2020-11-09 08:23:09 -07:00
Jason E. Aten
957cba1768 performance tuning: rbfcfg package, binary search for wal segment 
- rbfcfg package holds Config for --rbf- command line flags
- wal.go: replace linear search with bisection for wal segment
 2020-10-27 00:49:56 +00:00
Jason E. Aten
8d2ad048fa pilosa server --norowcache disables the row cache 
- this can lessen memory pressure
- certain backends may not need it
- enables performance benchmarking and tuning
 2020-10-22 22:20:48 +00:00
Jason E. Aten
81013999e5 introduce a per shard blue-green RWMutex 
- allows blue-green testing with concurrent readers/writers.
 - otherwise we don't start/end the blue and green Tx
   together, and they get split by a read/write concurrently.
 2020-10-19 17:28:35 -05:00
J
3ad8ec7f0f rbf: add runtime options to DB struct 
- FsyncEnabled and DoAllocZero moved to DB struct.
 - deletes unused xrbrsupport.go and cmd/convert
 - fixes #941
 2020-10-12 18:35:00 -05:00
Jason Aten
55d4c29933 turn off debug machinery on tx backends 
- enable row cache again. Was off for tx perf measurement.
 - centralize UseRowCache choice to just one point, in rbf.EnableRowCache
 2020-09-22 15:47:52 -05:00
Jason Aten
748f6a61bc fine tune Tx placement, isolate Tx backends more. 
- Tx creation is lazier so we don't create xtra shards.

 - Then the dir scan for blue-green state checking finds only the right shards.
 2020-09-14 18:03:25 -05:00
Jason Aten
7028bcfc9d fix resource leaks in fragment_internal_test.go under roaring, better skipForRoaring func 
- add tournament.sh to do all pair-wise comparisons of blue-green backends.
- isolate txstores away from roaring index/ directories with indexname.index.txstores@@@ dirs.
 2020-09-13 22:45:39 -05:00
Jason Aten
2eb097c14d blue_green migration. holdbkg.go holder goroutine. 
- blue_green for doing migration. Called before Holder.Open finishes.
- holdbkg.go added for index lookup. Less wedging between a deadlock and a race.
- fix fault under read-only map under lmdb at
  TestExecutor_Execute_Row_Range/RowIDColumnID by doing cow in roaring.
- roaring -tags gofuzz builds again
- roaringparanoia build tag added to make test targets in Makefile
- add rbf.NewDBWithAllocZero for out-of-bounds memory checks
- .circleci/config.yml test-shardwidth-22 with large run container, kept OOM-ing we suspect.

Fixes #819
 2020-09-11 14:24:15 -05:00
Ben Johnson
a51c530763 Add direct import option; remove tx from fragment.Open() 2020-09-08 08:13:16 -06:00
Ben Johnson
150c8a5b06 database per shard, HolderConfig, rbf bit-wise import speedups. 
- introduce Query Context (Qcx) for managing database-per-shard.
- replaces the MultiTx, so mtx.go is retired and removed.
- introduces the HolderConfig struct and all Holders now have
  a path from birth.
- rbf speedups on bitwise writes
- badgerdb is removed due to unresolvable write conflicts.

fixes #703 #676
 2020-09-04 13:00:33 -05:00
Todd Gruben
edf6608129 copied optimized add/remove to rbf 
use large test container for CI
 2020-08-24 10:37:25 -05:00
Jason Aten
123ce41840 add lmdb, Tx call stats, and prep for db/shard. 
- lmdb as a backend (lmdb.go)
   (lmdb is the fastest known transactional storage backend)
 - per Tx call statics report enabled with PILOSA_CALLSTAT=true (stattx.go)
 - framework for per-shard db (dbshard.go)
 - txfactory handles any pair under blue-green testing (txfactory.go)
 - enable CGO in Dockerfiles for lmdb
 2020-08-17 18:26:58 -05:00
Todd Gruben
547ee14f5b all test green on rbf. WOOT. 
- rbf had races around the new rootRecords cache in tx
 - rbf tx needed a write lock on the db now that rootRecords are written
 - added a global registry for rbfDB to correctly dedup instances
 - implement DeleteFragment, DeleteIndex for rbf
 - use badger style keys for rbf to allow content checksumming to be list
    containers in the same order
 - lots of other integration of rbf into pilosa layer.
 2020-08-12 21:08:30 +00:00