Commit graph

8 commits

Author SHA1 Message Date
Jason E. Aten
957cba1768 performance tuning: rbfcfg package, binary search for wal segment
- rbfcfg package holds Config for --rbf- command line flags
- wal.go: replace linear search with bisection for wal segment
2020-10-27 00:49:56 +00:00
Jason E. Aten
8ff6e8e0fd versioned readonly shards map 2020-10-23 23:50:22 +00:00
Jason E. Aten
2503fe5b66 pilosa: avoid re-scanning shards during Holder.Open()
- view.openFragmentInTx was forcing a directory scan
   for shards on every open fragment during Holder.Open().
   Seen by pprof profile having excessive allocations
   from dbshard.go listDirUnderDir().
2020-10-23 23:50:22 +00:00
Jason E. Aten
266b92c025 Use boltdb instead of badger as our all Go Tx oracle.
- remove all badgerdb code.
 - use boltdb instead.
2020-10-16 17:21:21 -05:00
Jason Aten
31d54010f8 blue_green verification and migration capabilities.
- on startup in blue_green mode, we will migrate
   blue to green if blue is empty.
 - otherwise, when blue has data, we verify
   against green before proceeding with the
   blue_green run.
 - small optimization in the rbf cursorx.go to
   short-circuit processing on a nil bitmap.
   This avoids a roaringparanoia tag panic.
 - back out holdbkg.go, was too slow.
   add a distinct Holder.imu lock instead.
2020-09-16 09:41:05 -05:00
Jason Aten
748f6a61bc fine tune Tx placement, isolate Tx backends more.
- Tx creation is lazier so we don't create xtra shards.

 - Then the dir scan for blue-green state checking finds only the right shards.
2020-09-14 18:03:25 -05:00
Jason Aten
2eb097c14d blue_green migration. holdbkg.go holder goroutine.
- blue_green for doing migration. Called before Holder.Open finishes.
- holdbkg.go added for index lookup. Less wedging between a deadlock and a race.
- fix fault under read-only map under lmdb at
  TestExecutor_Execute_Row_Range/RowIDColumnID by doing cow in roaring.
- roaring -tags gofuzz builds again
- roaringparanoia build tag added to make test targets in Makefile
- add rbf.NewDBWithAllocZero for out-of-bounds memory checks
- .circleci/config.yml test-shardwidth-22 with large run container, kept OOM-ing we suspect.

Fixes #819
2020-09-11 14:24:15 -05:00
Ben Johnson
150c8a5b06 database per shard, HolderConfig, rbf bit-wise import speedups.
- introduce Query Context (Qcx) for managing database-per-shard.
- replaces the MultiTx, so mtx.go is retired and removed.
- introduces the HolderConfig struct and all Holders now have
  a path from birth.
- rbf speedups on bitwise writes
- badgerdb is removed due to unresolvable write conflicts.

fixes #703 #676
2020-09-04 13:00:33 -05:00