Commit graph

15 commits

Author SHA1 Message Date
Jason E. Aten
458095a707 rbf default. Add TODO comments, slurp -profile returns a cpu profile
- default Tx is once again RBF, changed from bolt.
- document the RBF code review comments that were not addressed
  before #1052 was merged, so they don't get lost.
- they should be easily addressed by replaying the entire WAL file
  rather than from the DB meta page 0 notion of the last WalID
- cleanup rbf/cfg/cfg.go stale comments, ensure default0 respected.
  1 msec checkpoint time, 1MB wal segment defaults.
- return a specific error, ErrNoMetaFound, from findNextWALMetaPage()
  rather than io.EOF, since there actually wasn't any file IO involved.
- add http handlers for /cpu-profile/start and /cpu-profile/stop
  in http/handler.go enable CPU profiling at specific time points
  during an ingest or other operation.
2020-11-04 01:36:17 +00:00
Jason E. Aten
1ca8a45357 pilosa server --rbf-checkpoint-dur to 0 by default
- we return to checkpointing after every commit, by default.
  - the internal rbf logic is not ready to have
    checkpoints deferred. Doing so results in
    references to WAL segments that are not
    in the current slice of live segments.
2020-10-27 22:38:18 +00:00
Jason E. Aten
957cba1768 performance tuning: rbfcfg package, binary search for wal segment
- rbfcfg package holds Config for --rbf- command line flags
- wal.go: replace linear search with bisection for wal segment
2020-10-27 00:49:56 +00:00
J
3ad8ec7f0f rbf: add runtime options to DB struct
- FsyncEnabled and DoAllocZero moved to DB struct.
 - deletes unused xrbrsupport.go and cmd/convert
 - fixes #941
2020-10-12 18:35:00 -05:00
Ben Johnson
c51ba69c09 Fix RBF sync calls 2020-10-06 09:36:31 -06:00
Ben Johnson
d826cec4b3 fix RBF WAL segment reference error 2020-10-06 09:36:31 -06:00
Ben Johnson
cdd10a26f0 Refactor RBF to use immutable list of WAL segments.
Previously, RBF shared a list of WAL segments between the DB & Tx.
However, this increased the need for mutexes to access the data.

WAL segments are effectively immutable on-disk so the list of segments
has been refactored so that changes to the segment list are done via
copy-on-write which allows read transactions to access segment data
without a mutex.

The database checkpointing can remove early, unused segments and
there is an update/add check to make sure that Tx segments pushed back
to the DB do not include removed segments.
2020-10-06 09:36:31 -06:00
Ben Johnson
f2bde49d25 Fix RBF WAL replay/truncation 2020-09-07 09:39:26 -06:00
Ben Johnson
159d01b55d Add exclusive write option for RBF.
This commit adds the ability to start a transaction with an exclusive
lock for the entire database. This ensures no other read or write
transactions can run at the same time. Writes in this mode write
directly to the database and skip the WAL entirely.
2020-09-04 12:46:42 -06:00
Ben Johnson
627b50d89d misc import optimizations 2020-08-31 08:46:53 -06:00
Ben Johnson
9dbb82cf3e WAL mutex fixes 2020-08-19 08:42:53 -06:00
Ben Johnson
51504f4fe4 Add WAL write cache mutex; update name; add benchmarks 2020-08-19 08:33:19 -06:00
Ben Johnson
5849794a1b Add RBF WAL write cache 2020-08-19 08:33:19 -06:00
Ben Johnson
f8cacd8081 Multiple RBF test fixes 2020-08-05 08:20:34 -06:00
Ben Johnson
de51b538f3 rbf: roaring bitmap format
Co-authored-by: Todd Gruben <todd@pilosa.com>
2020-07-08 13:27:18 -06:00