Commit graph

9 commits

Author SHA1 Message Date
Travis
08fae2be4c
introduce storage.Config 2021-01-20 22:05:38 -06:00
Jason E. Aten
9c7bc603af rbf: reuse cursors with sync.Pool or arena
- the sync.Pool default uses little memory under CI.

- arena approach provides ability to control the maximum memory
  used by rbf Cursors.

- cursor caching is adjustable with --rbf-cursor-cache
  currently 0 by default (meaning use sync.Pool), and
  larger than 0 meaning use an arena of this size.
  With the arena, 20 or less is needed to pass CI.

- rbf test suite runs ~ 4x faster

- kitchen sink ingest test runs 16% faster.

- report TotalAlloc in CALLSTATs

fixes #1105
2020-11-19 17:02:49 +00:00
Ben Johnson
81a64c5902 Add RBF halting; remove time based checkpoint 2020-11-11 11:17:07 -07:00
Ben Johnson
3554048877 Refactor RBF to use a single WAL file 2020-11-09 08:03:22 -07:00
Jason E. Aten
a38d4fa46c apply --fsync flag to all tx backend
- rename from --rbf-fsync to --fsync, as it
   now effects bolt, lmdb too.
2020-11-06 18:11:02 +00:00
Jason E. Aten
458095a707 rbf default. Add TODO comments, slurp -profile returns a cpu profile
- default Tx is once again RBF, changed from bolt.
- document the RBF code review comments that were not addressed
  before #1052 was merged, so they don't get lost.
- they should be easily addressed by replaying the entire WAL file
  rather than from the DB meta page 0 notion of the last WalID
- cleanup rbf/cfg/cfg.go stale comments, ensure default0 respected.
  1 msec checkpoint time, 1MB wal segment defaults.
- return a specific error, ErrNoMetaFound, from findNextWALMetaPage()
  rather than io.EOF, since there actually wasn't any file IO involved.
- add http handlers for /cpu-profile/start and /cpu-profile/stop
  in http/handler.go enable CPU profiling at specific time points
  during an ingest or other operation.
2020-11-04 01:36:17 +00:00
Ben Johnson
db74f0dffa Increase RBF WAL write cache size to 1MB.
Previously we dropped the cache size to 64KB but that seems much
too low. Write performance suffers considerably.
2020-10-30 17:09:33 -06:00
Jason E. Aten
1ca8a45357 pilosa server --rbf-checkpoint-dur to 0 by default
- we return to checkpointing after every commit, by default.
  - the internal rbf logic is not ready to have
    checkpoints deferred. Doing so results in
    references to WAL segments that are not
    in the current slice of live segments.
2020-10-27 22:38:18 +00:00
Jason E. Aten
957cba1768 performance tuning: rbfcfg package, binary search for wal segment
- rbfcfg package holds Config for --rbf- command line flags
- wal.go: replace linear search with bisection for wal segment
2020-10-27 00:49:56 +00:00