- the sync.Pool default uses little memory under CI.
- arena approach provides ability to control the maximum memory
used by rbf Cursors.
- cursor caching is adjustable with --rbf-cursor-cache
currently 0 by default (meaning use sync.Pool), and
larger than 0 meaning use an arena of this size.
With the arena, 20 or less is needed to pass CI.
- rbf test suite runs ~ 4x faster
- kitchen sink ingest test runs 16% faster.
- report TotalAlloc in CALLSTATs
fixes#1105
This commit fixes a bug where the root record cache was being
updated in-place causing a race condition with other transactions
using it. The cache implementation has been changed from `rbtree`
to an `immutable.SortedMap`.
- fix a bug in computing leafCell.BitN in a run after a bit Remove
- shrink bitmaps on remove
- util_test.go has Cursor.DebugSlowCheckAllPages to verify;
used by cursor_test.go
- default Tx is once again RBF, changed from bolt.
- document the RBF code review comments that were not addressed
before #1052 was merged, so they don't get lost.
- they should be easily addressed by replaying the entire WAL file
rather than from the DB meta page 0 notion of the last WalID
- cleanup rbf/cfg/cfg.go stale comments, ensure default0 respected.
1 msec checkpoint time, 1MB wal segment defaults.
- return a specific error, ErrNoMetaFound, from findNextWALMetaPage()
rather than io.EOF, since there actually wasn't any file IO involved.
- add http handlers for /cpu-profile/start and /cpu-profile/stop
in http/handler.go enable CPU profiling at specific time points
during an ingest or other operation.
This commit changes the checkpointing to determine a minimum WAL ID
for readers and a max ID based on the writer. Pages are checkpointed
from the WAL up to the writer's max WAL ID but segments are removed
only up to the reader's minimum WAL ID. This ensures that WAL pages
are not removed out from under current read transactions.
- we return to checkpointing after every commit, by default.
- the internal rbf logic is not ready to have
checkpoints deferred. Doing so results in
references to WAL segments that are not
in the current slice of live segments.
I have a newer staticcheck and golangci-lint on my laptop, and it started
complaining about something. The first comment added disables the check
in staticcheck-as-a-command, the second disables it when it's being done
by golangci-lint, which invokes the analysis passes directly and displays
the output differently, and also doesn't recognize the hints used by
staticcheck.
Newer golangci-lint doesn't find anything else that it wants to complain
about.
In rare cases, RBF can produce containers which have a recorded N value which
is incorrect. This rarely affects anything, but on some particular queries,
this can result in very strange outcomes, like array containers with more
than 1<<16 entries.
To fix this, we have toContainer specify that it doesn't know the correct
N for the bitmap containers it's creating, which costs extra time for counting,
and should be considered a temporary workaround.
Also, we add a CheckN() function which is controlled by the
roaringparanoia flag, and add a number of calls to it, for instance, as
deferred calls after every container operation when roaringparanoia is
enabled. This means that we get improved confidence that we've caught
the relevant errors, but is not suitable for production use.
Previously, RBF shared a list of WAL segments between the DB & Tx.
However, this increased the need for mutexes to access the data.
WAL segments are effectively immutable on-disk so the list of segments
has been refactored so that changes to the segment list are done via
copy-on-write which allows read transactions to access segment data
without a mutex.
The database checkpointing can remove early, unused segments and
there is an update/add check to make sure that Tx segments pushed back
to the DB do not include removed segments.
