- Previously, on timequantum schemas, we would
create and open a view for the cartesian
product of every possible view and shard.
- This caused us to be very slow on re-open,
and to use lots of memory for views that
held nothing.
- This change makes startup faster, memory
use much lower, and should speed migration.
- view.openFragmentInTx was forcing a directory scan
for shards on every open fragment during Holder.Open().
Seen by pprof profile having excessive allocations
from dbshard.go listDirUnderDir().
- fix a CI/Makefile issue that was hiding red tests in CI.
- the testv and testv-race targets now require /bin/bash
- In executor.go, the top-level query context Qcx now
has a write flag. It will upgrade read-Tx to write-Tx
when Store() wraps some inner local-read operations,
to avoid deadlocking against its own query. This deadlock
happens in TestExecutor_Execute_SetRow/Set_NewRow
under rbf_lmdb blue-green testing without the upgrade.
- correct string constants for txtype so that
blue-green cleanup correctly detects when
2nd transaction in a pair has Committed and
thus the blue-green RWMutex can be relased
- test that txtype.String() is consistent with
the corresponding string constants.
- document in bluegreentx.go the current limitations
of blue-green testing: only one github archive import
(a single writing client) is supported by blue-green
testing. Multiple importers will deadlock eventually
on the DBShard.mut RWMutex. We could fix this by
ordering the write locks and obtaining them in
strictly increasing order (by shard number), but
that would require alot of change to the executor
and that would introduce more risk for a test-only
pathway.
- allows blue-green testing with concurrent readers/writers.
- otherwise we don't start/end the blue and green Tx
together, and they get split by a read/write concurrently.
- on startup in blue_green mode, we will migrate
blue to green if blue is empty.
- otherwise, when blue has data, we verify
against green before proceeding with the
blue_green run.
- small optimization in the rbf cursorx.go to
short-circuit processing on a nil bitmap.
This avoids a roaringparanoia tag panic.
- back out holdbkg.go, was too slow.
add a distinct Holder.imu lock instead.
- add tournament.sh to do all pair-wise comparisons of blue-green backends.
- isolate txstores away from roaring index/ directories with indexname.index.txstores@@@ dirs.
- blue_green for doing migration. Called before Holder.Open finishes.
- holdbkg.go added for index lookup. Less wedging between a deadlock and a race.
- fix fault under read-only map under lmdb at
TestExecutor_Execute_Row_Range/RowIDColumnID by doing cow in roaring.
- roaring -tags gofuzz builds again
- roaringparanoia build tag added to make test targets in Makefile
- add rbf.NewDBWithAllocZero for out-of-bounds memory checks
- .circleci/config.yml test-shardwidth-22 with large run container, kept OOM-ing we suspect.
Fixes#819
- introduce Query Context (Qcx) for managing database-per-shard.
- replaces the MultiTx, so mtx.go is retired and removed.
- introduces the HolderConfig struct and all Holders now have
a path from birth.
- rbf speedups on bitwise writes
- badgerdb is removed due to unresolvable write conflicts.
fixes#703#676
- lmdb as a backend (lmdb.go)
(lmdb is the fastest known transactional storage backend)
- per Tx call statics report enabled with PILOSA_CALLSTAT=true (stattx.go)
- framework for per-shard db (dbshard.go)
- txfactory handles any pair under blue-green testing (txfactory.go)
- enable CGO in Dockerfiles for lmdb
