This is a unification of a number of bug fixes, feature additions,
and so on. Features include:
* Dropping the "New" from NewWrite/NewRead.
* IndexName->keys.Index, etc.
* Add a new "Flush" operation which is necessary to get the
intended behavior of Delete, which allows us to commit/flush
changes without letting go of a write lock.
* Some additional wrapping and locking in rbfTxWrappers to
support that. rbfQueryRead/Write now forward their calls
to the parent rbfTxWrappers, so it can lock around the
reference to its underlying tx, so the flush operation can
replace that tx safely.
* AddIndexShards now treats no shards as "all shards", to
simplify call sites.
* Added parameters to NewRBFTxStore to let it interact with
executor's logger and worker pool.
* Internally, support explicit closes of parts of the database
which can also check for errors and fail if it's in use.
* Add ability to request a map of fields and views in use
for a given index/shard pair. This is probably deprecated
but we need it for the way backup/restore work.
* Add ability to request a complete map of the database showing
which shards exist for which index/field/view tuples. This is
backwards from how we store things on disk, but we need it
to allow creating the right in-memory data structures on
database open.
* Add "Backend()" method to let us distinguish backends in case
we some day have them again.
* Support deleting indexes, fields, or fragments.
* Support Backup (returning a ReadCloser that dumps the RBF
file, implicitly merging any current WAL) and Restore (create
a new RBF file).
* Change directory structure and fragment keys to match existing
databases, so we should in theory be able to open an existing
data directory.
* Fragment delete probably doesn't lock correctly and this
should be reviewed.
* Export the DOT-format Dump so we can hook it up to a debug
endpoint. This wants to be explored more; ideally the front-end
UI should be able to display this.
* Create a NopTxStore which can be used like a TxStore but everything
that can error errors out. This is then used to let a holder that
hasn't had a txstore initialized work anyway.
There's at least a couple of open issues that need to be revisited
here.
At 50ms, we see sporadic failures in CI. So much for "this should
only need a couple milliseconds". Bumped timeout to avoid that.
The challenge here is that we have some tests which *want* to hit
the timeout to confirm that we aren't allowing things we shouldn't.
But we don't want the test to hang forever. But we want to be sure
it is actually stuck and not just being slow...
This provides us with most of the existing Tx interface, split
across QueryRead and QueryWrite. The functions not included here
are the ones that are used *only* for anti-entropy (ForEach
and ForEachRange).
We add additional testing to verify that TxStores are getting
closed correctly, to go with cleaning up the test directories they're
made in.
We also introduce some test wrappers that can automatically
fail tests on error, so tests don't need to be full of error
checks.
Also, now that I'm starting to think more about the flow of
writing tests using QueryScope, we add the missing "full
database" scope option, and make the Add methods return
their operand so (1) you can chain them, (2) you can use
the AddIndex(...) inline in a NewWriteQueryContext.
Also addressed a plausible performance concern in shardList,
and some comments that were stale or incorrect.
The test coverage here is skimpy on the actual RBF-calling
functions because those are trivial. We do, however, significantly
expand coverage in the random write requests, which are now
a mix of random writes and random reads, and add test cases
that at least hit a lot of the error checks once.
The Error() method is changed to be like (testing.T).Error(),
taking ...interface{} and using fmt.Sprint on them.
There's also some minor tweaks such as making the visualizations
more consistent, testing visualization generation on two kinds
of keysplitter, and so on.
This is living in a subdirectory for now so we can have better
turnaround time on tests and not have to build everything else
along with it.
This covers the logic that we can have *without* actually using
databases or the filesystem in any way, just to provide a framework
that lets us validate the logic handling overlapping queries.
The overall purpose of this is to prevent deadlocks, by ensuring
that database locks are only taken when we have already proven
that they are available. In short, the QueryContext preregisters
its "scope" -- the set of things it may want to lock. The operation
of creating the QueryContext can block, but it blocks with no
database locks held. Once it is unblocked, the scope it has reported
is now considered unavailable, and no other QueryContext using any
overlapping scope can complete creation until this QueryContext
completes. While it's running, the QueryContext can't request write
access to anything outside its scope. Thus, once created, a
QueryContext can always proceed, without being blocked, until it's
done.
Note that this does not fully address multi-node behaviors;
once you have a QueryContext blocking things, you need to not
make queries to other nodes that could be blocked in turn by those
nodes. In short, no write queries to other nodes while holding a
write-type QueryContext on the local node, because if two nodes
do that to each other at once, they can both be blocked.
We believe RBF is currently designed such that read-only accesses
don't block progress on writes, so non-write access doesn't
create problems.
We also have some code to allow us to create dot-format output
from the components of this system, which is mostly intended to
be a debugging tool.
