mirror of
https://github.com/featurebasedb/featurebase.git
synced 2026-08-28 10:54:59 +00:00
0c94a53a73
This is living in a subdirectory for now so we can have better
turnaround time on tests and not have to build everything else
along with it.
This covers the logic that we can have *without* actually using
databases or the filesystem in any way, just to provide a framework
that lets us validate the logic handling overlapping queries.
The overall purpose of this is to prevent deadlocks, by ensuring
that database locks are only taken when we have already proven
that they are available. In short, the QueryContext preregisters
its "scope" -- the set of things it may want to lock. The operation
of creating the QueryContext can block, but it blocks with no
database locks held. Once it is unblocked, the scope it has reported
is now considered unavailable, and no other QueryContext using any
overlapping scope can complete creation until this QueryContext
completes. While it's running, the QueryContext can't request write
access to anything outside its scope. Thus, once created, a
QueryContext can always proceed, without being blocked, until it's
done.
Note that this does not fully address multi-node behaviors;
once you have a QueryContext blocking things, you need to not
make queries to other nodes that could be blocked in turn by those
nodes. In short, no write queries to other nodes while holding a
write-type QueryContext on the local node, because if two nodes
do that to each other at once, they can both be blocked.
We believe RBF is currently designed such that read-only accesses
don't block progress on writes, so non-write access doesn't
create problems.
We also have some code to allow us to create dot-format output
from the components of this system, which is mostly intended to
be a debugging tool.
(cherry picked from commit e3d137f29c)
63 lines
3.2 KiB
Go
63 lines
3.2 KiB
Go
// Copyright 2022 Molecula Corp (DBA FeatureBase). All rights reserved.
|
|
|
|
/*
|
|
Package querycontext provides a semi-transactional layer wrapping access
|
|
to multiple underlying transactional databases.
|
|
|
|
# Background
|
|
|
|
Within a single transactional database, a transaction provides the
|
|
ability to make multiple changes atomically (either all or none are
|
|
visible, you can never observe only some of them), and manages the
|
|
lifetime of data access; for instance, a database could return objects
|
|
which point into database-managed storage, but which will be invalidated
|
|
after the transaction completes. A transaction also provides a stable
|
|
view of the data; even if you aren't writing, a transaction you open
|
|
will be able to see the same data across multiple queries, even if
|
|
changes are being made to the database.
|
|
|
|
Featurebase uses multiple transactional databases in parallel, and we
|
|
want to preserve transactional semantics across these databases as well
|
|
as we can. We want to be able to see a consistent view of each database,
|
|
to be able to make multiple changes in sequence which are committed
|
|
atomically, and to ensure that data we're still using isn't invalidated.
|
|
|
|
We can approximate this by having an object which tracks transactions
|
|
for each individual database, and then closes them all at once, or commits
|
|
them all at once. However, if we do this naively, it becomes very likely
|
|
for us to end up deadlocked -- we can have two queries running each of
|
|
which is holding a lock, and will hold it until it finishes running,
|
|
and each of which is waiting for access to a lock the other is holding.
|
|
|
|
The QueryContext offers a resolution to this by tracking the scope of
|
|
each query's prospective writes. Queries register their prospective writes
|
|
when they're created, and cannot begin running (potentially acquiring
|
|
locks) until there are no existing queries that could contest any locks
|
|
with them.
|
|
|
|
# Data Organization
|
|
|
|
The overall data set maintained by Featurebase is divided into Indexes
|
|
(which correspond roughly to SQL tables), Fields, Views, and Shards. Shards
|
|
divide the set of records into blocks of adjacent records, while the
|
|
Index/Field/View hierarchy corresponds more to tables and columns. The
|
|
intersection of {index, field, view, shard} is called a fragment. The
|
|
QueryContext/TxStore design abstracts away the question of which fragments
|
|
are stored in which backend database files; you specify your scope in
|
|
terms of indexes, fields, and shards, and you request access to fragments.
|
|
|
|
# Usage
|
|
|
|
The major exported types from this package are [KeySplitter], [QueryContext],
|
|
[QueryScope], and [TxStore]. In general, the usage of the package is that
|
|
you create a TxStore representing your underlying collection of databases,
|
|
and using a KeySplitter to determine how the overall data stored is split
|
|
into individual database files. When you wish to operate on data, you
|
|
request a new QueryContext from the TxStore. If you want to write, you
|
|
use a QueryScope to identify the scope of which parts of the data you
|
|
want to write to. (The request for a QueryContext will block until it
|
|
can be satisfied.) The QueryContext then provides read and write access
|
|
to individual fragments, and handles any necessary multiplexing between
|
|
fragment access and database access.
|
|
*/
|
|
package querycontext
|