We switch everything to use QueryContext/QueryRead/etc instead
of Qcx/Tx.
We drop the short_txkey subpackage (it's now handled by either
keys or querycontext).
We drop all the dbshard stuff, and all the tx/txfactory stuff.
We remove all the things that related to the old "Block" concept,
which was mostly used by the anti-entropy code, but had one
fragmentary usage left in the ImportRoaringOverwrite case of
ImportRoaring. That's replaced by using a rewriter that deletes
all bits (not just bits in specific columns) from an existing
thing, but writes in new bits. Actually we could probably do that
better with a custom "eradicate-rewriter" that doesn't try to
be clever, and just eliminates things.
This includes a number of minor bug fixes that were
exposed by getting the testing to work. For example:
* When checking whether an operation "requires write", we
now consider a Delete a kind of a Write, because it is.
* Several tests were relying on the fact that writes through
Qcx were being committed whether or not the Qcx was ever
told to finish. With QueryContext, you actually have to
reach a Commit() or the writes don't happen (except for
special cases in Delete).
* Replaced a lot of panics with t.Fatalf in tests.
There's also some minor staticcheck fixes, like deleting the
unused "db" member of a boltdb transaction wrapper.
This is a unification of a number of bug fixes, feature additions,
and so on. Features include:
* Dropping the "New" from NewWrite/NewRead.
* IndexName->keys.Index, etc.
* Add a new "Flush" operation which is necessary to get the
intended behavior of Delete, which allows us to commit/flush
changes without letting go of a write lock.
* Some additional wrapping and locking in rbfTxWrappers to
support that. rbfQueryRead/Write now forward their calls
to the parent rbfTxWrappers, so it can lock around the
reference to its underlying tx, so the flush operation can
replace that tx safely.
* AddIndexShards now treats no shards as "all shards", to
simplify call sites.
* Added parameters to NewRBFTxStore to let it interact with
executor's logger and worker pool.
* Internally, support explicit closes of parts of the database
which can also check for errors and fail if it's in use.
* Add ability to request a map of fields and views in use
for a given index/shard pair. This is probably deprecated
but we need it for the way backup/restore work.
* Add ability to request a complete map of the database showing
which shards exist for which index/field/view tuples. This is
backwards from how we store things on disk, but we need it
to allow creating the right in-memory data structures on
database open.
* Add "Backend()" method to let us distinguish backends in case
we some day have them again.
* Support deleting indexes, fields, or fragments.
* Support Backup (returning a ReadCloser that dumps the RBF
file, implicitly merging any current WAL) and Restore (create
a new RBF file).
* Change directory structure and fragment keys to match existing
databases, so we should in theory be able to open an existing
data directory.
* Fragment delete probably doesn't lock correctly and this
should be reviewed.
* Export the DOT-format Dump so we can hook it up to a debug
endpoint. This wants to be explored more; ideally the front-end
UI should be able to display this.
* Create a NopTxStore which can be used like a TxStore but everything
that can error errors out. This is then used to let a holder that
hasn't had a txstore initialized work anyway.
There's at least a couple of open issues that need to be revisited
here.
In developing QueryContext, I found it more expressive to have defined
types for index/field/view/shard keys. Then I wanted to refer to those
names from elsewhere. This package has no dependencies so anything
can import it and we can all agree on these types. Later I want to
migrate everything else to use these names.
* Change JSON response name from exec_time to execution-time
Execution time stopped working in the CLI because it uses the latest
json tag.
* Wait, don't break the interface.
* Add a test for the sql response json tags.
This is to make sure that if someone like Travis just goes and changes a
tag name to be more consistent, that we perhaps catch that before it
gets to the end user.
* Change exec_time to execution-time after all
* implemented distinct
* implemented distinct
* uses first cut of a buffer pool, and extendible hashing with thresholded spill to disk
* tests
* cleaned up some stuff around query plan output to make developing tooling easier
* added optimization to call PQL Distinct()
* fixed test
* fix for passing wrong index name in orchestrator
* back out change to DistinctTimestamp
* fix other instance of wrong table name being passed
* use full index name instead of abbreviated one for translation. sigh.
* removed some unused code
Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
- had to make sure we don't snapshot until directive is fully applied
on a computer... otherwise there's races between loading the files and
truncating the write log.
- added a dirty bit to resources and a bool return to incrementing the
write log... don't snapshot if it returns false because that means
there's been no writes. (but make sure you close the storage transaction!)
- added the actually snapshotting routine which just fires every
<timeout> and serially snapshots everything.
- tweaked some logging
- added ability to get all tables in an org/db or literally all. I
think I just needed the "literally all", but it was natural to allow
it to be scoped to org or DB as well.
* expose Transaction on TranslateStore for DAX Snapshotting
* try to fix ramdisk nonsense
apparently, we were running in either a shell env or docker env
randomly, so this could sometimes pass and sometimes fail since the
shell env had the ramdisk set up and docker didn't.
Now we force to run in docker always and set up ramdisk explicitly.
* ramdisk mount should be defined on gitlab runner config now
* debug ramdisk issue?
* fix tests... and a buncha other stuff
Took retry out of CI config because I think it's doing more harm than
good at this point.
The executor test I modified failed when I changed DefaultPartitionN
to 8, but just because stuff was out of order so I made it more
robust.
I edited some data gen stuff to make shorter lines because it was
making grep results unusable.
the actual fix is in translate_boltdb_test.go
* clean up, fix code review feedback
* delete implementation with test coverage
* optimize IN expressions; stop linter complaining
* fixed some uncovered query cases
* skip test in DAX for now
CLOUD-1252
Implemented Jaffee's fix of checking for b.useShardTransactionalEndpoint
and only running the start/finish transaction block if it's false. Moved
stats timing to a separate defer so it could stay out of the if.
* Fix PQL distinct in dax
When issuing a PQL Distinct() call (or any other call with a "index=" arg),
this commit will attempt to convert the value in the index arg with a
TableKeyer.
* Apply change to call.Children as well
* Add some PQL Distinct (join) test coverage
