- had to make sure we don't snapshot until directive is fully applied
on a computer... otherwise there's races between loading the files and
truncating the write log.
- added a dirty bit to resources and a bool return to incrementing the
write log... don't snapshot if it returns false because that means
there's been no writes. (but make sure you close the storage transaction!)
- added the actually snapshotting routine which just fires every
<timeout> and serially snapshots everything.
- tweaked some logging
- added ability to get all tables in an org/db or literally all. I
think I just needed the "literally all", but it was natural to allow
it to be scoped to org or DB as well.
(cherry picked from commit b8b08bc9eb)
* return 200 once plan compilation starts; if error, return error in response.
* removed some commented out code that is definitely not needed.
(cherry picked from commit 9dda3ff215)
* expose Transaction on TranslateStore for DAX Snapshotting
* try to fix ramdisk nonsense
apparently, we were running in either a shell env or docker env
randomly, so this could sometimes pass and sometimes fail since the
shell env had the ramdisk set up and docker didn't.
Now we force to run in docker always and set up ramdisk explicitly.
* debug ramdisk issue?
* fix tests... and a buncha other stuff
The executor test I modified failed when I changed DefaultPartitionN
to 8, but just because stuff was out of order so I made it more
robust.
I edited some data gen stuff to make shorter lines because it was
making grep results unusable.
the actual fix is in translate_boltdb_test.go
* clean up, fix code review feedback
(cherry picked from commit 3693b9950a)
* delete implementation with test coverage
* optimize IN expressions; stop linter complaining
* fixed some uncovered query cases
* skip test in DAX for now
(cherry picked from commit 021219935f)
now that min/max queries don't use a write Tx it seems we need to
separate read and write in the tests. Not sure I 100% understand this.
(cherry picked from commit a58299def4)
CLOUD-1252
Implemented Jaffee's fix of checking for b.useShardTransactionalEndpoint
and only running the start/finish transaction block if it's false. Moved
stats timing to a separate defer so it could stay out of the if.
(cherry picked from commit b1f5264a4b)
* Fix PQL distinct in dax
When issuing a PQL Distinct() call (or any other call with a "index=" arg),
this commit will attempt to convert the value in the index arg with a
TableKeyer.
* Apply change to call.Children as well
* Add some PQL Distinct (join) test coverage
(cherry picked from commit 4e8fe488de)
- check that serverlessStorage is not nil before closing it
- check that we don't already hold a lock on a serverless storage
Manager before trying to load it. This fixed at least one test failure.
(cherry picked from commit 87d1c31607)
* Prevent file corruption when writing tar backup to stdout
FB-1794
Tar backups written to stdout were coming out corrupt. This turned
out to be due to log messages getting written to stdout and ending
up in the tar file. We now check to see if the tar file and the log
are both going to stdout, and if they are, send the logs to stderr
instead.
Testing did not have any kind of consistency or validity check. We
now compare a tar file sent to a file and a tar file sent to stdout
to make sure they're the same. This does not guarantee correctness
but does at least catch this form of corruption.
* trying different index name
Co-authored-by: tgruben <tgruben@gmail.com>
Co-authored-by: Todd Gruben <todd@molecula.com>
(cherry picked from commit a8996a149d)
* Make interfaces more specific than "MDS"
- Introduce `dax.Schemar` interface
- Introduce `dax.Noder` interface
- The rest is generally to standardize on the new interfaces.
- Remove `pilosa.SchemaInfoAPI` interface
- Move `TranslateNode` and `ComputeNode` types from controller to dax package
- Remove `queryer.FeatureBaseImporter`
- Remove `queryer.MDS` interface
- Remove `queryer.Importer` interface
- Identify types using an "MDS" interface and split into Noder/Schemar as necessary
- Changed `Queryer.orchestrator` to a `map[qual]*qualifiedOrchestrator` because we can't share an orchestrator across quals
* Convert orchestrator to use TableKeyer
(cherry picked from commit 14f1930004)
* Fix "qualifer" misspellings
* Remove `track_existence` and `shard_width` from SHOW TABLES output
* Thread Owner, UpdatedAt, UpdatedBy through SchemaAPI
I took the liberty of renaming "LastUpdatedUser" to "UpdateBy" to align
with "UpdatedAt".
(cherry picked from commit 63cfdb5078)
This commit introduces an interface called `TableKeyer` which anything that means to represent a "table"
can implement. Examples are `dax.QualifiedTable`, `dax.Table`, and `string` (for legacy pilosa calls
where Execute simply took `index string`).
In the case of `orchestrator.Execute()` and `qualifiedOrchestrator.Execute()`, we are intentionally strict
about which type of `TableKeyer` the respective method accepts. If we find, in the future, this is too
restrictive, we can loosen that; but for now it helps us understand what is expected.
(cherry picked from commit a61d1a9571)
* Move batch.Importer interface to pilosa.Importer
In addition to moving the interface, it updates all the methods to use
dax.TableID (for example) intead of a string pilosa index name.
* Change unused onPremImporter methods to no-op.
onPremImporter is a wrapper around API which implements the Importer
interface. This is currently only used by sql3 running locally in standard
(i.e not "serverless") mode. Because sql3 always sets
`useShardTransactionalEndpoint = true`, There are several methods which this
implemtation of the Importer interface does not use, and therefore they
intentionally no-op.
(cherry picked from commit 12d608c80d)
* Thread Table.Description through SchemaAPI
* Thread Table.CreatedAt through SchemaAPI
* Thread Field.CreatedAt through SchemaAPI
(cherry picked from commit 734477aaee)
