468461fbcf
* Support Drop Table in serverless (include Snapshotter, Writelogger) * Finish Database methods Things like: - `Databases` - `DatabaseByID` - `DatabaseByName` - `DropDatabase` * Change Poller to use NodeService instead of its own map Instead of the Poller maintaining its own map of Addresses to poll, this commit changes the Poller to use the NodeService interface to get all known nodes from the Controller. The next commit needs to: Next, the logic in the boltdb NodeService implementation was moved to the boltdb Balancer implementation. That way, the Balancer can be the source of truth for all things nodes/workers/jobs. * Move NodeService from Controller to Balancer This commit moves the implementation of the NodeService into the Balancer, and aligns `Balancer.AddWorker` with `NodeService.CreateNode` so that they stay in sync. (Same for `Balancer.RemoveWorker` and `NodeService.DeleteNode`). * fix import of private repo * Fix go vet issues * Fix bug in DeregisterNode We need to remove the node from the NodeService even if it's not assigned to a database. The logic had a bug in it. This also adds some no-op implementations for SnapshotService and WriteloggerService. If a directory was not configured for that, then the computer node would panic on trying to read from the Snapshotter upon receiving a Directive. * queryer response content-type: json * Add support for NULL to WriteloggerDir and SnapshotterDir configs This commit changes the way WriteLoggerDir and SnapshotterDir are handled. If value is empty `""`, an error will be returned on computer startup. If value is `"NULL"`, a no-op implementation of the service will be used. This would be for a case that wanted to run serverless on-prem with no durable storage. Finally, any other value will be used as the directory to use. Some things which aren't considered here and may result in unexpected behavior: - a value with spaces `" "` - any "null" which is not "NULL"... like lowercase. * Finish the DropTable test * Change "disable service" value to case-insensitive "off" This commit also removes an unnecessary sleep in the tests. * Fix docker-compose variables for IDK test Co-authored-by: Matthew Jaffee <jaffee@pilosa.com> |
||
|---|---|---|
| .. | ||
| boltdb | ||
| computer | ||
| http | ||
| mds | ||
| queryer | ||
| server | ||
| snapshotter | ||
| storage | ||
| test | ||
| writelogger | ||
| address.go | ||
| address_test.go | ||
| controller.go | ||
| dax.go | ||
| directive.go | ||
| docker-compose.yml | ||
| errors.go | ||
| Makefile | ||
| node.go | ||
| README.md | ||
| role.go | ||
| schema.go | ||
| service_manager.go | ||
| snapshot.go | ||
| table.go | ||
| table_test.go | ||
| time.go | ||
| transaction.go | ||
| versioned_partition.go | ||
| versioned_shard.go | ||
| workerjob.go | ||
| workerjob_test.go | ||
DAX
DAX encapsulates anything which covers all of the services which make up the "disaggregation of storage and compute" project. Initially, this will include integration tests which pull in things like Metadata Services (MDS), including the Controller, as well as FeatureBase and IDK-based ingesters.
Setting up the tests
The DAX test currently requires docker images for: featurebase and datagen.
If at any point you run into problems with go mod failing to reference a private
repo, make sure that you have gitlab.com/featurebasedb in your GOPRIVATE
environment variable.
Note that during the docker image build step, go mod vendor is run, which
creates a vendor directory in the root directory, and copies that to docker
during the build stage. Just be aware of this; you may want to remove that
vendor directory after you're done building docker images.
Possible Configuruation
The following were relevent when the dax code was in a separate repository. These may no longer be relevant.
I needed to but this in my ~/.profile file:
export GOPRIVATE=github.com/featurebasedb,gitlab.com/featurebasedb
And this in my ~/.gitconfig
[url "ssh://git@github.com/"]
insteadOf = https://github.com/
[url "ssh://git@gitlab.com/"]
insteadOf = https://gitlab.com/
Then make docker ran successfully.
Build the FeatureBase docker image
- Check out the dax branch of the featurebase repository.
- Run
make docker-image-featurebaseto build the docker image - You should now have an image in docker named
dax/featurebasewith the taglatest.
Build the Datagen docker image
cd <featurebase_repo_root>/idk- Run
make docker-image-datagento build the docker image - You should now have an image in docker named
dax/datagenwith the taglatest.
Running the tests
- Check out the dax branch of the featurebase repository.
- Change into the
daxdirectory:cd dax - Run
make test-integration.