Add DAX - full list of squashed commits below

In this commit, the Directive is mocked; it doesn't actually reach out
to a controller.

Limits key translation to only those partitions (per index) specified in
the Directive. Attempting to create or find a key (or ID) for a
partition which is not handled by this node will result in an error;
translation requests are no longer forwarded to other nodes.

Limits import into only those shards specified, per index, by the
Directive. Attempting to import into a shard which is not handled by
this node will result in an error.

Stub out /directive endpoint

The `applyDirective()` method still needs to be implemented.

Update mds references to use the new /mds/types structure

In mds, we moved the shared types to mds/types. FeatureBase needs to
reference those instead.

This also bumps the mds version in go.mod.

Implement the Add/Remove Index part of Holder.ApplyDirective()

This adds functionality to `Holder.ApplyDirective()` which adds or
removes indexes (tables) based on those provided in the Directive. Still
to be implemented here: shards and partitions.

WIP: remove client from Batch

Move Batch into its own package: batch

Also, in order to avoid import loops, this introduces packages:
/batch/types
/client/types

Reorganize the Importer-related code

Moved the Importer interface to package: batch
Move the "pilosa client" implementation of the Importer interface to
package: client

Modify batch.NewBatch to take an Importer (not client)

This commit modifies the batch.NewBatch() function to use a functinal
option on Batch to inject an Importer into the Batch. Prior to this,
NewBatch() took a pointer to a client, which was a little too
restrictive. Now, MDS can implement an Importer which uses information
from MDS to determine to which node(s) the import calls should be directed.

Add client.SetAuthToken() method to satisfy SchemaManager interface

Update ApplyDirective logic to include fields.

This needs more work, but it was enough to get a basic test passing.

Move Transaction type into /types package.

Add interface check on batch.Importer no-op implementation

Updated ApplyDirective to create all currently support Field types

There are still the following TODOs:
- [ ] impolement field options (ex: decimal scale, int min/max, etc).
- [ ] `time` fields

Added support for Decimal.Scale in ApplyDirective

Update mds dependency

Add /health endpoint

Update to use dax (dax/mds) instead of mds.

After moving the mds repository into the dax repository as a
sub-package, this commit changes everything in FeatureBase to use the
dax repo instead of the now abandoned mds repo.

Introduce and implment the WriteLogger interfaces.

This adds both a `WriteLogReader` and `WriteLogWriter` interface. They
are both implemented by the implementation: `fileWriteLogger`. The
`fileWriteLogger` uses the dax/writelogger API to append log messages to
files on disk.

Add WriteLogWriter.ImportRoaring method to interface

This commit adds the `ImportRoaring` method to the `WriteLogWriter`
interface. Still to implement are the `Import` and `ImportValue`
methods.

Reorganize the ApplyDirective code

The primary goal was to cache the incoming Directive on the Holder prior
to applying all of the changes in the directive (i.e. loading data from
the WriteLogger) because applying those changes often validated against
the accepted state of the node.

Implement all of the WriteLogger read/write methods

Implement the HTTP WriteLogger implementation

WIP: Introduce shard.Version. Implement snapshotter.

Add HTTP Snapshotter implementation

This also recofigures server to use the HTTPSnapshotter instead of the
FileSnapshotter.

Implement snapshotter: TableKeys

Implement snapshotter: FieldKeys

Dependency dance

last of the dependency dance

Add support for prototype

This adds the Makefile targets to build the docker container and push it
to ECR.

SQL3 changes which break with dax changes

Missed TODO: implement FieldVersion version to WriteLogger

Address bug causing missing TranslateStores to error

Originally, we tried to limit the TranslateStores which get allocated to
only those for which the node is responsible. This works when adding a
new table. But if a table already exists, there's no logic to start
missing TranslateStores.

This reverts back to the old FeatureBase logic which brutishly allocates
a TranslateStore for every partition, even if one is not needed.

We need to address this by allowing the ApplyDirective logic to
initialize TranslateStores when they don't yet exist.

Move the ImportRoaringShardRequest type to the types package

Since the ImportRoaringShardRequest object is part of the Importer
interface, we need to move it to a non-root (i.e. pilosa) package. All
the other interface types are either concrete types or part of a
sub-package (such as roaring). We do this to prevent an implementer of
the interface from having to import the entire pilosa package and risk
circular imports.

buncha changes to support latest dax stuff

Move dax related types to /dax sub-package

This commit moves all the common "dax" types into the /dax sub-package.
The idea is to ensure that featurebase does not import dax at all.
It's ok if dax imports featurebase.
In the future, we might need to split the dax sub-package (common data
types used by muliple molecula data-plan services) into it's own repo.

Add type: dax.Schema

This isn't currently being used; I started to use is as a replacement
for pilosa_client.Schema, but then deferred that. But we'll need to do
it eventually, so it doesn't hurt to have this type in place.

Export RowIDs.Merge() method for use in orchestrator.

Add CreateSQL method to dax.Table type

The CreateSQL() method will return the "CREATE TABLE" statement required
to create the dax.Table.

Comment out confusing writelogger log message.

We need to revisit this, but for now, this log message is confusing.

Also, rename daxSharder to versionStore.

Remove hard-coded AWS account

Implement more FieldOptions such as Epoch

Some of the FieldOption logic was stubbed out in the dax package. This
commit fills that out more; specifically, it adds the
dax.Field.Options.Epoch parameter.

export stuff needed for TopK in orchestrator

export ValCount stuff to implement Percentile in orchestrator

export more stuff to support less code in orchestrator, shared objs

Port dax repo over to featurebase/dax (run all as sub-services)

This commit does ALOT. Sorry.
It introduces a `featurebase dax` sub-command which can be configured to
run the various dax services as sub-services within the same process, or
individually as the lone service in process.

It also changes all the URL paths to be prefixed with the service name.
So for example, instead of calling localhost:8080/status, you would now
call localhost:8080/featurebase/status.

Also, note that all services provide a /health endpoint to confirm they
are running in process.

Clean up integration tests. Remove PILOSA_ config prefix.

Remove duplicate clients (mistake from porting dax to featurebase)

Rename sub-service "featurebase" to "computer"

In the places where we have hard-coded the sub-service name into a URI
path, I've tried to tag the line with a comment containing:
`// #SERVICEPATHPREFIX`

Update copilot manifest files to reference "computer"

Port dax/README.md from dax repository

Separate (toml) Queryer Config from Injections

We needed to separate the toml config from the configuration required to
inject sub-services into the Queryer. I'm not sure this is the best
solution, but it's *a* solution. So here we are.

Clean up (i.e. remove) the queryer "implementations" package

Remove old test file

Run WriteLogger and Snapshotter as local sub-services.

Prior to this commit, the writelogger and snapshotter services only
worked when run as separate services. This allows them to be run in the
same process as all the other dax services.

There is still some naming issues that we should address, but it's
functional for now.

Clean up (i.e. organize) the intra-service interfaces.

Implement alpha Director for local messages from MDS to Computer

Prior to this commit, messages from MDS to the computer service were
still going over http. This commit introduces an interface
implementation which registers the local computer command, and use that
command's API to directly reference methods used by the Director.

Clean up a few more interface names

Add Queryer OpenAPI document.

Update copilot manifests to reflect latest changes

Add OpenAPI documents for WriteLogger and Snapshotter

Add OpenAPI document for MDS service

Add OpenAPI document for Computer service

Consolidate errors to use fb/errors package.

This commit is a first pass at trying to ensure that all of the DAX code
uses:
"github.com/molecula/featurebase/v3/errors"

This package is a wrapper for "github.com/pkg/errors", so going forward
we want to avoid importing that package.

The only method which isn't backward-compatible is `New()`; the
New() method in the featurebase/errors package takes an errors.Code. If
this becomes a problem, we could change this by reverting New() and then
introducing something like NewCoded(). But for now I think it might
actually discourage someone from just creating a New() error without
thinking about how it should be coded.

Introduce VersionStore interface

Move the existing VersionStore code to the `inmem` package as the
in-memory implementation of the new dax.VersionStore interface.

Introduce NodeService interface

With this, the Controller can maintain a registry of nodes by using this
NodeService interface as opposed to an in-memory map of nodes on the
Controller struct.

This also adds an inmem implementation of the NodeService interface.

Introduce controller.Balancer interface

This moves the existing balancer package to controller/naive package.
The idea is to allow us to add a different Balancer implementation in
the future.

Introduce DirectiveVersion interface

This commit also includes *A LOT* of refactoring to use dax.Worker and
dax.Job types everywhere instead of strings.

Introduce Schemar interface

The previous `Schemar` struct was moved to the `schemar/inmem` package,
and `Schemar` is now an interface implemented by that inmem package.

Remove unused type `nUnit`

Add boltdb implementation of VersionStore interface.

This removed the previous sqlite implementation; we decided not to use
sqlite for now (as a basic, local disk implementation) because it
requires CGO.

--------------------------------------------
No longer applicable:

Add sqlite implementation of VersionStore interface.

This commit implements the VersionStore interface using sqlite. Sqlite
requires CGO, so this may not be something we want to include, but it's
implemented here to get a feel for how an external implementation might
be used; the next step will be to determine how the user configured
FeatureBase to run using sqlite as a backing store for services like
MDS.

Add boltdb implementation of NodeService and DirectiveVersion interfaces.

Add boltdb implementation of naive Balancer interfaces.

This includes the two interfaces defined in `naive/balancer.go`:
- WorkerJobService
- FreeJobService

Add boltdb implementation of Schemar interface.

clean up a linter issue

Thread context.Context through all the interfaces.

Some of the interface implementations are going to use context, so we
need to make that part of the interface. The boltdb implementations, for
example, take a context. This is probably so we can do things like
cancel or timeout operations.

Update interfaces to return error; remove `panic(err)` everywhere.

Down-rev grpc version to 1.38.0

Later versions (after 1.42.0?) cause MustRunCluster.Close() in tests to
deadlock.

This commit also adds an `isComputeNode` feature flag around some of the
write log and shard/partition check functionality so that it doesn't run
under normal conditions (this is excercised by running the sql3 tests
for example).

Add MDS_Persistence test to cover meta data persistence

This adds a basic test which configures the MDS container to use boltdb
as its persistence storage, saved on a docker volume. Then, the mds
container is stopped/replaced, and we confirm that the data stored on
the volume is availble to the new MDS container.

Fix a few things after rebase with sql-experiment branch

The lastest version of sql-experiment contains a fairly significan
refactor of the way query iteration works. This commit adjusts for those
changes.

pull dax IDK changes in to FB IDK (#2177)

* pull dax IDK changes in to FB IDK

* Move docker-related IDK build stuff to featurebase root

Building the docker image required the root level go.mod and vendor
directory. This change moves the make targets to the root level
Makefile, and the Dockerfiles now copy the root level vendor directory
(and everything else in the root for that matter).

* Fix batch- and client-related tests

* InitializePoller on MDS restart/replacement

Prior to this change, if MDS was restarted, its internal poller (which
maintains an in-memory list of nodes to poll) is empty. This is bad,
because it doesn't know about nodes that it should be polling.

This change fixes that. Upon MDS startup, it intializes the poller with
the list of nodes that MDS keeps in persistent storage (currently:
boltdb).

* Add EFS volume to MDS Copilot manifest

This allows us to use MDS's persistent storage (via boltdb) in the
Copilot demo by saving metadata in a boltdb file on EFS.

* Thread logger.Logger through all dax components

* Revert some of the breaking changes from DAX development.

When we first started prototyping DAX, we made changes to the
featurebase core code which would have broken the existing featurebase
functionality. This commit reverts some of those changes. Anywhere that
we need to modify core featurebase functionilty, we put it behind some
kind of feature flag. This flag is typically determined by whether the
running node is a "compute" node (i.e. DAX.COMPUTER.RUN = true).

Co-authored-by: Travis Turner <travis@molecula.com>

add packaging for DAX

need cgo for datagen build

bind to 0.0.0.0, pass GOOS and GOARCH explicitly

not sure if the explicit GOOS/GOARCH is actually necessary...

Get INSERT INTO (aka ingest) working through SQL3

This commit does a few things which I'll try do describe here.

- Introduces a Qctx interface. The existing Qcx is an implementation of
  this interface, and can be used exactly how it has been. But this
  allows us to abstract away the notion of Qcx in the Queryer (which is
  handling SQL3) until we're ready to address that. As an example, the
  Qcx has a notion of a featurebase Holder, but that doesn't make sense
  when we're at the Queryer layer. For now, the Qctx used in the Queryer
  is a no-op.

- Adds a ComputeAPI interface implementation for the Queryer. This is
  effectively the Import() and ImportValues() methods used for ingest.
  The logic here handles the incoming ImportRequest by first doing any
  necessary column and row translation for the entire request, then it
  splits the records by shard, and generates a new ImportRequest per
  shard with only the shard-appropriate records.

- Changes the mds.Importer to take an MDS interface implementation
  (which can be an mds client) instead of an mdsAddress. This allows us
  to use a localy MDS implementation rather than assuming we need a
  client to make calls over a network.

Add queryer.Importer interface to handle ingest via SQL (#2203)

* Add queryer.Importer interface to handle ingest via SQL

This is meant to support ingest through SQL when the queryer and the
compute services are running in the same process, or when they are on
seperate processes and need to talk via http client.

* remove datagen from RPM

was originally added as a convenience to generate test data, but is
unused and annoying because datagen doesn't easily cross-compile due
to cgo

* add marshalUnmarshal to controller to avoid passing pointers

passing pointers across API boundaries can cause unpredictable things
in local vs remote configurations.

Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>

"fix" a few issues with wrong default partition numbers

these still need to be properly fixed and actually get the correct
data from MDS

go mod tidy

Introduce TableQualifier (OrganizationID/DatabaseID) (#2220)

* add check in ApplyDirective that version is increasing

fix TestAPIDirective to make version always increasing

* fix docker image build and break out dax test in CI

We have to run the DAX integration tests separately as they call out
to Docker, and so it isn't easy to run them in a Docker container as
the other tests do. So we run them directly on the CI runner which has
Docker and Go installed.

We also explicitly exclude these tests from running during the other
tests.

Also my editor was automatically reformatting some comments badly
which is why I added the "data" thing in those two places

* add timeout to poller

* give Poller a default Logger

apparently we can NPE sometimes, seen in CI: https://gitlab.com/molecula/featurebase/-/jobs/3028286364

* bunch of testing fixes, mostly IDK/DAX related

make MDS error if sendDirectives errors, don't just
log. sendDirectives can error if computer nodes disagree about the
validity of a schema (for example), in which case it might need to get
deleted and user notified somehow. very messy, needs more thought.

re-introduce old env prefix to maintain compatibility with master
branch

make self-contained dax container for IDK testing

build IDK images from source (now that all the source is available
since it's in the same repo)

catch errors in DoExtractQuery in idktest.go

fix IDK bug where prefix path was hardcoded in all cases rather than
only when useMDS was true

fix TestBatchTargetMDS... needed to add field options and catch error
when creating table. also needed an _id field

* fix env prefix in tests

* WIP getting tests to pass, wanna see CI

* don't error if we get a zero version directive and we don't have a

directive yet

* cleanup debugging junk

* "fix" future.rename thing, run IDK tests

* Introduce TableQualifier (OrganizationID/DatabaseID)

This commit introduces a lot of new types (in dax/table.go) related to
TableQualifer (which is made up of OrganizationID and DatabaseID), as
well as things like TableID and TableKey.

For the most part, we try to thread a QualifiedTableID through the
entirety of DAX. There are some places (for example in the Balancers,
which are just aware of string keys) which use a string TableKey
(tbl__org__db__tableid).

* Remove some debugging comments

* Add Org/DB support to CLI.

This commit adds support for special commands:

SET
SET ORG acme
SET DB db1
USE db1

* remove ".pulled" from IDK Makefile

I don't think we need it any more as most things can be built
locally. I think it was only there to refresh the FeatureBase images
that were tagged as master, but we don't need to do that any more.

* Change DAX json tags to kebab-case (i.e. hyphenated)

This commit also renames some struct arguments to more accurately
reflect their type: for example, renaming `Table` to `TableKey` when the
type is TableKey.

* Return DAX TableName in SHOW TABLES (instead of Index.Name)

There are cases where SchemaAPI is used to return DAX friendly table
names (as opposed to featurebase index names, which are DAX TableKey).

This is an attempt to do that. With that said, it's not ideal because
anything could call those API methods and expect the other type.

* Fix a bug which wasn't completely dropping a table.

When using boltdb as a backend, DROP TABLE wasn't removing the
reverse-lookup key for the table in boltdb.

* Remove idk/testenv/certs which got accidentally committed.

also update .gitignore to include those.

* Fix IDK ingest tests to be TableQualifier aware.

* Add example Table types to dax/table.com godoc.

* ignore idk.Main fields for flags, upgrade commandeer

* go mod tidy

* Fix DAX integration tests: ingester using wrong ENV VARs

We change from ORGANIZATION_ID to ORG_ID
and from DATABASE_ID to DB_ID

* Clarify things around idk (docker) tests

* Stop running TestKafkaSourceIntegration with t.Parallel()

This test can't be run in parallel as it's currently written. Doing so
allows for interleaving of messages to the same kafka topic between
tests.

I didn't attempt to modify the test so it could be run in parallel. That
could be done, but left for someone more ambitious.

Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>

Require Directive.Version be a non-zero value. (#2227)

Because the directive cached on the holder is not a pointer, its default
version is 0. In order to avoid having to compare against that, we just
require that Directive.Version start at 1.

General, non-invasive code cleanup and comment adjustment.

Move ImportRoaringShardRequest out of the types package

Early on in the DAX development, I moved ImportRoaringShardRequest into
a types package. There must have been some import loop going on, but
since that is not longer the case, it's safe to move this back into the
core featurebase (er... pilosa) package.

Move Transaction struct back into the pilosa package (from types)

Revert some name changes (cli -> client)

Add DAX Handler CloseTimeout

This was implemented in htt_handler.go, but it had been commented out in
the DAX handler. This just uncomments that and finishes the
implementation.

Remove Qcx from queryer.Importer interface

This sets us up to revert the Qctx interface that was initially
introduced to allow us to abstract away the need for a Qcx when calling
the ComputeAPI from a remote service (i.e. the queryer).

Add some go-doc comments and remove unused code.

Move SchemaManager setup from datagen to idk.Main (#2233)

The set for idk.SchemaManager (for dax implementations) was previously
in datagen. This may have been because of some import loop problem
during development, but that's no longer an issue.

The setup for this should be in idk.Main so anything using that can
leverage the MDS-specific SchemaManager setup.

Fix issues around nil TxFactory

First, don't return a nil. Rather return a new *TxFactory (with no
holder).

Second, don't call `f.holder` in the testhook outside of checking if
`f.holder` is nil.

Wrap all bare errors

Make service prefixes constants

Instead of having `"computer"` throughout the code, use instead a
constant: `dax.ServicePrefixComputer`.

MDS skip errors when sending empty directives

also add in the docker-login and ecr-push changes for serverless DAX

Fix the logic in Directive.IsEmpty() (#2236)

Update the cached value for Index.translatePartitions

In the case where a node already knows about an index, but its
assignment of partitions for that index changes (for example, when
another node goes down and the node in question is now responsible for
more partitions than it previously was), then we need to update the
cached value of Index.translatePartitions because that's used in
translation checks.

minor fixes for IDK-related bugs

WIP: tokenize CLI to access cloud

FB CLI cloud support with automatic token refresh

Also adds support for a GET command which allows making HTTP GET
queries to cloud CP which can be handy for debugging stuff. E.g. GET /v2/databases

buncha little fixes working on writelogger stuff

fix writelogger/snapshotter setup bugs

implement writelogging for importRoaringShard

add debug endpoint to MDS

use shard transactional endpoint in MDS datagen

add debugging to API related to writelogger

revert handleroption change

clean up big PR

remove "GET" command from CLI for making arbitrary HTTP request to
cloud control plane (was a messy hack and not that useful)

remove json tags from FB objects where we had to duplicate the object
elsewhere due to import loops and weren't actually json encoding it

unexport handlerOption which was exported to try to avoid doing
certain things if we're in DAX mode, but I didn't end up merging that code.

remove (hopefully) unecessary extra call to api.indexField

fix some formatting, unexport some vars, godoc, etc

oops, fix build failure

Update FeatureBase CLI to support a standard deployment

The standard deployment uses a different endpoint and request payload.
This commit tries to detect is the standard deployment is being used,
and if so, it uses a standard-specific FBQueryer.

It also modifies the auto-detection logic to try standard featurebase
and dax ports in the case where a port was not provided.

MDS API refactor (#2259)

* MDS API refactor

table IDs are exposed but only created server side

also cleaned up dax Makefile

* clean up review feedback

Co-authored-by: Travis Turner <travis@pilosa.com>

* remove TablesByName

* rip out inmem implementations and use boltdb everywhere

* remove inmem balancer, create bolt tempfile by default on startup

* WIP on snapshot table impl and test

* Minor comment and code layout adjustments.

This commit also adds the `Equals` method to `QualifiedTableID` for
equality comparisons. It's no longer safe to compare struct (two structs
might still be equal even if one of the structs doesn't have a `Name`
value.

* Use a unique docker network for each dax test

Ocassionally we would see some test failures due to a network already
existing. This shouldn't happen, but to avoid that, this commit
generates a unique name for each sub test (which gets deleted at the end
of every test).

* Fix one instance of NewQualifiedTableID losing Name

We should probably check the other instances and see if Name is getting
lost.

* simplify unique network stuff and fix api directive tests

* Remove TableIDRequest and TableIDResponse types for /table-id (#2267)

For the mds/table-id http requests, just use dax.QualifiedTableID as
both the request and response types.

* remove lattice from dax, no error on node re-reg, dax docker-compose

* various updates

* WIP: mds-refactor branch review

* no-op on SnapshotTableKeys if table is not keyed

* Makefile helpers

* add doWeCare so controller doesn't fail unnecessarily

* clean up table creation (#2272)

* Strip underscores from TableID stub name

* fix boltdb versionstore tests: generate unique, sorted tables

* fix controller test related to reregistering a node

* JobSet -> generic Set

Co-authored-by: Travis Turner <travis@pilosa.com>
Co-authored-by: Travis Turner <travis@molecula.com>

Cleanup after rebase on master

The latest rebase on master entailed all the client/batch changes as
well as some of the qcx refactoring. It made for a hairy rebase. This
commit fixes some of the tests that were failing after that rebase.

Fix batch/client import loop missed during rebase (#2280)

It's not surprising that `batch` can't import `client`. It was doing
that here (importing an error type from the `client` package). What is
surprising is that it's okay for `batch_test.go` to import `client` even
though `batch_test.go` is an internal test and therefore part of the
`batch` package.

different boltDB's for schemar/controller, explicit balancers

nice helpers for dax docker-compose, make build really fast

build FB binary outside of docker, then create Docker image with its
working dir in an empty subdirectory so it doesn't send a GB of
context to the daemon.

error on unassigned jobs and use client with timeout

fix CR feedback

deregister batch of nodes

also make removal faster via director dial timeout

implement WorkersForJobPrefix so orchestrator doesn't make up shards

also fix some godocs and remove unused method

Run sub-tasks of a Directive concurrently in a worker pool. (#2275)

* Run sub-tasks of a Directive concurrently in a worker pool.

This allows the compute node to concurrently load shapshot and writelog
data concurrently, instead of one keyset/partition/shard at a time.

It introduces a config parameter called `DirectiveWorkerPoolSize`.

* code review cleanup

* Use unique container names in DAX integration tests

We were seeing "container already exists" errors in CI, so just to be
safe, this commit constructs a unique container name for every container
in the DAX integration test run.

Stub in SystemAPI to Queryer (note: will not work if used)

This just makes is so that dax can compile. Actually implementing
system-table functionality for dax will take some planning.

Tlt/dax merge prep (#2282)

* Remove copilot directory

* Remove Dockerfile-datagen-long

* Remove orphaned RegisterNodeRequest

This type is not defined in the dax/mds/http package.

* implement TIMEQUANTUM and TTL in Table.Field type

* Remove the "service" misdirection in queryer/writelogger/snapshotter.

We had originally used an additional layer, er.. package, for a "service".
The main distinction was that the Config differed in that it was
internal, unlike the Config that we need to provide for the top-level
server config (i.e. toml). Having that additional layer just to support
a different Config seemed premature at best. So I'm removing it.

* Remove dax docker containers no longer used in tests

Since we run everything as "featurebase", we don't have multiple
container types anymore.

* Some minor comment updates

* Remove nfpm stuff related to dax

* Fix linter issues

Fix "duplicate" issues raised by sonarcloud.

run docker components of dax integration tests with coverage

trying to get dax integration coverage

add coverate volume mounts throughout dax integration tests

add a lock, tweak dax Makefile, remote flag on query handler

remove some unused code

convert batch tests to use clustertests to get coverage

maybe fix clustertests

more authclustertests fixes, test is failing locally

but also seems to have been silently failing in CI prior to these
changes... let's see if it's still silent

fix some lint to kick CI

just re-running the job wasn't working... strange behavior

remove RetryLogic test and pipe which don't work

RetryLogic test removed due to etcd changes. Seebs thinks we shouldn't
test this here.

Pipe was being ignored since we're no longer using "bash -c" to
execute the command. If we need to generate that output file we'll
either have to reintroduce bash -c and set -o pipefail so that it
actually fails properly, or figure out some other solution.

shooting into the dark...

first cut at bulk node registration

remove unused stuff from batch tests, set coverpkg to ../...

batch registration timeout and fix tests

disable most tests and don't run fb background batch test

debuggin!!!!!!!!!

and then he tried this....

Implement importer (for INSERT INTO) in the Queryer

Prior to this, we we passing a nil value in for the importer to the
planner.NewExecutionPlanner in the Queryer. This meant that INSERT INTO
statements didn't work. Now they should.

It uses the importer that we build for IDK in /idk/mds/importer.go, and
wrapps that with a type that can determine if the provided string
"index" is of the form indexName or TableKey.

turn off debug mode, fix log saving

Run sql3 test definitions in a dax integration test

There are currently 22 tests which are not passing. They are skipped in
the "skips" slice.

WIP, not working, pql queries to tests

Add TableQualifier to PQL query logic in the Queryer

Add more PQL tests to the keyed table

Allow instant node registration if registration-batch-timeout=0

When running dax services in process, we don't want to wait 3s for the
compute node to register; we know it's there because it's in the same
process.

Fixes related to IncludesColumn PQL test.

Tests for ConstRow and FieldValue

cleanup

add UnionRows and Options, better error reporting on bad queries

delete unused schemar client.go, clean up unused in batch test CI

move test timeouts into more reasonable territory

apparently this had already been done, but got merge-stommped at some point

move dax bolt test helpers into dax package

Add computer CheckIn routine (#2296)

* Add computer CheckIn routine

This adds a background routine which sends a "check-in" request to MDS
every <interval>. This is to address the case where the poller has
removed a computer node from the node list (due to a network fault, for
example), but the node is still healthy and becomes available again. In
that case, the node needs to "check-in" to tell MDS it is still there.
MDS will likely send the node a new directive with Method=reset telling
the node to delete all of its data an apply the latest directive.

* Don't send directives to Deregistered (i.e. removed) nodes

We have an issue where we're locking on sendDirective in the
controller, and when the node is unavailable, the send hangs and never
releases the lock. This is a temporary fix for that until we address the
real problem.

Fix .gitlab-ci.yml after rebase

fix some indentation shenanigans

(cherry picked from commit 20a8b5713a)

.gitignore

@@ -75,3 +75,11 @@ tags.dot
 
 # SQL3
 /sql3/sql3.html
+
+staticcheck.conf
+
+
+.quick
+dax/dax-data
+
+coverage-from-docker

Dockerfile

@@ -38,7 +38,7 @@ FROM alpine:3.13.2 as runner
 
 LABEL maintainer "dev@molecula.com"
 
-RUN apk add --no-cache curl jq
+RUN apk add --no-cache curl jq tree
 
 COPY --from=pilosa-builder /pilosa/build/featurebase /
 

Dockerfile-clustertests

@@ -20,16 +20,19 @@ RUN apt install -y docker.io
 ADD https://github.com/docker/compose/releases/latest/download/docker-compose-Linux-x86_64 /usr/local/bin/docker-compose
 RUN chmod +x /usr/local/bin/docker-compose
 
+WORKDIR /go/src/github.com/molecula/featurebase/cmd/featurebase
+
 # generate an instrumented binary to allow for calculating code coverage for clustertests
 # the entrypoint for the binary is TestRunMain, which is wrapper for main
-RUN cd /go/src/github.com/featurebasedb/featurebase/cmd/featurebase && \
+RUN go test -covermode=atomic -coverpkg=../../... -c -tags testrunmain -o featurebase
-    go test -covermode=atomic -coverpkg=../../... -c -tags testrunmain -o featurebase && \
+RUN cp /go/src/github.com/molecula/featurebase/cmd/featurebase/featurebase /featurebase
-    cp /go/src/github.com/featurebasedb/featurebase/cmd/featurebase/featurebase /featurebase
 
 COPY NOTICE /NOTICE
 
 EXPOSE 10101
 VOLUME /data
 
-ENTRYPOINT ["bash", "-c"]
+
-CMD ["/featurebase", "-test.run=TestRunMain", "-test.coverprofile=/results/coverage.out", "server", "--data-dir", "/data", "--bind", "http://0.0.0.0:10101"]
+# use e.g. "-test.coverprofile=/results/coverage.out"
+CMD ["/featurebase", "-test.run=TestRunMain", "server"]
+

Dockerfile-clustertests-client

@@ -19,9 +19,10 @@ RUN apt install -y docker.io
 ADD https://github.com/docker/compose/releases/latest/download/docker-compose-Linux-x86_64 /usr/local/bin/docker-compose
 RUN chmod +x /usr/local/bin/docker-compose
 
-RUN cd /go/src/github.com/featurebasedb/featurebase/cmd/featurebase && \
+WORKDIR /go/src/github.com/molecula/featurebase/cmd/featurebase
-    go test -covermode=atomic -coverpkg=../../... -c -tags testrunmain -o featurebase && \
+
-    cp /go/src/github.com/featurebasedb/featurebase/cmd/featurebase/featurebase /featurebase
+RUN go test -covermode=atomic -coverpkg=../../... -c -tags testrunmain -o featurebase
+RUN cp /go/src/github.com/molecula/featurebase/cmd/featurebase/featurebase /featurebase
 
 
 COPY NOTICE /NOTICE
@@ -31,5 +32,6 @@ COPY ./internal/clustertests /go/src/github.com/featurebasedb/featurebase/intern
 EXPOSE 10101
 VOLUME /data
 
-ENTRYPOINT ["bash", "-c"]
+WORKDIR /go/src/github.com/molecula/featurebase
+
 CMD ["/featurebase", "-test.run=TestRunMain", "-test.coverprofile=/results/coverage.out", "server", "--data-dir", "/data", "--bind", "http://0.0.0.0:10101"]

Dockerfile-datagen

@@ -0,0 +1,47 @@
+# syntax=docker/dockerfile:1
+
+##########################
+### datagen builder    ###
+##########################
+
+FROM golang:alpine as builder
+
+WORKDIR /featurebase
+
+COPY . ./
+
+RUN apk add --no-cache build-base bash git make librdkafka pkgconfig
+
+# install librdkafka
+RUN git clone https://github.com/edenhill/librdkafka.git
+RUN cd librdkafka && ./configure --prefix /usr && make && make install
+
+ENV PKG_CONFIG_PATH=/usr/lib/pkgconfig/
+
+RUN cd idk && make build-datagen
+
+# ENTRYPOINT ["tail", "-f", "/dev/null"]
+
+#########################
+### datagen runner    ###
+#########################
+
+FROM alpine:3.15.3 as runner
+
+WORKDIR /
+
+LABEL maintainer "dev@molecula.com"
+
+RUN apk add --no-cache curl jq
+
+COPY --from=builder /featurebase/idk/build/datagen /bin/
+COPY --from=builder /usr/lib/librdkafka* /usr/lib/
+COPY idk/datagen/testdata/* /testdata/
+
+EXPOSE 8080
+
+# VOLUME /data
+# ENV ADDR 0.0.0.0:8080
+
+#ENTRYPOINT ["sleep", "infinity"]
+ENTRYPOINT ["datagen"]

Dockerfile-dax

@@ -0,0 +1,32 @@
+ARG GO_VERSION=latest
+
+
+###########################
+### FeatureBase Builder ###
+###########################
+
+FROM golang:${GO_VERSION} as featurebase-builder
+ARG MAKE_FLAGS
+WORKDIR /fb
+
+COPY . ./
+RUN make build FLAGS="-o build/featurebase" ${MAKE_FLAGS}
+
+##########################
+### FeatureBase runner ###
+##########################
+
+FROM alpine:3.13.2 as runner
+
+LABEL maintainer "dev@featurebase.com"
+
+RUN apk add --no-cache curl jq tree
+
+COPY --from=featurebase-builder /fb/build/featurebase /
+
+COPY NOTICE /NOTICE
+
+EXPOSE 8080
+
+ENTRYPOINT ["/featurebase"]
+CMD ["dax"]

Dockerfile-dax-quick

@@ -0,0 +1,18 @@
+ARG GO_VERSION=latest
+
+##########################
+### FeatureBase runner ###
+##########################
+
+FROM alpine:3.13.2 as runner
+
+LABEL maintainer "dev@featurebase.com"
+
+RUN apk add --no-cache curl jq tree
+
+COPY ./fb_linux /featurebase
+
+EXPOSE 8080
+
+ENTRYPOINT ["/featurebase"]
+CMD ["dax"]

Makefile

@@ -1,4 +1,5 @@
-.PHONY: build clean build-lattice cover cover-viz default docker docker-build docker-tag-push generate generate-protoc generate-pql generate-statik generate-stringer install install-protoc-gen-gofast install-protoc install-statik install-peg test 
+.PHONY: build clean build-lattice cover cover-viz default docker docker-build docker-tag-push generate generate-protoc generate-pql generate-statik generate-stringer install install-protoc-gen-gofast install-protoc install-statik install-peg test docker-login
+
 VERSION := $(shell git describe --tags 2> /dev/null || echo unknown)
 VARIANT = Molecula
 GO=go
@@ -18,6 +19,7 @@ RACE_TEST_TIMEOUT=10m
 export GO111MODULE=on
 export GOPRIVATE=github.com/molecula
 export CGO_ENABLED=0
+AWS_ACCOUNTID ?= undefined
 
 # Run tests and compile Pilosa
 default: test build
@@ -26,7 +28,7 @@ default: test build
 clean:
 	rm -rf vendor build
 	rm -f *.rpm *.deb
-	
+
 # Set up vendor directory using `go mod vendor`
 vendor: go.mod
 	$(GO) mod vendor
@@ -92,10 +94,10 @@ build:
 	$(GO) build -tags='$(BUILD_TAGS)' -ldflags $(LDFLAGS) $(FLAGS) ./cmd/featurebase
 
 package:
-	go build -o featurebase ./cmd/featurebase
+	GOOS=$(GOOS) GOARCH=$(GOARCH) FLAGS="-o featurebase" $(MAKE) build
 	GOARCH=$(GOARCH) VERSION=$(VERSION) nfpm package --packager deb --target featurebase.$(VERSION).$(GOARCH).deb
 	GOARCH=$(GOARCH) VERSION=$(VERSION) nfpm package --packager rpm --target featurebase.$(VERSION).$(GOARCH).rpm
-	
+
 # We allow setting a custom docker-compose "project". Multiple of the
 # same docker-compose environment can exist simultaneously as long as
 # they use different projects (the project name is prepended to
@@ -122,10 +124,15 @@ authclustertests: vendor
 	PROJECT=$(PROJECT) ENABLE_AUTH=1 $(DOCKER_COMPOSE) -f internal/clustertests/docker-compose.yml run client1
 	CLUSTERTESTS_FB_ARGS=$(AUTH_ARGS) $(DOCKER_COMPOSE) -f internal/clustertests/docker-compose.yml down
 
-# Install FeatureBase
+# Install FeatureBase and IDK
-install:
+install: install-featurebase install-idk
+
+install-featurebase:
 	$(GO) install -tags='$(BUILD_TAGS)' -ldflags $(LDFLAGS) $(FLAGS) ./cmd/featurebase
 
+install-idk:
+	$(MAKE) -C ./idk install
+
 # Build the lattice assets
 build-lattice:
 	docker build -t lattice:build ./lattice
@@ -187,6 +194,48 @@ docker-image: vendor
 	    --tag featurebase:$(VERSION) .
 	@echo Created docker image: featurebase:$(VERSION)
 
+docker-image-featurebase: vendor
+	docker build \
+	    --build-arg GO_VERSION=$(GO_VERSION) \
+		--file Dockerfile-dax \
+		--tag dax/featurebase .
+
+docker-image-featurebase-test: vendor
+	docker build \
+	    --build-arg GO_VERSION=$(GO_VERSION) \
+		--file Dockerfile-clustertests \
+		--tag dax/featurebase-test .
+
+
+# build-for-quick builds a linux featurebase binary outside of docker
+# (which is much faster for some reason), and places it in the .quick
+# subdirectory.
+build-for-quick:
+	GOOS=linux $(MAKE) build FLAGS="-o .quick/fb_linux"
+
+# docker-image-featurebase-quick uses a pre-built featurebase binary
+# to quickly create a fresh docker image without needing to send the
+# context of the featurebase top level directory.
+docker-image-featurebase-quick: build-for-quick
+	docker build \
+	    --build-arg GO_VERSION=$(GO_VERSION) \
+	    --file Dockerfile-dax-quick ./.quick/
+
+
+docker-image-datagen: vendor
+	docker build --tag dax/datagen --file Dockerfile-datagen .
+
+ecr-push-featurebase: docker-login
+	docker tag dax/featurebase:latest $(AWS_ACCOUNTID).dkr.ecr.us-east-2.amazonaws.com/dax:latest
+	docker push $(AWS_ACCOUNTID).dkr.ecr.us-east-2.amazonaws.com/dax:latest
+
+ecr-push-datagen: docker-login
+	docker tag dax/datagen:latest $(AWS_ACCOUNTID).dkr.ecr.us-east-2.amazonaws.com/dax/datagen:latest
+	docker push $(AWS_ACCOUNTID).dkr.ecr.us-east-2.amazonaws.com/dax/datagen:latest
+
+docker-login:
+	aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin $(AWS_ACCOUNTID).dkr.ecr.us-east-2.amazonaws.com
+
 # Create docker image (alias)
 docker: docker-image # alias
 

api.go

@@ -22,8 +22,10 @@ import (
 	"sync"
 	"time"
 
-	"github.com/featurebasedb/featurebase/v3/disco"
+	"github.com/molecula/featurebase/v3/dax"
-	"github.com/featurebasedb/featurebase/v3/rbf"
+	"github.com/molecula/featurebase/v3/dax/computer"
+	"github.com/molecula/featurebase/v3/disco"
+	"github.com/molecula/featurebase/v3/rbf"
 
 	//"github.com/featurebasedb/featurebase/v3/pg"
 	"github.com/featurebasedb/featurebase/v3/pql"
@@ -51,6 +53,16 @@ type API struct {
 	importWork           chan importJob
 
 	Serializer Serializer
+
+	writeLogReader     computer.WriteLogReader
+	writeLogWriter     computer.WriteLogWriter
+	snapshotReadWriter computer.SnapshotReadWriter
+
+	directiveWorkerPoolSize int
+
+	// isComputeNode is set to true if this node is running as a DAX compute
+	// node.
+	isComputeNode bool
 }
 
 func (api *API) Holder() *Holder {
@@ -77,10 +89,50 @@ func OptAPIImportWorkerPoolSize(size int) apiOption {
 	}
 }
 
+func OptAPIWriteLogReader(wlr computer.WriteLogReader) apiOption {
+	return func(a *API) error {
+		a.writeLogReader = wlr
+		return nil
+	}
+}
+
+func OptAPIWriteLogWriter(wlw computer.WriteLogWriter) apiOption {
+	return func(a *API) error {
+		a.writeLogWriter = wlw
+		return nil
+	}
+}
+
+func OptAPISnapshotter(snap computer.SnapshotReadWriter) apiOption {
+	return func(a *API) error {
+		a.snapshotReadWriter = snap
+		return nil
+	}
+}
+
+func OptAPIDirectiveWorkerPoolSize(size int) apiOption {
+	return func(a *API) error {
+		a.directiveWorkerPoolSize = size
+		return nil
+	}
+}
+
+func OptAPIIsComputeNode(is bool) apiOption {
+	return func(a *API) error {
+		a.isComputeNode = is
+		return nil
+	}
+}
+
 // NewAPI returns a new API instance.
 func NewAPI(opts ...apiOption) (*API, error) {
 	api := &API{
 		importWorkerPoolSize: 2,
+		writeLogReader:       computer.NewNopWriteLogReader(),
+		writeLogWriter:       computer.NewNopWriteLogWriter(),
+		snapshotReadWriter:   computer.NewNopSnapshotReadWriter(),
+
+		directiveWorkerPoolSize: 2,
 	}
 
 	for _, opt := range opts {
@@ -104,7 +156,7 @@ func NewAPI(opts ...apiOption) (*API, error) {
 	return api, nil
 }
 
-// Setter for API options.
+// SetAPIOptions applies the given functional options to the API.
 func (api *API) SetAPIOptions(opts ...apiOption) error {
 	for _, opt := range opts {
 		err := opt(api)
@@ -554,11 +606,20 @@ func (api *API) ImportRoaring(ctx context.Context, indexName, fieldName string,
 		return errors.Wrap(err, "validating api method")
 	}
 
+	api.server.logger.Debugf("ImportRoaring: %v %v %v", indexName, fieldName, shard)
 	index, field, err := api.indexField(indexName, fieldName, shard)
 	if index == nil || field == nil {
 		return err
 	}
 
+	// This node only handles the shard(s) that it owns.
+	if api.isComputeNode {
+		directive := api.holder.Directive()
+		if !shardInShards(dax.ShardNum(shard), directive.ComputeShards(dax.TableKey(index.Name()))) {
+			return errors.Errorf("import request shard is not supported (roaring): %d", shard)
+		}
+	}
+
 	if err = req.ValidateWithTimestamp(index.CreatedAt(), field.CreatedAt()); err != nil {
 		return newPreconditionFailedError(err)
 	}
@@ -608,11 +669,66 @@ func (api *API) ImportRoaring(ctx context.Context, indexName, fieldName string,
 
 		// Exit once all nodes are processed.
 		if maxNode == len(nodes) {
+			if api.isComputeNode && !req.SuppressLog {
+				// Write the request to the write logger.
+				partition := disco.ShardToShardPartition(indexName, shard, disco.DefaultPartitionN)
+				msg := &computer.ImportRoaringMessage{
+					Table:           indexName,
+					Field:           fieldName,
+					Partition:       partition,
+					Shard:           shard,
+					Clear:           req.Clear,
+					Action:          req.Action,
+					Block:           req.Block,
+					UpdateExistence: req.UpdateExistence,
+					Views:           req.Views,
+				}
+
+				// Get the current version for shard.
+				version, err := api.getOrCreateShardVersion(ctx, indexName, shard)
+				if err != nil {
+					return errors.Wrap(err, "get or creating shard version")
+				}
+
+				tkey := dax.TableKey(indexName)
+				qtid := tkey.QualifiedTableID()
+				partitionNum := dax.PartitionNum(partition)
+				shardNum := dax.ShardNum(shard)
+
+				api.server.logger.Debugf("importroaring writing to writelogger: %+v, %[1]T len(msg.Views): %d, table: %s", api.writeLogWriter, len(msg.Views), msg.Table)
+				if err := api.writeLogWriter.WriteShard(ctx, qtid, partitionNum, shardNum, version, msg); err != nil {
+					return err
+				}
+			}
+
 			return qcx.Finish()
 		}
 	}
 }
 
+func (api *API) getOrCreateShardVersion(ctx context.Context, indexName string, shard uint64) (int, error) {
+	tableName := dax.TableName(indexName)
+	shardNum := dax.ShardNum(shard)
+
+	// Here we assume that indexName is the string encoding of QualifiedTableID.
+	qtid, err := dax.QualifiedTableIDFromKey(indexName)
+	if err != nil {
+		return -1, errors.Wrap(err, "decoding qtid from key (indexName)")
+	}
+
+	version, found, err := api.holder.versionStore.ShardVersion(ctx, qtid, shardNum)
+	if err != nil {
+		return -1, errors.Wrap(err, "getting shard version")
+	} else if !found {
+		version = 0
+		api.server.logger.Printf("could not find version for shard: %s, %d, so creating 0", tableName, shardNum)
+		if err := api.holder.versionStore.AddShards(ctx, qtid, dax.NewShard(shardNum, version)); err != nil {
+			return -1, errors.Wrap(err, "adding shard 0")
+		}
+	}
+	return version, nil
+}
+
 // DeleteField removes the named field from the named index. If the index is not
 // found, an error is returned. If the field is not found, it is ignored and no
 // action is taken.
@@ -1050,6 +1166,22 @@ func (api *API) IndexInfo(ctx context.Context, name string) (*IndexInfo, error)
 	return nil, ErrIndexNotFound
 }
 
+// FieldInfo returns the same information as Schema(), but only for a single
+// field.
+func (api *API) FieldInfo(ctx context.Context, indexName, fieldName string) (*FieldInfo, error) {
+	idx, err := api.IndexInfo(ctx, indexName)
+	if err != nil {
+		return nil, err
+	}
+
+	fld := idx.Field(fieldName)
+	if fld == nil {
+		return nil, ErrFieldNotFound
+	}
+
+	return fld, nil
+}
+
 // ApplySchema takes the given schema and applies it across the
 // cluster (if remote is false), or just to this node (if remote is
 // true). This is designed for the use case of replicating a schema
@@ -1183,6 +1315,7 @@ type ImportOptions struct {
 	IgnoreKeyCheck bool
 	Presorted      bool
 	fullySorted    bool // format-aware sorting, internal use only please.
+	suppressLog    bool
 
 	// test Tx atomicity if > 0
 	SimPowerLossAfter int
@@ -1216,6 +1349,13 @@ func OptImportOptionsPresorted(b bool) ImportOption {
 	}
 }
 
+func OptImportOptionsSuppressLog(b bool) ImportOption {
+	return func(o *ImportOptions) error {
+		o.suppressLog = b
+		return nil
+	}
+}
+
 var ErrAborted = fmt.Errorf("error: update was aborted")
 
 func (api *API) ImportAtomicRecord(ctx context.Context, qcx *Qcx, req *AtomicRecord, opts ...ImportOption) error {
@@ -1306,10 +1446,70 @@ func (api *API) Import(ctx context.Context, qcx *Qcx, req *ImportRequest, opts .
 	if err != nil {
 		return errors.Wrap(err, "setting up import options")
 	}
+
+	/////////////////////////////////////////////////////////////////////////////
+	// We build the ImportMessage here BEFORE the call to api.ImportWithTx(),
+	// because something in that method is modifying the values of req, so if we
+	// build ImportMessage after the call to api.ImportWithTx(), then the values
+	// that get logged are incorrect. An example I saw were RowIDS going from:
+	// shard 0 [1, 2, 3]
+	// shard 4 [0]
+	//
+	// to:
+	// shard 0 [1048577, 2097154, 3145731]
+	// shard 4 [1]
+	//
+	// which seem to be the offset in the field shard bitmap.
+	var partition int
+	var msg *computer.ImportMessage
+	if api.isComputeNode && !options.suppressLog {
+		partition = disco.ShardToShardPartition(req.Index, req.Shard, disco.DefaultPartitionN)
+		msg = &computer.ImportMessage{
+			Table:      req.Index,
+			Field:      req.Field,
+			Partition:  partition,
+			Shard:      req.Shard,
+			RowIDs:     make([]uint64, len(req.RowIDs)),
+			ColumnIDs:  make([]uint64, len(req.ColumnIDs)),
+			RowKeys:    make([]string, len(req.RowKeys)),
+			ColumnKeys: make([]string, len(req.ColumnKeys)),
+			Timestamps: make([]int64, len(req.Timestamps)),
+			Clear:      req.Clear,
+
+			IgnoreKeyCheck: options.IgnoreKeyCheck,
+			Presorted:      options.Presorted,
+		}
+		copy(msg.RowIDs, req.RowIDs)
+		copy(msg.ColumnIDs, req.ColumnIDs)
+		copy(msg.RowKeys, req.RowKeys)
+		copy(msg.ColumnKeys, req.ColumnKeys)
+		copy(msg.Timestamps, req.Timestamps)
+	}
+	/////////////////////////////////////////////////////////////////////////////
+
 	err = api.ImportWithTx(ctx, qcx, req, options)
 	if err != nil {
 		return err
 	}
+
+	if api.isComputeNode && !options.suppressLog {
+		// Get the current version for shard.
+		version, err := api.getOrCreateShardVersion(ctx, req.Index, req.Shard)
+		if err != nil {
+			return errors.Wrap(err, "get or creating shard version")
+		}
+
+		tkey := dax.TableKey(req.Index)
+		qtid := tkey.QualifiedTableID()
+		partitionNum := dax.PartitionNum(partition)
+		shardNum := dax.ShardNum(req.Shard)
+
+		// Write the request to the write logger.
+		if err := api.writeLogWriter.WriteShard(ctx, qtid, partitionNum, shardNum, version, msg); err != nil {
+			return err
+		}
+	}
+
 	return nil
 }
 
@@ -1322,11 +1522,20 @@ func (api *API) ImportWithTx(ctx context.Context, qcx *Qcx, req *ImportRequest,
 		return errors.Wrap(err, "validating api method")
 	}
 
+	api.server.logger.Debugf("ImportWithTx: %v %v %v", req.Index, req.Field, req.Shard)
 	idx, field, err := api.indexField(req.Index, req.Field, req.Shard)
 	if err != nil {
 		return errors.Wrap(err, "getting index and field")
 	}
 
+	// This node only handles the shard(s) that it owns.
+	if api.isComputeNode {
+		directive := api.holder.Directive()
+		if !shardInShards(dax.ShardNum(req.Shard), directive.ComputeShards(dax.TableKey(idx.Name()))) {
+			return errors.Errorf("import request shard is not supported (with tx): %d", req.Shard)
+		}
+	}
+
 	if err := req.ValidateWithTimestamp(idx.CreatedAt(), field.CreatedAt()); err != nil {
 		return errors.Wrap(err, "validating import value request")
 	}
@@ -1450,7 +1659,8 @@ func (api *API) ImportRoaringShard(ctx context.Context, indexName string, shard
 	defer finisher(&err1)
 
 	if !req.Remote {
-		return errors.New("forwarding unimplemented on this endpoint")
+		err1 = errors.New("forwarding unimplemented on this endpoint")
+		return err1
 	}
 
 	for _, viewUpdate := range req.Views {
@@ -1514,6 +1724,42 @@ func (api *API) ImportRoaringShard(ctx context.Context, indexName string, shard
 		}
 	}
 
+	if api.isComputeNode && !req.SuppressLog {
+		partition := disco.ShardToShardPartition(indexName, shard, disco.DefaultPartitionN)
+		msg := &computer.ImportRoaringShardMessage{
+			Table:     indexName,
+			Partition: partition,
+			Shard:     shard,
+			Views:     make([]computer.RoaringUpdate, len(req.Views)),
+		}
+		for i, view := range req.Views {
+			msg.Views[i] = computer.RoaringUpdate{
+				Field:        view.Field,
+				View:         view.View,
+				Clear:        view.Clear,
+				Set:          view.Set,
+				ClearRecords: view.ClearRecords,
+			}
+		}
+		// Get the current version for shard.
+		version, err := api.getOrCreateShardVersion(ctx, indexName, shard)
+		if err != nil {
+			err1 = errors.Wrap(err, "get or creating shard version")
+			return err1
+		}
+		tkey := dax.TableKey(indexName)
+		qtid := tkey.QualifiedTableID()
+		partitionNum := dax.PartitionNum(partition)
+		shardNum := dax.ShardNum(shard)
+
+		api.server.logger.Debugf("importroaringshard writing shard to writelogger: %+v, len(msg.Views): %d, table: %s", api.writeLogWriter, len(msg.Views), msg.Table)
+
+		if err := api.writeLogWriter.WriteShard(ctx, qtid, partitionNum, shardNum, version, msg); err != nil {
+			err1 = errors.Wrap(err, "writing import-roaring-shard to writelogger")
+			return err1
+		}
+	}
+
 	return nil
 }
 
@@ -1549,7 +1795,64 @@ func (api *API) ImportValue(ctx context.Context, qcx *Qcx, req *ImportValueReque
 	if err != nil {
 		return errors.Wrap(err, "setting up import options")
 	}
-	return api.ImportValueWithTx(ctx, qcx, req, options)
+
+	/////////////////////////////////////////////////////////////////////////////
+	// We build the ImportValueMessage here BEFORE the call to
+	// api.ImportValueWithTx() because we don't trust that req doesn't get
+	// changed out from under us. See the similar comment in the API.Import()
+	// method above.
+	var partition int
+	var msg *computer.ImportValueMessage
+	if api.isComputeNode && !options.suppressLog {
+		partition = disco.ShardToShardPartition(req.Index, req.Shard, disco.DefaultPartitionN)
+		msg = &computer.ImportValueMessage{
+			Table:           req.Index,
+			Field:           req.Field,
+			Partition:       partition,
+			Shard:           req.Shard,
+			ColumnIDs:       make([]uint64, len(req.ColumnIDs)),
+			ColumnKeys:      make([]string, len(req.ColumnKeys)),
+			Values:          make([]int64, len(req.Values)),
+			FloatValues:     make([]float64, len(req.FloatValues)),
+			TimestampValues: make([]time.Time, len(req.TimestampValues)),
+			StringValues:    make([]string, len(req.StringValues)),
+			Clear:           req.Clear,
+
+			IgnoreKeyCheck: options.IgnoreKeyCheck,
+			Presorted:      options.Presorted,
+		}
+		copy(msg.ColumnIDs, req.ColumnIDs)
+		copy(msg.ColumnKeys, req.ColumnKeys)
+		copy(msg.Values, req.Values)
+		copy(msg.FloatValues, req.FloatValues)
+		copy(msg.TimestampValues, req.TimestampValues)
+		copy(msg.StringValues, req.StringValues)
+	}
+	/////////////////////////////////////////////////////////////////////////////
+
+	if err := api.ImportValueWithTx(ctx, qcx, req, options); err != nil {
+		return errors.Wrap(err, "importing value with tx")
+	}
+
+	if api.isComputeNode && !options.suppressLog {
+		// Get the current version for shard.
+		version, err := api.getOrCreateShardVersion(ctx, req.Index, req.Shard)
+		if err != nil {
+			return errors.Wrap(err, "get or creating shard version")
+		}
+
+		tkey := dax.TableKey(req.Index)
+		qtid := tkey.QualifiedTableID()
+		partitionNum := dax.PartitionNum(partition)
+		shardNum := dax.ShardNum(req.Shard)
+
+		// Write the request to the write logger.
+		if err := api.writeLogWriter.WriteShard(ctx, qtid, partitionNum, shardNum, version, msg); err != nil {
+			return errors.Wrap(err, "writing shard to write logger")
+		}
+	}
+
+	return nil
 }
 
 // ImportValueWithTx bulk imports values into a particular field.
@@ -1570,19 +1873,24 @@ func (api *API) ImportValueWithTx(ctx context.Context, qcx *Qcx, req *ImportValu
 		return nil
 	}
 
+	api.server.logger.Debugf("ImportValueWithTx: %v %v %v", req.Index, req.Field, req.Shard)
 	idx, field, err := api.indexField(req.Index, req.Field, req.Shard)
 	if err != nil {
 		return errors.Wrap(err, fmt.Sprintf("getting index '%v' and field '%v'; shard=%v", req.Index, req.Field, req.Shard))
 	}
 
+	// This node only handles the shard(s) that it owns.
+	if api.isComputeNode {
+		directive := api.holder.Directive()
+		if !shardInShards(dax.ShardNum(req.Shard), directive.ComputeShards(dax.TableKey(idx.Name()))) {
+			return errors.Errorf("import request shard is not supported (value with tx): %d", req.Shard)
+		}
+	}
+
 	if err := req.ValidateWithTimestamp(idx.CreatedAt(), field.CreatedAt()); err != nil {
 		return errors.Wrap(err, "validating import value request")
 	}
 
-	idx, field, err = api.indexField(req.Index, req.Field, req.Shard)
-	if err != nil {
-		return errors.Wrap(err, "getting index and field")
-	}
 	span.LogKV(
 		"index", req.Index,
 		"field", req.Field)
@@ -1821,8 +2129,6 @@ func (api *API) validateShardOwnership(indexName string, shard uint64) error {
 }
 
 func (api *API) indexField(indexName string, fieldName string, shard uint64) (*Index, *Field, error) {
-	api.server.logger.Debugf("importing: %v %v %v", indexName, fieldName, shard)
-
 	// Find the Index.
 	index := api.holder.Index(indexName)
 	if index == nil {
@@ -2270,7 +2576,7 @@ func (api *API) TranslateFieldDB(ctx context.Context, indexName, fieldName strin
 	return err
 }
 
-// RestoreShard
+// RestoreShard is used by the restore tool to restore previously backed up data. This call is specific to RBF data for a shard.
 func (api *API) RestoreShard(ctx context.Context, indexName string, shard uint64, rd io.Reader) error {
 	snap := api.cluster.NewSnapshot()
 	if !snap.OwnsShard(api.server.nodeID, indexName, shard) {
@@ -2760,6 +3066,144 @@ func (api *API) RBFDebugInfo() map[string]*rbf.DebugInfo {
 	return infos
 }
 
+func (api *API) Directive(ctx context.Context, d *dax.Directive) error {
+	return api.ApplyDirective(ctx, d)
+}
+
+// SnapshotShardData triggers the node to perform a shard snapshot based on the
+// provided SnapshotShardDataRequest.
+func (api *API) SnapshotShardData(ctx context.Context, req *dax.SnapshotShardDataRequest) error {
+	qtid := req.TableKey.QualifiedTableID()
+
+	// Confirm that this node is currently responsible for table/shard/fromVersion.
+	var version int
+	if v, ok, err := api.holder.versionStore.ShardVersion(ctx, qtid, req.ShardNum); err != nil {
+		return err
+	} else if !ok {
+		return errors.Errorf("shard not managed by this node: %s, %d", req.TableKey, req.ShardNum)
+	} else if v != req.FromVersion {
+		return errors.Errorf("shard managed by this node is at version: %d, not: %d", v, req.FromVersion)
+	} else {
+		version = v
+	}
+
+	partition := disco.ShardToShardPartition(string(req.TableKey), uint64(req.ShardNum), disco.DefaultPartitionN)
+	partitionNum := dax.PartitionNum(partition)
+
+	// Create the snapshot for the current version.
+	rc, err := api.IndexShardSnapshot(ctx, string(req.TableKey), uint64(req.ShardNum))
+	if err != nil {
+		return errors.Wrap(err, "getting index/shard readcloser")
+	}
+
+	// The following closes rc, the ReadCloser.
+	if err := api.snapshotReadWriter.WriteShardData(ctx, qtid, partitionNum, req.ShardNum, version, rc); err != nil {
+		return errors.Wrap(err, "snapshotting shard data")
+	}
+
+	// Increment the version of the shard managed by this node.
+	if err := api.holder.versionStore.AddShards(ctx, qtid,
+		dax.NewShard(req.ShardNum, req.ToVersion),
+	); err != nil {
+		return errors.Wrap(err, "incrementing shard version locally")
+	}
+
+	// Update the cached directive on the holder.
+	api.holder.SetDirective(&req.Directive)
+
+	// Finally, delete the log file for the previous version.
+	return api.writeLogWriter.DeleteShard(ctx, qtid, partitionNum, req.ShardNum, req.FromVersion)
+}
+
+// SnapshotTableKeys triggers the node to perform a table keys snapshot based on
+// the provided SnapshotTableKeysRequest.
+func (api *API) SnapshotTableKeys(ctx context.Context, req *dax.SnapshotTableKeysRequest) error {
+	// If the index is not keyed, no-op on snapshotting its keys.
+	if idx, err := api.Index(ctx, string(req.TableKey)); err != nil {
+		return newNotFoundError(ErrIndexNotFound, string(req.TableKey))
+	} else if !idx.Keys() {
+		return nil
+	}
+
+	qtid := req.TableKey.QualifiedTableID()
+
+	// Confirm that this node is currently responsible for table/partition/fromVersion.
+	var version int
+	if v, ok, err := api.holder.versionStore.PartitionVersion(ctx, qtid, req.PartitionNum); err != nil {
+		return err
+	} else if !ok {
+		return errors.Errorf("partition not managed by this node: %s, %d", req.TableKey, req.PartitionNum)
+	} else if v != req.FromVersion {
+		return errors.Errorf("partition managed by this node is at version: %d, not: %d", v, req.FromVersion)
+	} else {
+		version = v
+	}
+
+	// Create the snapshot for the current version.
+	wrTo, err := api.TranslateData(ctx, string(req.TableKey), int(req.PartitionNum))
+	if err != nil {
+		return errors.Wrapf(err, "getting index/partition writeto: %s/%d", req.TableKey, req.PartitionNum)
+	}
+
+	if err := api.snapshotReadWriter.WriteTableKeys(ctx, qtid, req.PartitionNum, version, wrTo); err != nil {
+		return errors.Wrap(err, "snapshotting table keys")
+	}
+
+	// Increment the version of the partition managed by this node.
+	if err := api.holder.versionStore.AddPartitions(ctx, qtid,
+		dax.NewPartition(req.PartitionNum, req.ToVersion),
+	); err != nil {
+		return errors.Wrap(err, "incrementing partition version locally")
+	}
+
+	// Update the cached directive on the holder.
+	api.holder.SetDirective(&req.Directive)
+
+	// Finally, delete the log file for the previous version.
+	return api.writeLogWriter.DeleteTableKeys(ctx, qtid, req.PartitionNum, req.FromVersion)
+}
+
+// SnapshotFieldKeys triggers the node to perform a field keys snapshot based on
+// the provided SnapshotFieldKeysRequest.
+func (api *API) SnapshotFieldKeys(ctx context.Context, req *dax.SnapshotFieldKeysRequest) error {
+	qtid := req.TableKey.QualifiedTableID()
+
+	// Confirm that this node is currently responsible for table/field/fromVersion.
+	var version int
+	if v, ok, err := api.holder.versionStore.FieldVersion(ctx, qtid, req.Field); err != nil {
+		return err
+	} else if !ok {
+		return errors.Errorf("field not managed by this node: %s, %s", req.TableKey, req.Field)
+	} else if v != req.FromVersion {
+		return errors.Errorf("field managed by this node is at version: %d, not: %d", v, req.FromVersion)
+	} else {
+		version = v
+	}
+
+	// Create the snapshot for the current version.
+	wrTo, err := api.FieldTranslateData(ctx, string(req.TableKey), string(req.Field))
+	if err != nil {
+		return errors.Wrap(err, "getting index/field writeto")
+	}
+
+	if err := api.snapshotReadWriter.WriteFieldKeys(ctx, qtid, req.Field, version, wrTo); err != nil {
+		return errors.Wrap(err, "snapshotting field keys")
+	}
+
+	// Increment the version of the field managed by this node.
+	if err := api.holder.versionStore.AddFields(ctx, qtid,
+		dax.NewFieldVersion(req.Field, req.ToVersion),
+	); err != nil {
+		return errors.Wrap(err, "incrementing field version locally")
+	}
+
+	// Update the cached directive on the holder.
+	api.holder.SetDirective(&req.Directive)
+
+	// Finally, delete the log file for the previous version.
+	return api.writeLogWriter.DeleteFieldKeys(ctx, qtid, req.Field, req.FromVersion)
+}
+
 type serverInfo struct {
 	ShardWidth       uint64 `json:"shardWidth"`
 	ReplicaN         int    `json:"replicaN"`
@@ -2880,19 +3324,39 @@ var methodsNormal = map[apiMethod]struct{}{
 	apiMutexCheck:           {},
 }
 
+func shardInShards(i dax.ShardNum, s dax.Shards) bool {
+	for _, o := range s {
+		if i == o.Num {
+			return true
+		}
+	}
+	return false
+}
+
 // SchemaAPI is a subset of the API methods which have to do with schema. This
 // interface was introduced in order to remove, from the sql3 package, the
 // pointer to API, and instead use this interface. In the current FeatureBase,
-// this interface can be implemented directly with API. But in an implementation
+// this interface can be implemented directly with API (well, not directly, but
-// for DAX, for example, we might want something else servicing the
+// with FeatureBaseSchemaAPI, which is a wrapper around API). But in an
-// schema-related calls to the SchemaAPI.
+// implementation for DAX, for example, we might want something else servicing
+// the schema-related calls to the SchemaAPI.
 type SchemaAPI interface {
 	CreateIndexAndFields(ctx context.Context, indexName string, options IndexOptions, fields []CreateFieldObj) error
 	CreateField(ctx context.Context, indexName string, fieldName string, opts ...FieldOption) (*Field, error)
 	DeleteField(ctx context.Context, indexName string, fieldName string) error
 	DeleteIndex(ctx context.Context, indexName string) error
-	IndexInfo(ctx context.Context, indexName string) (*IndexInfo, error)
+
+	// Schema returns the list of tables and fields. While it might make sense
+	// to have this as part of the SchemaInfoAPI interface instead of here, it's
+	// never used by consumers of that interface.
 	Schema(ctx context.Context, withViews bool) ([]*IndexInfo, error)
+
+	SchemaInfoAPI
+}
+
+type SchemaInfoAPI interface {
+	IndexInfo(ctx context.Context, indexName string) (*IndexInfo, error)
+	FieldInfo(ctx context.Context, indexName, fieldName string) (*FieldInfo, error)
 }
 
 type ClusterNode struct {
@@ -2936,6 +3400,28 @@ type QueryAPI interface {
 	Query(ctx context.Context, req *QueryRequest) (QueryResponse, error)
 }
 
+// Ensure type implements interface.
+var _ ComputeAPI = (*NopComputeAPI)(nil)
+
+// NopComputeAPI is a no-op implementation of the ComputeAPI interface.
+type NopComputeAPI struct{}
+
+func NewNopComputeAPI() *NopComputeAPI {
+	return &NopComputeAPI{}
+}
+
+func (c *NopComputeAPI) Import(ctx context.Context, qcx *Qcx, req *ImportRequest, opts ...ImportOption) error {
+	return nil
+}
+func (c *NopComputeAPI) ImportValue(ctx context.Context, qcx *Qcx, req *ImportValueRequest, opts ...ImportOption) error {
+	return nil
+}
+
+func (c *NopComputeAPI) Txf() *TxFactory { return nil }
+
+// Ensure type implements interface.
+var _ SchemaAPI = (*FeatureBaseSchemaAPI)(nil)
+
 // FeatureBaseSchemaAPI is a wrapper around pilosa.API. It implements the
 // SchemaAPI interface with methods which are not a part of pilosa.API.
 type FeatureBaseSchemaAPI struct {

api_directive.go

@@ -0,0 +1,983 @@
+// Copyright 2021 Molecula Corp. All rights reserved.
+package pilosa
+
+import (
+	"context"
+	"io"
+	"log"
+	"sync"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/computer"
+	"github.com/molecula/featurebase/v3/disco"
+	"github.com/pkg/errors"
+)
+
+// ApplyDirective applies a Directive received, from the Controller, at the
+// /directive endpoint.
+func (api *API) ApplyDirective(ctx context.Context, d *dax.Directive) error {
+	// Get the current directive for comparison.
+	previousDirective := api.holder.Directive()
+
+	// Check that incoming version is newer.
+	// Note: 0 is an invalid Directive version. This decision was made because
+	// previousDirective is not a pointer to a directive, but a concrete
+	// Directive. Which means we can't check for nil, and by default it has a
+	// version of 0. So in order to ensure the version has increased, we need to
+	// require that incoming directive versions are greater than 0.
+	if d.Version == 0 {
+		return errors.Errorf("directive version cannot be 0")
+	} else if previousDirective.Version >= d.Version {
+		return errors.Errorf("directive version mismatch, got %d, but already have %d", d.Version, previousDirective.Version)
+	}
+
+	// Handle the operations based on the directive method.
+	switch d.Method {
+	case dax.DirectiveMethodDiff:
+		// pass: normal operation
+
+	case dax.DirectiveMethodReset:
+		// Delete all tables.
+		if err := api.deleteAllIndexes(ctx); err != nil {
+			return errors.Wrap(err, "deleting all indexes")
+		}
+
+		// Set previousDirective to empty so the diff handles everything as new.
+		previousDirective = dax.Directive{}
+
+	case dax.DirectiveMethodSnapshot:
+		// TODO(tlt): this was the existing logic, but we should really diff the
+		// directive and ensure that overwriting the value in the cache doesn't
+		// have a negative effect.
+		api.holder.SetDirective(d)
+		return nil
+
+	default:
+		return errors.Errorf("invalid directive method: %s", d.Method)
+	}
+
+	// Cache this directive as the latest applied. There is functionality within
+	// the "enactDirective" stage of ApplyDirective which validates against this
+	// cached Directive, so it's important that it be set before calling
+	// enactDirective(). An example: when loading partition data from the
+	// WriteLogger, there are validations to ensure that the partition being
+	// loaded is meant to be handled by this node; that validation is done
+	// against the cached Directive.
+	// TODO(tlt): despite what this comment says, this logic is not sound; we
+	// shouldn't be setting the directive until enactiveDirective() succeeds.
+	api.holder.SetDirective(d)
+
+	return api.enactDirective(ctx, &previousDirective, d)
+}
+
+// deleteAllIndexes deletes all indexes handled by this node.
+func (api *API) deleteAllIndexes(ctx context.Context) error {
+	indexes, err := api.Schema(ctx, false)
+	if err != nil {
+		return errors.Wrap(err, "getting schema")
+	}
+
+	for i := range indexes {
+		if err := api.DeleteIndex(ctx, indexes[i].Name); err != nil {
+			return errors.Wrapf(err, "deleting index: %s", indexes[i].Name)
+		}
+	}
+
+	return nil
+}
+
+// directiveJobType allows us to switch on jobType in the directiveWorker in
+// order to use a single worker pool for all job types (as opposed to having a
+// separate worker pool for each job type).
+type directiveJobType interface {
+	// We have this method just to prevent *any* struct from implementing this
+	// interface automatically. But, interestingly enough, we don't actually
+	// have to have this method on the implementation because we embed the
+	// interface.
+	isJobType() bool
+}
+
+type directiveJobTableKeys struct {
+	directiveJobType
+	idx       *Index
+	tkey      dax.TableKey
+	partition dax.Partition
+}
+
+type directiveJobFieldKeys struct {
+	directiveJobType
+	tkey  dax.TableKey
+	field dax.FieldVersion
+}
+
+type directiveJobShards struct {
+	directiveJobType
+	tkey  dax.TableKey
+	shard dax.Shard
+}
+
+// directiveWorker is a worker in a worker pool which handles portions of a
+// directive. Multiple instances of directiveWorker run in goroutines in order
+// to load data from snapshotter and writelogger concurrently. Note: unlike the
+// api.ingestWorkerPool, of which one pool is always running, the
+// directiveWorker pool is only running during the life of the
+// api.ApplyDirective call. Technically, this means that multiple
+// directiveWorker pools could be active at the same time, but we should never
+// be running more than once instance of ApplyDirective concurrently.
+func (api *API) directiveWorker(ctx context.Context, jobs <-chan directiveJobType, errs chan<- error) {
+	for j := range jobs {
+		switch job := j.(type) {
+		case directiveJobTableKeys:
+			if err := api.loadTableKeys(ctx, job.idx, job.tkey, job.partition); err != nil {
+				errs <- errors.Wrapf(err, "loading table keys: %s, %s", job.tkey, job.partition)
+			}
+		case directiveJobFieldKeys:
+			if err := api.loadFieldKeys(ctx, job.tkey, job.field); err != nil {
+				errs <- errors.Wrapf(err, "loading field keys: %s, %s", job.tkey, job.field)
+			}
+		case directiveJobShards:
+			if err := api.loadShard(ctx, job.tkey, job.shard); err != nil {
+				errs <- errors.Wrapf(err, "loading shard: %s, %s", job.tkey, job.shard)
+			}
+		default:
+			errs <- errors.Errorf("unsupported job type: %T %[1]v", job)
+		}
+
+		select {
+		case <-ctx.Done():
+			return
+		default:
+			// continue pulling jobs off the channel
+		}
+	}
+}
+
+func (api *API) enactDirective(ctx context.Context, fromD, toD *dax.Directive) error {
+	// enactTables is called before the jobs that run in the worker pool because
+	// it probably makes sense to apply the schema before trying to load data
+	// concurrently.
+	if err := api.enactTables(ctx, fromD, toD); err != nil {
+		return errors.Wrap(err, "enactTables")
+	}
+
+	// The following types use a shared pool of workers to run each
+	// directiveJobType.
+
+	var wg sync.WaitGroup
+
+	// open job channel
+	jobs := make(chan directiveJobType, api.directiveWorkerPoolSize)
+	errs := make(chan error)
+	done := make(chan struct{})
+
+	// Spin up n workers in goroutines that pull jobs from the jobs channel.
+	for i := 0; i < api.directiveWorkerPoolSize; i++ {
+		wg.Add(1)
+		go func() {
+			api.directiveWorker(ctx, jobs, errs)
+			defer wg.Done()
+		}()
+	}
+
+	// Wait for the WaitGroup counter to reach 0. When it has, indicate that
+	// we're done processing all jobs by closing the done channel.
+	go func() {
+		wg.Wait()
+		close(done)
+	}()
+
+	// Run through all the "enact" methods. These push jobs onto the jobs
+	// channel. Once all the jobs have been queued to the channel, we close the
+	// jobs channel. This allows the directiveWorkers to exit out of the
+	// function, which will then decrement the WaitGroup counter.
+	go func() {
+		api.pushJobsTableKeys(ctx, jobs, fromD, toD)
+		api.pushJobsFieldKeys(ctx, jobs, fromD, toD)
+		api.pushJobsShards(ctx, jobs, fromD, toD)
+		close(jobs)
+	}()
+
+	// Keep running until we get an error or until the done channel is closed.
+	// Note: the code is written such that only non-nil errors are pushed to the
+	// errs channel.
+	for {
+		select {
+		case err := <-errs:
+			return err
+		case <-done:
+			return nil
+		}
+	}
+}
+
+func (api *API) enactTables(ctx context.Context, fromD, toD *dax.Directive) error {
+	currentIndexes := api.holder.Indexes()
+
+	// Make a list of indexes that currently exist (from).
+	from := make(dax.TableKeys, 0, len(currentIndexes))
+	for _, idx := range currentIndexes {
+		qtid, err := dax.QualifiedTableIDFromKey(idx.Name())
+		if err != nil {
+			return errors.Wrap(err, "converting index name to qualified table id")
+		}
+		from = append(from, qtid.Key())
+	}
+
+	// TODO sanity check holder against fromD. We're getting existing
+	// indexes from holder, but in theory fromD should be
+	// identical. If we have an error in our directive-caching logic
+	// (it has happened before (just now, in fact!) and we'd be
+	// foolish to think it won't happen again), or we have schema
+	// mutations that are not going through the directive path, we
+	// could potentially catch them here.
+
+	// Make a list of tables that are in the directive (to) along with a map of
+	// tableKey to table (m).
+	m := make(map[dax.TableKey]*dax.QualifiedTable, len(toD.Tables))
+	to := make(dax.TableKeys, 0, len(toD.Tables))
+	for _, t := range toD.Tables {
+		m[t.Key()] = t
+		to = append(to, t.Key())
+	}
+
+	sc := newSliceComparer(from, to)
+
+	// Remove all indexes that are no longer part of the directive.
+	for _, tkey := range sc.removed() {
+		idx := string(tkey)
+		if err := api.holder.deleteIndex(idx); err != nil {
+			return errors.Wrapf(err, "deleting index: %s", tkey)
+		}
+	}
+
+	// Put partitions into a map by table.
+	partitionMap := toD.TranslatePartitionsMap()
+
+	// Add all indexes that weren't previously (but now are) a part of the
+	// directive.
+	for _, tkey := range sc.added() {
+		if qtbl, found := m[tkey]; !found {
+			return errors.Errorf("table '%s' was not in map", tkey)
+		} else if err := api.createTableAndFields(qtbl, partitionMap[tkey]); err != nil {
+			return err
+		}
+	}
+
+	// Check fields on all indexes present in both from and to.
+	for _, tkey := range sc.same() {
+		if err := api.enactFieldsForTable(ctx, tkey, fromD, toD); err != nil {
+			return errors.Wrapf(err, "enacting fields for table: '%s'", tkey)
+		}
+	}
+
+	return nil
+}
+
+func (api *API) enactFieldsForTable(ctx context.Context, tkey dax.TableKey, fromD, toD *dax.Directive) error {
+	qtid := tkey.QualifiedTableID()
+
+	fromT, err := fromD.Table(qtid)
+	if err != nil {
+		return errors.Wrap(err, "getting from table")
+	}
+	toT, err := toD.Table(qtid)
+	if err != nil {
+		return errors.Wrap(err, "getting to table")
+	}
+
+	// Get the index for tkey.
+	idx := api.holder.Index(string(tkey))
+	if idx == nil {
+		return errors.Errorf("index not found: %s", tkey)
+	}
+
+	sc := newSliceComparer(fromT.FieldNames(), toT.FieldNames())
+
+	// Add fields new to toT.
+	for _, fldName := range sc.added() {
+		if field, found := toT.Field(fldName); !found {
+			return dax.NewErrFieldDoesNotExist(fldName)
+		} else if err := createField(idx, field); err != nil {
+			return errors.Wrapf(err, "creating field: %s/%s", tkey, fldName)
+		}
+	}
+
+	// Remove fields which don't exist in toT.
+	for _, fldName := range sc.removed() {
+		if err := api.DeleteField(ctx, string(tkey), string(fldName)); err != nil {
+			return errors.Wrapf(err, "deleting field: %s/%s", tkey, fldName)
+		}
+	}
+
+	// // Update any field options which have changed for existing fields.
+	// for _, fldName := range sc.same() {
+	// 	// handle changed field options??
+	// }
+
+	return nil
+}
+
+func (api *API) pushJobsTableKeys(ctx context.Context, jobs chan<- directiveJobType, fromD, toD *dax.Directive) {
+	toPartitionsMap := toD.TranslatePartitionsMap()
+
+	// Get the diff between from/to directive.partitions.
+	partComp := newPartitionsComparer(fromD.TranslatePartitionsMap(), toPartitionsMap)
+
+	// Loop over the partition map and load from WriteLogger.
+	for tkey, partitions := range partComp.added() {
+		// Get index in order to find the translate stores (by partition) for
+		// the table.
+		idx := api.holder.Index(string(tkey))
+		if idx == nil {
+			log.Printf("index not found in holder: %s", tkey)
+			continue
+		}
+
+		// Update the cached version of translate partitions that we keep on the
+		// Index.
+		idx.SetTranslatePartitions(toPartitionsMap[tkey])
+
+		for _, partition := range partitions {
+			jobs <- directiveJobTableKeys{
+				idx:       idx,
+				tkey:      tkey,
+				partition: partition,
+			}
+		}
+	}
+}
+
+func (api *API) loadTableKeys(ctx context.Context, idx *Index, tkey dax.TableKey, partition dax.Partition) error {
+	qtid := tkey.QualifiedTableID()
+
+	// Load the previous snapshot. Version 0 doesn't have a snapshot
+	// file; it only has log entries.
+	if partition.Version > 0 {
+		// Load partition snapshot: version - 1
+		previousVersion := partition.Version - 1
+		rc, err := api.snapshotReadWriter.ReadTableKeys(ctx, qtid, partition.Num, previousVersion)
+		if err != nil {
+			return errors.Wrap(err, "reading table keys snapshot")
+		}
+		defer rc.Close()
+
+		if err := api.TranslateIndexDB(ctx, string(tkey), int(partition.Num), rc); err != nil {
+			return errors.Wrap(err, "restoring table keys")
+
+		}
+	}
+
+	if err := func() error {
+		store := idx.TranslateStore(int(partition.Num))
+
+		reader := api.writeLogReader.TableKeyReader(ctx, qtid, partition.Num, partition.Version)
+		if err := reader.Open(); err != nil {
+			// TODO: this log can be confusing because on a create
+			// table, there is no log file yet, so an error is expected.
+			// Instead of swallowing this error, we need to check the
+			// error code and handle it differently. This means the
+			// writelogger will need to return an error indicating that
+			// the log file does not exist, but that that is expected.
+			// log.Printf("could not open log file for table: %s, partition: %d: version: %d, err: %s", table, partition.Num, partition.Version, err)
+			return nil
+		}
+		defer reader.Close()
+
+		for msg, err := reader.Read(); err != io.EOF; msg, err = reader.Read() {
+			if err != nil {
+				return errors.Wrap(err, "reading from log reader")
+			}
+			for key, id := range msg.StringToID {
+				if err := store.ForceSet(id, key); err != nil {
+					return errors.Wrapf(err, "forcing set id, key: %d, %s", id, key)
+				}
+			}
+		}
+
+		return nil
+	}(); err != nil {
+		return err
+	}
+
+	// Set the table/partition/version in the holder.
+	if err := api.holder.versionStore.AddPartitions(ctx, qtid, partition); err != nil {
+		return errors.Wrap(err, "adding partition to sharder")
+	}
+
+	return nil
+}
+
+func (api *API) pushJobsFieldKeys(ctx context.Context, jobs chan<- directiveJobType, fromD, toD *dax.Directive) {
+	// Get the diff between from/to directive.fields.
+	fieldComp := newFieldsComparer(fromD.TranslateFieldsMap(), toD.TranslateFieldsMap())
+
+	// Loop over the field map and load from WriteLogger.
+	for tkey, fields := range fieldComp.added() {
+		for _, field := range fields {
+			jobs <- directiveJobFieldKeys{
+				tkey:  tkey,
+				field: field,
+			}
+		}
+	}
+}
+
+func (api *API) loadFieldKeys(ctx context.Context, tkey dax.TableKey, field dax.FieldVersion) error {
+	qtid := tkey.QualifiedTableID()
+
+	// Load the previous snapshot. Version 0 doesn't have a snapshot
+	// file; it only has log entries.
+	if field.Version > 0 {
+		// Load field snapshot: version - 1
+		previousVersion := field.Version - 1
+		rc, err := api.snapshotReadWriter.ReadFieldKeys(ctx, qtid, field.Name, previousVersion)
+		if err != nil {
+			return errors.Wrap(err, "reading field keys snapshot")
+		}
+		defer rc.Close()
+
+		if err := api.TranslateFieldDB(ctx, string(tkey), string(field.Name), rc); err != nil {
+			return errors.Wrap(err, "restoring field keys")
+		}
+	}
+
+	if err := func() error {
+		// Get field in order to find the translate store.
+		fld := api.holder.Field(string(tkey), string(field.Name))
+		if fld == nil {
+			log.Printf("field not found in holder: %s", field.Name)
+			return nil
+		}
+		store := fld.TranslateStore()
+
+		reader := api.writeLogReader.FieldKeyReader(ctx, qtid, field.Name, field.Version)
+		if err := reader.Open(); err != nil {
+			// TODO: this log can be confusing because on a create
+			// table, there is no log file yet, so an error is expected.
+			// Instead of swallowing this error, we need to check the
+			// error code and handle it differently. This means the
+			// writelogger will need to return an error indicating that
+			// the log file does not exist, but that that is expected.
+			// log.Printf("could not open log file for table: %s, field: %s: version: %d, err: %s", table, field.Name, field.Version, err)
+			return nil
+		}
+		defer reader.Close()
+
+		for msg, err := reader.Read(); err != io.EOF; msg, err = reader.Read() {
+			if err != nil {
+				return errors.Wrap(err, "reading from log reader")
+			}
+			for key, id := range msg.StringToID {
+				if err := store.ForceSet(id, key); err != nil {
+					return errors.Wrapf(err, "forcing set id, key: %d, %s", id, key)
+				}
+			}
+		}
+
+		return nil
+	}(); err != nil {
+		return err
+	}
+
+	// Set the table/field/version in the holder.
+	if err := api.holder.versionStore.AddFields(ctx, qtid, field); err != nil {
+		return errors.Wrap(err, "adding field to sharder")
+	}
+
+	return nil
+}
+
+func (api *API) pushJobsShards(ctx context.Context, jobs chan<- directiveJobType, fromD, toD *dax.Directive) {
+	// Put shards into a map by table.
+	shardMap := toD.ComputeShardsMap()
+
+	// Get the diff between from/to directive shards.
+	shardComp := newShardsComparer(fromD.ComputeShardsMap(), shardMap)
+
+	// Loop over the shard map and load from WriteLogger.
+	for tkey, shards := range shardComp.added() {
+		for _, shard := range shards {
+			jobs <- directiveJobShards{
+				tkey:  tkey,
+				shard: shard,
+			}
+		}
+	}
+}
+
+func (api *API) loadShard(ctx context.Context, tkey dax.TableKey, shard dax.Shard) error {
+	qtid := tkey.QualifiedTableID()
+
+	partition := disco.ShardToShardPartition(string(tkey), uint64(shard.Num), disco.DefaultPartitionN)
+	partitionNum := dax.PartitionNum(partition)
+
+	// Load the previous snapshot. Version 0 doesn't have a snapshot
+	// file; it only has log entries.
+	if shard.Version > 0 {
+		// Load shard snapshot: version - 1
+		previousVersion := shard.Version - 1
+		rc, err := api.snapshotReadWriter.ReadShardData(ctx, qtid, partitionNum, shard.Num, previousVersion)
+		if err != nil {
+			return errors.Wrap(err, "reading shard data snapshot")
+		}
+
+		if err := api.RestoreShard(ctx, string(tkey), uint64(shard.Num), rc); err != nil {
+			return errors.Wrap(err, "restoring shard data")
+		}
+	}
+
+	// WriteLog reader.
+	if err := func() error {
+		reader := api.writeLogReader.ShardReader(ctx, qtid, partitionNum, shard.Num, shard.Version)
+		if err := reader.Open(); err != nil {
+			// TODO: this log can be confusing because on a create
+			// table, there is no log file yet, so an error is expected.
+			// Instead of swallowing this error, we need to check the
+			// error code and handle it differently. This means the
+			// writelogger will need to return an error indicating that
+			// the log file does not exist, but that that is expected.
+			// log.Printf("could not open log file for table: %s, partition: %d: version: %d, shard: %d, err: %s", table, partition, shard.Version, shard.Num, err)
+			return nil
+		}
+		defer reader.Close()
+
+		for logMsg, err := reader.Read(); err != io.EOF; logMsg, err = reader.Read() {
+			if err != nil {
+				return errors.Wrap(err, "reading from log reader")
+			}
+			switch msg := logMsg.(type) {
+			case *computer.ImportRoaringMessage:
+				req := &ImportRoaringRequest{
+					Clear:           msg.Clear,
+					Action:          msg.Action,
+					Block:           msg.Block,
+					Views:           msg.Views,
+					UpdateExistence: msg.UpdateExistence,
+					SuppressLog:     true,
+				}
+				if err := api.ImportRoaring(ctx, msg.Table, msg.Field, msg.Shard, true, req); err != nil {
+					return errors.Wrapf(err, "import roaring, table: %s, field: %s, shard: %d", msg.Table, msg.Field, msg.Shard)
+				}
+
+			case *computer.ImportMessage:
+				req := &ImportRequest{
+					Index:      msg.Table,
+					Field:      msg.Field,
+					Shard:      msg.Shard,
+					RowIDs:     msg.RowIDs,
+					ColumnIDs:  msg.ColumnIDs,
+					RowKeys:    msg.RowKeys,
+					ColumnKeys: msg.ColumnKeys,
+					Timestamps: msg.Timestamps,
+					Clear:      msg.Clear,
+				}
+
+				qcx := api.Txf().NewQcx()
+				defer qcx.Abort()
+
+				opts := []ImportOption{
+					OptImportOptionsClear(msg.Clear),
+					OptImportOptionsIgnoreKeyCheck(msg.IgnoreKeyCheck),
+					OptImportOptionsPresorted(msg.Presorted),
+					OptImportOptionsSuppressLog(true),
+				}
+				if err := api.Import(ctx, qcx, req, opts...); err != nil {
+					return errors.Wrapf(err, "import, table: %s, field: %s, shard: %d", msg.Table, msg.Field, msg.Shard)
+				}
+
+			case *computer.ImportValueMessage:
+				req := &ImportValueRequest{
+					Index:           msg.Table,
+					Field:           msg.Field,
+					Shard:           msg.Shard,
+					ColumnIDs:       msg.ColumnIDs,
+					ColumnKeys:      msg.ColumnKeys,
+					Values:          msg.Values,
+					FloatValues:     msg.FloatValues,
+					TimestampValues: msg.TimestampValues,
+					StringValues:    msg.StringValues,
+					Clear:           msg.Clear,
+				}
+
+				qcx := api.Txf().NewQcx()
+				defer qcx.Abort()
+
+				opts := []ImportOption{
+					OptImportOptionsClear(msg.Clear),
+					OptImportOptionsIgnoreKeyCheck(msg.IgnoreKeyCheck),
+					OptImportOptionsPresorted(msg.Presorted),
+					OptImportOptionsSuppressLog(true),
+				}
+				if err := api.ImportValue(ctx, qcx, req, opts...); err != nil {
+					return errors.Wrapf(err, "import value, table: %s, field: %s, shard: %d", msg.Table, msg.Field, msg.Shard)
+				}
+			case *computer.ImportRoaringShardMessage:
+				req := &ImportRoaringShardRequest{
+					Remote:      true,
+					Views:       make([]RoaringUpdate, len(msg.Views)),
+					SuppressLog: true,
+				}
+				for i, view := range msg.Views {
+					req.Views[i] = RoaringUpdate{
+						Field:        view.Field,
+						View:         view.View,
+						Clear:        view.Clear,
+						Set:          view.Set,
+						ClearRecords: view.ClearRecords,
+					}
+				}
+				if err := api.ImportRoaringShard(ctx, msg.Table, msg.Shard, req); err != nil {
+					return errors.Wrapf(err, "import roaring shard table: %s, shard: %d", msg.Table, msg.Shard)
+				}
+			}
+		}
+
+		return nil
+	}(); err != nil {
+		return err
+	}
+
+	// Set the table/shard/version in the holder.
+	if err := api.holder.versionStore.AddShards(ctx, qtid, shard); err != nil {
+		return errors.Wrap(err, "adding shard to sharder")
+	}
+
+	return nil
+}
+
+//////////////////////////////////////////////////////////////
+
+// sliceComparer is used to compare the differences between two slices of comparables.
+type sliceComparer[K comparable] struct {
+	from []K
+	to   []K
+}
+
+func newSliceComparer[K comparable](from []K, to []K) *sliceComparer[K] {
+	return &sliceComparer[K]{
+		from: from,
+		to:   to,
+	}
+}
+
+// added returns the items which are present in `to` but not in `from`.
+func (s *sliceComparer[K]) added() []K {
+	return thingsAdded(s.from, s.to)
+}
+
+// removed returns the items which are present in `from` but not in `to`.
+func (s *sliceComparer[K]) removed() []K {
+	return thingsAdded(s.to, s.from)
+}
+
+// same returns the items which are in both `to` and `from`.
+func (s *sliceComparer[K]) same() []K {
+	var same []K
+	for _, fromThing := range s.from {
+		for _, toThing := range s.to {
+			if fromThing == toThing {
+				same = append(same, fromThing)
+				break
+			}
+		}
+	}
+	return same
+}
+
+// thingsAdded returns the comparable things which are present in `to` but not
+// in `from`.
+func thingsAdded[K comparable](from []K, to []K) []K {
+	var added []K
+	for i := range to {
+		var found bool
+		for j := range from {
+			if from[j] == to[i] {
+				found = true
+				break
+			}
+		}
+		if !found {
+			added = append(added, to[i])
+		}
+	}
+	return added
+}
+
+// partitionsComparer is used to compare the differences between two maps of
+// table:[]partition.
+type partitionsComparer struct {
+	from map[dax.TableKey]dax.Partitions
+	to   map[dax.TableKey]dax.Partitions
+}
+
+func newPartitionsComparer(from map[dax.TableKey]dax.Partitions, to map[dax.TableKey]dax.Partitions) *partitionsComparer {
+	return &partitionsComparer{
+		from: from,
+		to:   to,
+	}
+}
+
+// added returns the partitions which are present in `to` but not in `from`. The
+// results remain in the format of a map of table:[]partition.
+func (p *partitionsComparer) added() map[dax.TableKey]dax.Partitions {
+	return partitionsAdded(p.from, p.to)
+}
+
+// removed returns the partitions which are present in `from` but not in `to`.
+// The results remain in the format of a map of table:[]partition.
+func (p *partitionsComparer) removed() map[dax.TableKey]dax.Partitions {
+	return partitionsAdded(p.to, p.from)
+}
+
+// partitionsAdded returns the partitions which are present in `to` but not in `from`.
+func partitionsAdded(from map[dax.TableKey]dax.Partitions, to map[dax.TableKey]dax.Partitions) map[dax.TableKey]dax.Partitions {
+	if from == nil {
+		return to
+	}
+
+	added := make(map[dax.TableKey]dax.Partitions)
+	for tt, tps := range to {
+		fps, found := from[tt]
+		if !found {
+			added[tt] = tps
+			continue
+		}
+
+		addedPartitions := dax.Partitions{}
+		for i := range tps {
+			var found bool
+			for j := range fps {
+				if fps[j] == tps[i] {
+					found = true
+					break
+				}
+			}
+			if !found {
+				addedPartitions = append(addedPartitions, tps[i])
+			}
+		}
+
+		if len(addedPartitions) > 0 {
+			added[tt] = addedPartitions
+		}
+	}
+	return added
+}
+
+// fieldsComparer is used to compare the differences between two maps of
+// table:[]fieldVersion.
+type fieldsComparer struct {
+	from map[dax.TableKey]dax.FieldVersions
+	to   map[dax.TableKey]dax.FieldVersions
+}
+
+func newFieldsComparer(from map[dax.TableKey]dax.FieldVersions, to map[dax.TableKey]dax.FieldVersions) *fieldsComparer {
+	return &fieldsComparer{
+		from: from,
+		to:   to,
+	}
+}
+
+// added returns the fields which are present in `to` but not in `from`. The
+// results remain in the format of a map of table:[]field.
+func (f *fieldsComparer) added() map[dax.TableKey]dax.FieldVersions {
+	return fieldsAdded(f.from, f.to)
+}
+
+// removed returns the fields which are present in `from` but not in `to`.
+// The results remain in the format of a map of table:[]field.
+func (f *fieldsComparer) removed() map[dax.TableKey]dax.FieldVersions {
+	return fieldsAdded(f.to, f.from)
+}
+
+// fieldsAdded returns the fields which are present in `to` but not in `from`.
+func fieldsAdded(from map[dax.TableKey]dax.FieldVersions, to map[dax.TableKey]dax.FieldVersions) map[dax.TableKey]dax.FieldVersions {
+	if from == nil {
+		return to
+	}
+
+	added := make(map[dax.TableKey]dax.FieldVersions)
+	for tt, tps := range to {
+		fps, found := from[tt]
+		if !found {
+			added[tt] = tps
+			continue
+		}
+
+		addedFieldVersions := dax.FieldVersions{}
+		for i := range tps {
+			var found bool
+			for j := range fps {
+				if fps[j] == tps[i] {
+					found = true
+					break
+				}
+			}
+			if !found {
+				addedFieldVersions = append(addedFieldVersions, tps[i])
+			}
+		}
+
+		if len(addedFieldVersions) > 0 {
+			added[tt] = addedFieldVersions
+		}
+	}
+	return added
+}
+
+// shardsComparer is used to compare the differences between two maps of
+// table:[]shardV.
+type shardsComparer struct {
+	from map[dax.TableKey]dax.Shards
+	to   map[dax.TableKey]dax.Shards
+}
+
+func newShardsComparer(from map[dax.TableKey]dax.Shards, to map[dax.TableKey]dax.Shards) *shardsComparer {
+	return &shardsComparer{
+		from: from,
+		to:   to,
+	}
+}
+
+// added returns the shards which are present in `to` but not in `from`. The
+// results remain in the format of a map of table:[]shard.
+func (s *shardsComparer) added() map[dax.TableKey]dax.Shards {
+	return shardsAdded(s.from, s.to)
+}
+
+// removed returns the shards which are present in `from` but not in `to`. The
+// results remain in the format of a map of table:[]shard.
+func (s *shardsComparer) removed() map[dax.TableKey]dax.Shards {
+	return shardsAdded(s.to, s.from)
+}
+
+// shardsAdded returns the shards which are present in `to` but not in `from`.
+func shardsAdded(from map[dax.TableKey]dax.Shards, to map[dax.TableKey]dax.Shards) map[dax.TableKey]dax.Shards {
+	if from == nil {
+		return to
+	}
+
+	added := make(map[dax.TableKey]dax.Shards)
+	for tt, tss := range to {
+		fss, found := from[tt]
+		if !found {
+			added[tt] = tss
+			continue
+		}
+
+		addedShards := dax.Shards{}
+		for i := range tss {
+			var found bool
+			for j := range fss {
+				if fss[j] == tss[i] {
+					found = true
+					break
+				}
+			}
+			if !found {
+				addedShards = append(addedShards, tss[i])
+			}
+		}
+
+		if len(addedShards) > 0 {
+			added[tt] = addedShards
+		}
+	}
+	return added
+}
+
+// createTableAndFields creates the FeatureBase Tables and Fields provided in
+// the dax.Directive format.
+func (api *API) createTableAndFields(tbl *dax.QualifiedTable, partitions dax.Partitions) error {
+	cim := &CreateIndexMessage{
+		Index:     string(tbl.Key()),
+		CreatedAt: 0,
+		Meta: IndexOptions{
+			Keys:           tbl.StringKeys(),
+			TrackExistence: true,
+		},
+	}
+
+	// Create the index in etcd as the system of record.
+	if err := api.holder.persistIndex(context.Background(), cim); err != nil {
+		return errors.Wrap(err, "persisting index")
+	}
+
+	idx, err := api.holder.createIndexWithPartitions(cim, partitions)
+	if err != nil {
+		return errors.Wrapf(err, "adding index: %s", tbl.Name)
+	}
+
+	// Add the fields
+	for _, fld := range tbl.Fields {
+		if fld.IsPrimaryKey() {
+			continue
+		}
+		if err := createField(idx, fld); err != nil {
+			return errors.Wrapf(err, "creating field: %s", fld.Name)
+		}
+	}
+
+	return nil
+}
+
+// createField creates a FeatureBase Field in the provided FeatureBase Index
+// based on the provided field's type.
+//
+// TODO: `time` fields
+func createField(idx *Index, fld *dax.Field) error {
+	// Set the cache type and size (or use default) for those fields which
+	// require them.
+	cacheType := DefaultCacheType
+	cacheSize := uint32(DefaultCacheSize)
+	if fld.Options.CacheType != "" {
+		cacheType = fld.Options.CacheType
+		cacheSize = fld.Options.CacheSize
+	}
+
+	opts := []FieldOption{}
+
+	switch fld.Type {
+	case dax.FieldTypeBool:
+		opts = append(opts,
+			OptFieldTypeBool(),
+		)
+	case dax.FieldTypeDecimal:
+		opts = append(opts,
+			OptFieldTypeDecimal(fld.Options.Scale),
+		)
+	case dax.FieldTypeID:
+		opts = append(opts,
+			OptFieldTypeMutex(cacheType, cacheSize),
+		)
+	case dax.FieldTypeIDSet:
+		opts = append(opts,
+			OptFieldTypeSet(cacheType, cacheSize),
+		)
+	case dax.FieldTypeInt:
+		opts = append(opts,
+			OptFieldTypeInt(fld.Options.Min.ToInt64(0), fld.Options.Max.ToInt64(0)),
+		)
+	case dax.FieldTypeString:
+		opts = append(opts,
+			OptFieldTypeMutex(cacheType, cacheSize),
+			OptFieldKeys(),
+		)
+	case dax.FieldTypeStringSet:
+		opts = append(opts,
+			OptFieldTypeSet(cacheType, cacheSize),
+			OptFieldKeys(),
+		)
+	case dax.FieldTypeTimestamp:
+		opts = append(opts,
+			OptFieldTypeTimestamp(fld.Options.Epoch, fld.Options.TimeUnit),
+		)
+	default:
+		return errors.Errorf("unsupport field type: %s", fld.Type)
+	}
+
+	if _, err := idx.CreateField(string(fld.Name), "", opts...); err != nil {
+		return errors.Wrapf(err, "creating field on index: %s", fld.Name)
+	}
+	return nil
+}

api_directive_internal_test.go

@@ -0,0 +1,25 @@
+package pilosa
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestThingsAddedGeneric(t *testing.T) {
+	from := []string{"a", "b", "c"}
+	to := []string{"b", "c", "d"}
+
+	added := thingsAdded(from, to)
+	assert.Equal(t, added, []string{"d"})
+}
+
+func TestSliceComparer(t *testing.T) {
+	from := []string{"a", "b", "c"}
+	to := []string{"b", "c", "d"}
+
+	sc := newSliceComparer(from, to)
+
+	added := sc.added()
+	assert.Equal(t, added, []string{"d"})
+}

api_directive_test.go

@@ -0,0 +1,97 @@
+package pilosa_test
+
+import (
+	"context"
+	"testing"
+
+	pilosa "github.com/molecula/featurebase/v3"
+	"github.com/molecula/featurebase/v3/dax"
+	daxtest "github.com/molecula/featurebase/v3/dax/test"
+	"github.com/molecula/featurebase/v3/test"
+	"github.com/stretchr/testify/assert"
+)
+
+// Ensure holder can handle an incoming directive.
+func TestAPI_Directive(t *testing.T) {
+
+	c := test.MustRunCluster(t, 1)
+	defer c.Close()
+
+	api := c.GetPrimary().API
+	ctx := context.Background()
+
+	qual := dax.NewTableQualifier("acme", "db1")
+	tbl1 := daxtest.TestQualifiedTableWithID(t, qual, "1", "tbl1", 12, false)
+	tbl2 := daxtest.TestQualifiedTableWithID(t, qual, "2", "tbl2", 12, false)
+	tbl3 := daxtest.TestQualifiedTableWithID(t, qual, "3", "tbl3", 12, false)
+
+	t.Run("Schema", func(t *testing.T) {
+
+		// Empty directive (and empty holder).
+		{
+			d := &dax.Directive{
+				Method:  dax.DirectiveMethodDiff,
+				Version: 1,
+			}
+			err := api.ApplyDirective(ctx, d)
+			assert.NoError(t, err)
+			assertTablesMatch(t, []string{}, api.Holder().Indexes())
+		}
+
+		// Add a new table.
+		{
+			d := &dax.Directive{
+				Method: dax.DirectiveMethodDiff,
+				Tables: []*dax.QualifiedTable{
+					tbl1,
+				},
+				Version: 2,
+			}
+			err := api.ApplyDirective(ctx, d)
+			assert.NoError(t, err)
+			assertTablesMatch(t, []string{"tbl__acme__db1__1"}, api.Holder().Indexes())
+		}
+
+		// Add a new table, and keep the existing table.
+		{
+			d := &dax.Directive{
+				Method: dax.DirectiveMethodDiff,
+				Tables: []*dax.QualifiedTable{
+					tbl1,
+					tbl2,
+				},
+				Version: 3,
+			}
+			err := api.ApplyDirective(ctx, d)
+			assert.NoError(t, err)
+			assertTablesMatch(t, []string{"tbl__acme__db1__1", "tbl__acme__db1__2"}, api.Holder().Indexes())
+		}
+
+		// Add a new table and remove one of the existing tables.
+		{
+			d := &dax.Directive{
+				Method: dax.DirectiveMethodDiff,
+				Tables: []*dax.QualifiedTable{
+					tbl2,
+					tbl3,
+				},
+				Version: 4,
+			}
+			err := api.ApplyDirective(ctx, d)
+			assert.NoError(t, err)
+			assertTablesMatch(t, []string{"tbl__acme__db1__2", "tbl__acme__db1__3"}, api.Holder().Indexes())
+		}
+	})
+}
+
+// assertTablesMatch is a helper function which asserts that the list of index
+// names in `actual` match those provided in `expected`.
+func assertTablesMatch(t *testing.T, expected []string, actual []*pilosa.Index) {
+	t.Helper()
+
+	act := make([]string, len(actual))
+	for i := range actual {
+		act[i] = actual[i].Name()
+	}
+	assert.ElementsMatch(t, expected, act)
+}

batch/Makefile

@@ -7,11 +7,6 @@ GO ?= go
 # we might be running multiple instances of the tests concurrently.
 PROJECT ?= batch
 DOCKER_COMPOSE = docker-compose -p $(PROJECT)
-BRANCH_NAME ?= ""
-
-.pulled:
-	$(DOCKER_COMPOSE) pull
-	touch .pulled
 
 vendor: ../go.mod
 	$(GO) mod vendor
@@ -19,24 +14,19 @@ vendor: ../go.mod
 build-%:
 	$(DOCKER_COMPOSE) build $*
 
-pull-%:
-	$(DOCKER_COMPOSE) pull $*
-
 test-all:
 	$(MAKE) startup
 	$(MAKE) test-run
 	$(MAKE) shutdown
 
-start-all: .pulled build-wait
+start-all: build-wait
-	echo "branch name" ${BRANCH_NAME}
+	$(DOCKER_COMPOSE) up -d featurebase
-	BRANCH_NAME=${BRANCH_NAME} $(DOCKER_COMPOSE) up -d featurebase
 	$(DOCKER_COMPOSE) run -T wait featurebase curl --silent --fail http://featurebase:10101/status
 
 startup: start-all
 
 shutdown:
 	$(DOCKER_COMPOSE) down -v --remove-orphans
-	rm -f .pulled
 
 save-%-logs:
 	$(DOCKER_COMPOSE) logs $* > ./testdata/$(PROJECT)_$*_logs.txt
@@ -48,7 +38,7 @@ test-run-local:
 	$(DOCKER_COMPOSE) build batch-test
 	$(DOCKER_COMPOSE) run -T batch-test go test -mod=vendor -tags=odbc,dynamic $(TCMD)
 
-TPKG ?= ./...
+TPKG ?= ../...
 test-run: vendor
 	$(DOCKER_COMPOSE) build batch-test
-	$(DOCKER_COMPOSE) run -T batch-test bash -c "set -o pipefail; go test -v -mod=vendor -tags=odbc,dynamic $(TPKG) -covermode=atomic -coverpkg=$(TPKG) -json -coverprofile=/testdata/$(PROJECT)_base_coverage.out  | tee /testdata/$(PROJECT)_report.out"
+	$(DOCKER_COMPOSE) run -T batch-test bash -c "set -o pipefail; go test -v -mod=vendor -tags=odbc,dynamic ./... -covermode=atomic -coverpkg=$(TPKG) -json -coverprofile=/testdata/$(PROJECT)_base_coverage.out  | tee /testdata/$(PROJECT)_report.out"

batch/README.md

@@ -22,7 +22,7 @@ In addition to these dependancies, you will need to be added to the molecula [Gi
 
 First start the test environment. This is a docker-compose environment that includes featurebase.
 
-    BRANCH_NAME=master make startup
+    make startup
 
 To build and run the integration tests, run:
 

batch/batch.go

@@ -1,3 +1,4 @@
+// Copyright 2021 Molecula Corp. All rights reserved.
 // Package batch provides tooling to prepare batches of records for ingest.
 package batch
 
@@ -182,6 +183,8 @@ type Batch struct {
 	clearFrags     fragments
 
 	useShardTransactionalEndpoint bool
+
+	mdsHost string
 }
 
 func (b *Batch) Len() int { return len(b.ids) }
@@ -332,6 +335,7 @@ func NewBatch(importer Importer, size int, index *featurebase.IndexInfo, fields
 			return nil, errors.Wrap(err, "applying options")
 		}
 	}
+
 	return b, nil
 }
 

batch/batch_test.go

@@ -1,3 +1,5 @@
+// Copyright 2021 Molecula Corp. All rights reserved.
+
 package batch
 
 import (
@@ -1842,7 +1844,7 @@ func testImportBatchSetsAndClears(t *testing.T, importer Importer, sapi featureb
 
 // testTopNCacheRegression recreates an issue we saw in an IDK test
 // where if a value is completely removed (all bits unset from a row),
-// it didn't get removed from the cache beacuse a full recalculation
+// it didn't get removed from the cache because a full recalculation
 // had no way to clear the cache, it would just reset existing
 // values. We added Clear on the cache interface to fix this.
 func testTopNCacheRegression(t *testing.T, importer Importer, sapi featurebase.SchemaAPI, qapi featurebase.QueryAPI) {

batch/docker-compose.yml

@@ -4,12 +4,13 @@ services:
   featurebase:
     build:
       context: ../.
-      dockerfile: ./Dockerfile
+      dockerfile: ./Dockerfile-clustertests
     environment:
       PILOSA_DATA_DIR: /data
       PILOSA_BIND: 0.0.0.0:10101
       PILOSA_BIND_GRPC: 0.0.0.0:20101
       PILOSA_ADVERTISE: featurebase:10101
+    command: /featurebase -test.run=TestRunMain -test.coverprofile=/testdata/batch_coverage.out server
     volumes:
       - ./testdata:/testdata
 
@@ -21,6 +22,8 @@ services:
       - ./testdata:/testdata
 
   wait:
+    depends_on:
+        - "featurebase"
     build:
       context: .
       dockerfile: Dockerfile-wait

bsi.go

@@ -8,14 +8,14 @@ import (
 	"github.com/featurebasedb/featurebase/v3/roaring"
 )
 
-// bsiData contains BSI-structured data.
+// BSIData contains BSI-structured data.
-type bsiData []*Row
+type BSIData []*Row
 
-// pivotDescending loops over nonzero BSI values in descending order.
+// PivotDescending loops over nonzero BSI values in descending order.
 // For each value, the provided function is called with the value and a slice of the associated columns.
 // If limit or offset are not-nil, they will be applied.
 // Applying a limit or offset may modify the pointed-to value.
-func (bsi bsiData) pivotDescending(filter *Row, branch uint64, limit, offset *uint64, fn func(uint64, ...uint64)) {
+func (bsi BSIData) PivotDescending(filter *Row, branch uint64, limit, offset *uint64, fn func(uint64, ...uint64)) {
 	// This "pivot" algorithm works by treating the BSI data as a tree.
 	// Each branch of this tree corresponds to a power-of-2-sized range of BSI values.
 	// Each range is subdivided into 2 ranges of half size, which form lower branches.
@@ -56,8 +56,8 @@ func (bsi bsiData) pivotDescending(filter *Row, branch uint64, limit, offset *ui
 		upperBranch, lowerBranch := branch|(1<<uint(len(bsi)-1)), branch
 		splitBit := bsi[len(bsi)-1]
 		lowerBits := bsi[:len(bsi)-1]
-		lowerBits.pivotDescending(filter.Intersect(splitBit), upperBranch, limit, offset, fn)
+		lowerBits.PivotDescending(filter.Intersect(splitBit), upperBranch, limit, offset, fn)
-		lowerBits.pivotDescending(filter.Difference(splitBit), lowerBranch, limit, offset, fn)
+		lowerBits.PivotDescending(filter.Difference(splitBit), lowerBranch, limit, offset, fn)
 	}
 }
 
@@ -69,7 +69,7 @@ func (bsi bsiData) pivotDescending(filter *Row, branch uint64, limit, offset *ui
 // - TopN on int
 func (bsi bsiData) distribution(filter *Row) bsiData {
 	var dist bsiData
-	bsi.pivotDescending(filter, 0, nil, nil, func(count uint64, values ...uint64) {
+	bsi.PivotDescending(filter, 0, nil, nil, func(count uint64, values ...uint64) {
 		dist.insert(count, uint64(len(values)))
 	})
 	return dist
@@ -78,20 +78,20 @@ func (bsi bsiData) distribution(filter *Row) bsiData {
 
 var placeholderBitmap = roaring.NewBitmap()
 
-// addBSI adds two BSI bitmaps together.
+// AddBSI adds two BSI bitmaps together.
 // It does not handle sign and has no concept of overflow.
-func addBSI(x, y bsiData) bsiData {
+func AddBSI(x, y BSIData) BSIData {
 	// Accumulate row segments.
-	segments := make([][]rowSegment, len(x)+len(y))
+	segments := make([][]RowSegment, len(x)+len(y))
 	xsegs, ysegs := segments[:len(x)], segments[len(x):]
 	for i, r := range x {
-		xsegs[i] = r.segments
+		xsegs[i] = r.Segments
 	}
 	for i, r := range y {
-		ysegs[i] = r.segments
+		ysegs[i] = r.Segments
 	}
 
-	var dst bsiData
+	var dst BSIData
 	var xbitmaps, ybitmaps []*roaring.Bitmap
 	for {
 		// Find the next shard.
@@ -162,7 +162,7 @@ func addBSI(x, y bsiData) bsiData {
 			for len(dst) <= i {
 				dst = append(dst, NewRow())
 			}
-			dst[i].segments = append(dst[i].segments, rowSegment{
+			dst[i].Segments = append(dst[i].Segments, RowSegment{
 				shard:    next,
 				writable: true,
 				data:     b,
@@ -273,10 +273,10 @@ func (b *bsiBuilder) Insert(col, val uint64) {
 
 // Build BSI data.
 // This resets the builder.
-func (b *bsiBuilder) Build() bsiData {
+func (b *bsiBuilder) Build() BSIData {
 	builders := *b
 	*b = builders[:0]
-	rows := make(bsiData, len(builders))
+	rows := make(BSIData, len(builders))
 	for i := range builders {
 		rows[i] = builders[i].Build()
 	}

bsi_test.go

@@ -69,11 +69,11 @@ func TestBSIAdd(t *testing.T) {
 				builderB.Insert(uint64(id), vb)
 			}
 			dataA, dataB := builderA.Build(), builderB.Build()
-			dataC := addBSI(dataA, dataB)
+			dataC := AddBSI(dataA, dataB)
 
 			// build results from added bsiData; results[i] should hold a[i]+b[i]
 			results := make([]uint64, len(a))
-			dataC.pivotDescending(NewRow().Union(dataC...), 0, nil, nil, func(count uint64, ids ...uint64) {
+			dataC.PivotDescending(NewRow().Union(dataC...), 0, nil, nil, func(count uint64, ids ...uint64) {
 				for _, id := range ids {
 					results[idToIndex[int(id)]] = count
 				}
@@ -141,10 +141,10 @@ func TestBSIAddCases(t *testing.T) {
 			}
 
 			dataA, dataB := builderA.Build(), builderB.Build()
-			dataC := addBSI(dataA, dataB)
+			dataC := AddBSI(dataA, dataB)
 			// maps id to count
 			results := make(map[uint64]uint64)
-			dataC.pivotDescending(NewRow().Union(dataC...), 0, nil, nil, func(count uint64, ids ...uint64) {
+			dataC.PivotDescending(NewRow().Union(dataC...), 0, nil, nil, func(count uint64, ids ...uint64) {
 				for _, id := range ids {
 					results[id] = count
 				}

catcher.go

@@ -217,6 +217,6 @@ func (c *catcherTx) GetSortedFieldViewList(idx *Index, shard uint64) (fvs []txke
 	return c.b.GetSortedFieldViewList(idx, shard)
 }
 
-func (tx *catcherTx) GetFieldSizeBytes(index, field string) (uint64, error) {
+func (c *catcherTx) GetFieldSizeBytes(index, field string) (uint64, error) {
 	return 0, nil
 }

client/api.go

@@ -3,8 +3,9 @@ package client
 import (
 	"context"
 
-	featurebase "github.com/featurebasedb/featurebase/v3"
+	featurebase "github.com/molecula/featurebase/v3"
-	"github.com/featurebasedb/featurebase/v3/errors"
+	"github.com/molecula/featurebase/v3/client/types"
+	"github.com/molecula/featurebase/v3/errors"
 )
 
 var _ featurebase.SchemaAPI = &schemaAPI{}
@@ -104,7 +105,7 @@ func (s *schemaAPI) addFieldToIndex(idx *Index, fieldName string, opts ...featur
 		cfos = append(cfos, OptFieldTypeDecimal(ffos.Scale, ffos.Min, ffos.Max))
 	case featurebase.FieldTypeTime:
 		cfos = append(cfos,
-			OptFieldTypeTime(TimeQuantum(ffos.TimeQuantum), ffos.NoStandardView),
+			OptFieldTypeTime(types.TimeQuantum(ffos.TimeQuantum), ffos.NoStandardView),
 			OptFieldKeys(ffos.Keys),
 		)
 	case featurebase.FieldTypeTimestamp:
@@ -152,6 +153,12 @@ func (s *schemaAPI) IndexInfo(ctx context.Context, indexName string) (*featureba
 	return FromClientIndex(idx), nil
 }
 
+// FieldInfo returns the same information as Schema(), but only for a single
+// index.
+func (s *schemaAPI) FieldInfo(ctx context.Context, indexName, fieldName string) (*featurebase.FieldInfo, error) {
+	return nil, nil
+}
+
 func (s *schemaAPI) Schema(ctx context.Context, withViews bool) ([]*featurebase.IndexInfo, error) {
 	return nil, errors.New("", "schemaAPI.Schema is not implemented")
 }

client/client.go

@@ -29,6 +29,15 @@ import (
 	"github.com/featurebasedb/featurebase/v3/roaring"
 	"github.com/featurebasedb/featurebase/v3/stats"
 	"github.com/golang/protobuf/proto" //nolint:staticcheck
+	pilosa "github.com/molecula/featurebase/v3"
+	"github.com/molecula/featurebase/v3/client/types"
+	fbproto "github.com/molecula/featurebase/v3/encoding/proto" // TODO use this everywhere and get rid of proto import
+	"github.com/molecula/featurebase/v3/logger"
+	pnet "github.com/molecula/featurebase/v3/net"
+	"github.com/molecula/featurebase/v3/pb"
+	"github.com/molecula/featurebase/v3/pql"
+	"github.com/molecula/featurebase/v3/roaring"
+	"github.com/molecula/featurebase/v3/stats"
 	"github.com/opentracing/opentracing-go"
 	"github.com/pkg/errors"
 	"golang.org/x/sync/errgroup"
@@ -62,6 +71,12 @@ type Client struct {
 	tick       *time.Ticker
 	done       chan struct{}
 
+	// pathPrefix is prepended to every URL path. This is used, for example,
+	// when running a compute nodes as a sub-service of the featurebase command.
+	// In that case, a path might look like `localhost:8080/compute/schema`,
+	// where `/compute` is the pathPrefix.
+	pathPrefix string
+
 	AuthToken string
 }
 
@@ -194,6 +209,8 @@ func newClientWithOptions(options *ClientOptions) *Client {
 		done:       make(chan struct{}),
 
 		nat: options.nat,
+
+		pathPrefix: options.pathPrefix,
 	}
 
 	if options.tracer == nil {
@@ -265,6 +282,15 @@ func NewClient(addrURIOrCluster interface{}, options ...ClientOption) (*Client,
 	return newClientWithCluster(cluster, clientOptions), nil
 }
 
+// prefix is a helper function which allows us to provide a pathPrefix value as
+// "compute" instead of "/compute".
+func (c *Client) prefix() string {
+	if c.pathPrefix == "" {
+		return ""
+	}
+	return "/" + c.pathPrefix
+}
+
 // Query runs the given query against the server with the given options.
 // Pass nil for default options.
 func (c *Client) Query(query PQLQuery, options ...interface{}) (*QueryResponse, error) {
@@ -284,7 +310,7 @@ func (c *Client) Query(query PQLQuery, options ...interface{}) (*QueryResponse,
 	if err != nil {
 		return nil, errors.Wrap(err, "making request data")
 	}
-	path := fmt.Sprintf("/index/%s/query", query.Index().name)
+	path := fmt.Sprintf("%s/index/%s/query", c.prefix(), query.Index().name)
 	_, respData, err := c.HTTPRequest("POST", path, reqData, c.augmentHeaders(defaultProtobufHeaders()))
 	if err != nil {
 		return nil, err
@@ -307,7 +333,7 @@ func (c *Client) CreateIndex(index *Index) error {
 	defer span.Finish()
 
 	data := []byte(index.options.String())
-	path := fmt.Sprintf("/index/%s", index.name)
+	path := fmt.Sprintf("%s/index/%s", c.prefix(), index.name)
 	status, body, err := c.HTTPRequest("POST", path, data, c.augmentHeaders(nil))
 	if err != nil {
 		return errors.Wrapf(err, "creating index: %s", index.name)
@@ -331,7 +357,7 @@ func (c *Client) CreateField(field *Field) error {
 	defer span.Finish()
 
 	data := []byte(field.options.String())
-	path := fmt.Sprintf("/index/%s/field/%s", field.index.name, field.name)
+	path := fmt.Sprintf("%s/index/%s/field/%s", c.prefix(), field.index.name, field.name)
 	status, body, err := c.HTTPRequest("POST", path, data, c.augmentHeaders(nil))
 	if err != nil {
 		return errors.Wrapf(err, "creating field: %s in index: %s", field.name, field.index.name)
@@ -399,7 +425,7 @@ func (c *Client) DeleteIndexByName(index string) error {
 	span := c.tracer.StartSpan("Client.DeleteIndex")
 	defer span.Finish()
 
-	path := fmt.Sprintf("/index/%s", index)
+	path := fmt.Sprintf("%s/index/%s", c.prefix(), index)
 	_, _, err := c.HTTPRequest("DELETE", path, nil, c.augmentHeaders(nil))
 	return err
 }
@@ -409,7 +435,7 @@ func (c *Client) DeleteField(field *Field) error {
 	span := c.tracer.StartSpan("Client.DeleteField")
 	defer span.Finish()
 
-	path := fmt.Sprintf("/index/%s/field/%s", field.index.name, field.name)
+	path := fmt.Sprintf("%s/index/%s/field/%s", c.prefix(), field.index.name, field.name)
 	_, _, err := c.HTTPRequest("DELETE", path, nil, c.augmentHeaders(nil))
 	return err
 }
@@ -520,7 +546,7 @@ func (c *Client) EncodeImport(field *Field, shard uint64, vals, ids []uint64, cl
 	if err != nil {
 		return "", nil, errors.Wrap(err, "marshaling Import to protobuf")
 	}
-	path = fmt.Sprintf("/index/%s/field/%s/import?clear=%s&ignoreKeyCheck=true", field.index.Name(), field.Name(), strconv.FormatBool(clear))
+	path = fmt.Sprintf("%s/index/%s/field/%s/import?clear=%s&ignoreKeyCheck=true", c.prefix(), field.index.Name(), field.Name(), strconv.FormatBool(clear))
 	return path, data, nil
 }
 
@@ -567,7 +593,7 @@ func (c *Client) EncodeImportValues(field *Field, shard uint64, vals []int64, id
 	if err != nil {
 		return "", nil, errors.Wrap(err, "marshaling ImportValue to protobuf")
 	}
-	path = fmt.Sprintf("/index/%s/field/%s/import?clear=%s&ignoreKeyCheck=true", field.index.Name(), field.Name(), strconv.FormatBool(clear))
+	path = fmt.Sprintf("%s/index/%s/field/%s/import?clear=%s&ignoreKeyCheck=true", c.prefix(), field.index.Name(), field.Name(), strconv.FormatBool(clear))
 	return path, data, nil
 }
 
@@ -598,7 +624,7 @@ func (c *Client) fetchFragmentNodes(indexName string, shard uint64) ([]fragmentN
 	if c.manualFragmentNode != nil {
 		return []fragmentNode{*c.manualFragmentNode}, nil
 	}
-	path := fmt.Sprintf("/internal/fragment/nodes?shard=%d&index=%s", shard, indexName)
+	path := fmt.Sprintf("%s/internal/fragment/nodes?shard=%d&index=%s", c.prefix(), shard, indexName)
 	_, body, err := c.HTTPRequest("GET", path, []byte{}, c.augmentHeaders(nil))
 	if err != nil {
 		return nil, err
@@ -661,7 +687,7 @@ func (c *Client) ImportRoaringShard(index string, shard uint64, request *pilosa.
 	for _, uri := range uris {
 		uri := uri
 		eg.Go(func() error {
-			return c.importData(uri, fmt.Sprintf("/index/%s/shard/%d/import-roaring", index, shard), data)
+			return c.importData(uri, fmt.Sprintf("%s/index/%s/shard/%d/import-roaring", c.prefix(), index, shard), data)
 		})
 	}
 	err = eg.Wait()
@@ -695,7 +721,7 @@ func (c *Client) importRoaringBitmap(uri *pnet.URI, field *Field, shard uint64,
 	}
 	params := url.Values{}
 	params.Add("clear", strconv.FormatBool(options.clear))
-	path := makeRoaringImportPath(field, shard, params)
+	path := makeRoaringImportPath(field, shard, params, c.prefix())
 	req := &pb.ImportRoaringRequest{
 		Clear:          options.clear,
 		Views:          protoViews,
@@ -749,7 +775,8 @@ func (c *Client) Info() (Info, error) {
 	span := c.tracer.StartSpan("Client.Info")
 	defer span.Finish()
 
-	_, data, err := c.HTTPRequest("GET", "/info", nil, c.augmentHeaders(nil))
+	path := fmt.Sprintf("%s/info", c.prefix())
+	_, data, err := c.HTTPRequest("GET", path, nil, c.augmentHeaders(nil))
 	if err != nil {
 		return Info{}, errors.Wrap(err, "requesting /info")
 	}
@@ -766,7 +793,8 @@ func (c *Client) Status() (Status, error) {
 	span := c.tracer.StartSpan("Client.Status")
 	defer span.Finish()
 
-	_, data, err := c.HTTPRequest("GET", "/status", nil, nil)
+	path := fmt.Sprintf("%s/status", c.prefix())
+	_, data, err := c.HTTPRequest("GET", path, nil, nil)
 	if err != nil {
 		return Status{}, errors.Wrap(err, "requesting /status")
 	}
@@ -779,7 +807,8 @@ func (c *Client) Status() (Status, error) {
 }
 
 func (c *Client) readSchema() ([]SchemaIndex, error) {
-	_, data, err := c.HTTPRequest("GET", "/schema", nil, c.augmentHeaders(nil))
+	path := fmt.Sprintf("%s/schema", c.prefix())
+	_, data, err := c.HTTPRequest("GET", path, nil, c.augmentHeaders(nil))
 	if err != nil {
 		return nil, errors.Wrap(err, "requesting /schema")
 	}
@@ -792,7 +821,8 @@ func (c *Client) readSchema() ([]SchemaIndex, error) {
 }
 
 func (c *Client) shardsMax() (map[string]uint64, error) {
-	_, data, err := c.HTTPRequest("GET", "/internal/shards/max", nil, nil)
+	path := fmt.Sprintf("%s/internal/shards/max", c.prefix())
+	_, data, err := c.HTTPRequest("GET", path, nil, nil)
 	if err != nil {
 		return nil, errors.Wrap(err, "requesting /internal/shards/max")
 	}
@@ -835,7 +865,8 @@ func (c *Client) httpRequest(method string, path string, data []byte, headers ma
 		// doRequest implements expotential backoff
 		status, body, err = c.doRequest(host, method, path, c.augmentHeaders(headers), data)
 		// conditions when primary should not be tried
-		if err == nil || usePrimary || path == "/status" {
+		pathCheck := fmt.Sprintf("%s/status", c.prefix())
+		if err == nil || usePrimary || path == pathCheck {
 			break
 		}
 
@@ -1018,7 +1049,7 @@ func (c *Client) augmentHeaders(headers map[string]string) map[string]string {
 // FindFieldKeys looks up the IDs associated with specified keys in a field.
 // If a key does not exist, the result will not include it.
 func (c *Client) FindFieldKeys(field *Field, keys ...string) (map[string]uint64, error) {
-	path := fmt.Sprintf("/internal/translate/field/%s/%s/keys/find", field.index.name, field.name)
+	path := fmt.Sprintf("%s/internal/translate/field/%s/%s/keys/find", c.prefix(), field.index.name, field.name)
 
 	reqData, err := json.Marshal(keys)
 	if err != nil {
@@ -1050,7 +1081,7 @@ func (c *Client) FindFieldKeys(field *Field, keys ...string) (map[string]uint64,
 // CreateFieldKeys looks up the IDs associated with specified keys in a field.
 // If a key does not exist, it will be created.
 func (c *Client) CreateFieldKeys(field *Field, keys ...string) (map[string]uint64, error) {
-	path := fmt.Sprintf("/internal/translate/field/%s/%s/keys/create", field.index.name, field.name)
+	path := fmt.Sprintf("%s/internal/translate/field/%s/%s/keys/create", c.prefix(), field.index.name, field.name)
 
 	reqData, err := json.Marshal(keys)
 	if err != nil {
@@ -1082,7 +1113,7 @@ func (c *Client) CreateFieldKeys(field *Field, keys ...string) (map[string]uint6
 // FindIndexKeys looks up the IDs associated with specified column keys in an index.
 // If a key does not exist, the result will not include it.
 func (c *Client) FindIndexKeys(idx *Index, keys ...string) (map[string]uint64, error) {
-	path := fmt.Sprintf("/internal/translate/index/%s/keys/find", idx.name)
+	path := fmt.Sprintf("%s/internal/translate/index/%s/keys/find", c.prefix(), idx.name)
 
 	reqData, err := json.Marshal(keys)
 	if err != nil {
@@ -1114,7 +1145,7 @@ func (c *Client) FindIndexKeys(idx *Index, keys ...string) (map[string]uint64, e
 // CreateIndexKeys looks up the IDs associated with specified column keys in an index.
 // If a key does not exist, it will be created.
 func (c *Client) CreateIndexKeys(idx *Index, keys ...string) (map[string]uint64, error) {
-	path := fmt.Sprintf("/internal/translate/index/%s/keys/create", idx.name)
+	path := fmt.Sprintf("%s/internal/translate/index/%s/keys/create", c.prefix(), idx.name)
 
 	reqData, err := json.Marshal(keys)
 	if err != nil {
@@ -1168,7 +1199,8 @@ func (c *Client) startTransaction(id string, timeout time.Duration, exclusive bo
 		return nil, errors.Wrap(err, "marshalling transaction")
 	}
 
-	status, data, err := c.httpRequest("POST", "/transaction", bod, c.augmentHeaders(defaultJSONHeaders()), true)
+	path := fmt.Sprintf("%s/transaction", c.prefix())
+	status, data, err := c.httpRequest("POST", path, bod, c.augmentHeaders(defaultJSONHeaders()), true)
 	if status == http.StatusConflict && time.Now().Before(deadline) {
 		// if we're getting StatusConflict after all the usual timeouts/retries, keep retrying until the deadline
 		time.Sleep(time.Second)
@@ -1195,7 +1227,8 @@ func (c *Client) startTransaction(id string, timeout time.Duration, exclusive bo
 }
 
 func (c *Client) FinishTransaction(id string) (*pilosa.Transaction, error) {
-	_, data, err := c.httpRequest("POST", "/transaction/"+id+"/finish", nil, c.augmentHeaders(defaultJSONHeaders()), true)
+	path := fmt.Sprintf("%s/transaction/%s/finish", c.prefix(), id)
+	_, data, err := c.httpRequest("POST", path, nil, c.augmentHeaders(defaultJSONHeaders()), true)
 	if err != nil && len(data) == 0 {
 		return nil, err
 	}
@@ -1217,7 +1250,8 @@ func (c *Client) FinishTransaction(id string) (*pilosa.Transaction, error) {
 }
 
 func (c *Client) Transactions() (map[string]*pilosa.Transaction, error) {
-	_, respData, err := c.httpRequest("GET", "/transactions", nil, c.augmentHeaders(defaultJSONHeaders()), true)
+	path := fmt.Sprintf("%s/transactions", c.prefix())
+	_, respData, err := c.httpRequest("GET", path, nil, c.augmentHeaders(defaultJSONHeaders()), true)
 	if err != nil {
 		return nil, errors.Wrap(err, "getting transactions")
 	}
@@ -1231,7 +1265,8 @@ func (c *Client) Transactions() (map[string]*pilosa.Transaction, error) {
 }
 
 func (c *Client) GetTransaction(id string) (*pilosa.Transaction, error) {
-	_, data, err := c.httpRequest("GET", "/transaction/"+id, nil, c.augmentHeaders(defaultJSONHeaders()), true)
+	path := fmt.Sprintf("%s/transaction/%s", c.prefix(), id)
+	_, data, err := c.httpRequest("GET", path, nil, c.augmentHeaders(defaultJSONHeaders()), true)
 	if err != nil {
 		return nil, err
 	}
@@ -1308,9 +1343,9 @@ func makeRequestData(query string, options *QueryOptions) ([]byte, error) {
 	return r, nil
 }
 
-func makeRoaringImportPath(field *Field, shard uint64, params url.Values) string {
+func makeRoaringImportPath(field *Field, shard uint64, params url.Values, pathPrefix string) string {
-	return fmt.Sprintf("/index/%s/field/%s/import-roaring/%d?%s",
+	return fmt.Sprintf("%s/index/%s/field/%s/import-roaring/%d?%s",
-		field.index.name, field.name, shard, params.Encode())
+		pathPrefix, field.index.name, field.name, shard, params.Encode())
 }
 
 type viewImports map[string]*roaring.Bitmap
@@ -1327,6 +1362,7 @@ type ClientOptions struct {
 	retries             *int
 	stats               stats.StatsClient
 	nat                 map[pnet.URI]pnet.URI
+	pathPrefix          string
 }
 
 func (co *ClientOptions) addOptions(options ...ClientOption) error {
@@ -1438,6 +1474,14 @@ func OptClientNAT(nat map[string]string) ClientOption {
 	}
 }
 
+// OptClientPathPrefix sets the http path prefix.
+func OptClientPathPrefix(prefix string) ClientOption {
+	return func(options *ClientOptions) error {
+		options.pathPrefix = prefix
+		return nil
+	}
+}
+
 func (co *ClientOptions) withDefaults() (updated *ClientOptions) {
 	// copy options so the original is not updated
 	updated = &ClientOptions{}
@@ -1690,7 +1734,7 @@ func (so SchemaOptions) asFieldOptions() *FieldOptions {
 		fieldType:      so.FieldType,
 		cacheSize:      int(so.CacheSize),
 		cacheType:      CacheType(so.CacheType),
-		timeQuantum:    TimeQuantum(so.TimeQuantum),
+		timeQuantum:    types.TimeQuantum(so.TimeQuantum),
 		ttl:            so.TTL,
 		min:            so.Min,
 		max:            so.Max,
@@ -1734,8 +1778,8 @@ func (r *exportReader) Read(p []byte) (n int, err error) {
 		headers := map[string]string{
 			"Accept": "text/csv",
 		}
-		path := fmt.Sprintf("/export?index=%s&field=%s&shard=%d",
+		path := fmt.Sprintf("%s/export?index=%s&field=%s&shard=%d",
-			r.field.index.Name(), r.field.Name(), r.currentShard)
+			r.client.prefix(), r.field.index.Name(), r.field.Name(), r.currentShard)
 		_, respData, err := r.client.doRequest(uri, "GET", path, headers, nil)
 		if err != nil {
 			return 0, errors.Wrap(err, "doing export request")
@@ -1751,3 +1795,9 @@ func (r *exportReader) Read(p []byte) (n int, err error) {
 	}
 	return
 }
+
+// SetAuthToken sets the Client.AuthToken value. We needed this to be a method
+// in order to satisfy the SchemaManager interface.
+func (c *Client) SetAuthToken(token string) {
+	c.AuthToken = token
+}

client/client_it_test.go

@@ -9,12 +9,13 @@ import (
 	"testing"
 	"time"
 
-	featurebase "github.com/featurebasedb/featurebase/v3"
+	featurebase "github.com/molecula/featurebase/v3"
-	"github.com/featurebasedb/featurebase/v3/disco"
+	client_types "github.com/molecula/featurebase/v3/client/types"
-	pnet "github.com/featurebasedb/featurebase/v3/net"
+	"github.com/molecula/featurebase/v3/disco"
-	"github.com/featurebasedb/featurebase/v3/roaring"
+	pnet "github.com/molecula/featurebase/v3/net"
-	"github.com/featurebasedb/featurebase/v3/shardwidth"
+	"github.com/molecula/featurebase/v3/roaring"
-	"github.com/featurebasedb/featurebase/v3/test"
+	"github.com/molecula/featurebase/v3/shardwidth"
+	"github.com/molecula/featurebase/v3/test"
 	"github.com/stretchr/testify/require"
 	"golang.org/x/sync/errgroup"
 )
@@ -45,7 +46,7 @@ func setup(t *testing.T, cli *Client) {
 		OptIndexTrackExistence(false),
 	)
 	testField = testIndex.Field("test-field")
-	testFieldTimeQuantum = testIndex.Field("test-field-timequantum", OptFieldTypeTime(TimeQuantumYear))
+	testFieldTimeQuantum = testIndex.Field("test-field-timequantum", OptFieldTypeTime(client_types.TimeQuantumYear))
 	testFieldTimestamp = testIndex.Field("test-field-timestamp", OptFieldTypeTimestamp(time.Date(1970, time.January, 1, 0, 0, 0, 0, time.UTC), "s"))
 	testFieldInt = testIndex.Field("test-field-int", OptFieldTypeInt(0, 100000))
 	testIndexKeyTranslation = testSchema.Index("test-index-key-translation", OptIndexKeys(true))
@@ -588,7 +589,7 @@ func TestClientAgainstCluster(t *testing.T) {
 				setup(t, cli)
 				defer tearDown(t, cli)
 
-				testFieldRange := testIndex.Field("test-field-range", OptFieldTypeTime(TimeQuantumMonthDayHour))
+				testFieldRange := testIndex.Field("test-field-range", OptFieldTypeTime(client_types.TimeQuantumMonthDayHour))
 				err := cli.EnsureField(testFieldRange)
 				require.NoError(t, err)
 

client/importer.go

@@ -39,8 +39,8 @@ func fromClientIndexOptions(cio IndexOptions) featurebase.IndexOptions {
 	}
 }
 
-// toClientIndex
+// ToClientIndex
-func toClientIndex(fi *featurebase.IndexInfo) *Index {
+func ToClientIndex(fi *featurebase.IndexInfo) *Index {
 	sch := NewSchema()
 	return sch.Index(fi.Name,
 		OptIndexKeys(fi.Options.Keys),
@@ -80,8 +80,8 @@ func fromClientFieldOptions(cfo FieldOptions) featurebase.FieldOptions {
 	}
 }
 
-// toClientField
+// ToClientField
-func toClientField(index string, ff *featurebase.FieldInfo) (*Field, error) {
+func ToClientField(index string, ff *featurebase.FieldInfo) (*Field, error) {
 	sch := NewSchema()
 	idx := sch.Index(index)
 
@@ -171,11 +171,11 @@ func (i *importer) FinishTransaction(ctx context.Context, id string) (*featureba
 }
 
 func (i *importer) CreateIndexKeys(ctx context.Context, idx *featurebase.IndexInfo, keys ...string) (map[string]uint64, error) {
-	return i.Client.CreateIndexKeys(toClientIndex(idx), keys...)
+	return i.Client.CreateIndexKeys(ToClientIndex(idx), keys...)
 }
 
 func (i *importer) CreateFieldKeys(ctx context.Context, index string, field *featurebase.FieldInfo, keys ...string) (map[string]uint64, error) {
-	fld, err := toClientField(index, field)
+	fld, err := ToClientField(index, field)
 	if err != nil {
 		return nil, errors.Wrap(err, "converting to client field")
 	}
@@ -183,7 +183,7 @@ func (i *importer) CreateFieldKeys(ctx context.Context, index string, field *fea
 }
 
 func (i *importer) ImportRoaringBitmap(ctx context.Context, index string, field *featurebase.FieldInfo, shard uint64, views map[string]*roaring.Bitmap, clear bool) error {
-	fld, err := toClientField(index, field)
+	fld, err := ToClientField(index, field)
 	if err != nil {
 		return errors.Wrap(err, "converting to client field")
 	}
@@ -195,7 +195,7 @@ func (i *importer) ImportRoaringShard(ctx context.Context, index string, shard u
 }
 
 func (i *importer) EncodeImportValues(ctx context.Context, index string, field *featurebase.FieldInfo, shard uint64, vals []int64, ids []uint64, clear bool) (path string, data []byte, err error) {
-	fld, err := toClientField(index, field)
+	fld, err := ToClientField(index, field)
 	if err != nil {
 		return "", nil, errors.Wrap(err, "converting to client field")
 	}
@@ -203,7 +203,7 @@ func (i *importer) EncodeImportValues(ctx context.Context, index string, field *
 }
 
 func (i *importer) EncodeImport(ctx context.Context, index string, field *featurebase.FieldInfo, shard uint64, vals, ids []uint64, clear bool) (path string, data []byte, err error) {
-	fld, err := toClientField(index, field)
+	fld, err := ToClientField(index, field)
 	if err != nil {
 		return "", nil, errors.Wrap(err, "converting to client field")
 	}

client/orm.go

@@ -14,7 +14,8 @@ import (
 	"sync"
 	"time"
 
-	"github.com/featurebasedb/featurebase/v3/pql"
+	"github.com/molecula/featurebase/v3/client/types"
+	"github.com/molecula/featurebase/v3/pql"
 	"github.com/pkg/errors"
 )
 
@@ -728,7 +729,7 @@ type FieldInfo struct {
 // FieldOptions contains options to customize Field objects and field queries.
 type FieldOptions struct {
 	fieldType      FieldType
-	timeQuantum    TimeQuantum
+	timeQuantum    types.TimeQuantum
 	ttl            time.Duration
 	cacheType      CacheType
 	cacheSize      int
@@ -748,9 +749,14 @@ func (fo FieldOptions) Type() FieldType {
 	return fo.fieldType
 }
 
+// Base returns the base of the field.
+func (fo FieldOptions) Base() int64 {
+	return fo.base
+}
+
 // TimeQuantum returns the configured time quantum for a time field. Empty
 // string otherwise.
-func (fo FieldOptions) TimeQuantum() TimeQuantum {
+func (fo FieldOptions) TimeQuantum() types.TimeQuantum {
 	return fo.timeQuantum
 }
 
@@ -798,10 +804,6 @@ func (fo FieldOptions) TimeUnit() string {
 	return fo.timeUnit
 }
 
-func (fo FieldOptions) Base() int64 {
-	return fo.base
-}
-
 // NoStandardView suppresses creating the standard view for supported field types (currently, time)
 func (fo FieldOptions) NoStandardView() bool {
 	return fo.noStandardView
@@ -869,12 +871,12 @@ func (fo *FieldOptions) addOptions(options ...FieldOption) {
 
 // MinTimestamp returns the minimum value for a timestamp field.
 func (o FieldOptions) MinTimestamp() time.Time {
-	return time.Unix(0, o.min.ToInt64(0)*int64(TimeUnitNano(o.TimeUnit())))
+	return time.Unix(0, o.min.ToInt64(0)*int64(types.TimeUnitNano(o.TimeUnit())))
 }
 
 // MaxTimestamp returns the maxnimum value for a timestamp field.
 func (o FieldOptions) MaxTimestamp() time.Time {
-	return time.Unix(0, o.max.ToInt64(0)*int64(TimeUnitNano(o.TimeUnit())))
+	return time.Unix(0, o.max.ToInt64(0)*int64(types.TimeUnitNano(o.TimeUnit())))
 }
 
 // FieldOption is used to pass an option to index.Field function.
@@ -917,7 +919,7 @@ func OptFieldTypeInt(limits ...int64) FieldOption {
 }
 
 // OptFieldTypeTime adds a time field.
-func OptFieldTypeTime(quantum TimeQuantum, opts ...bool) FieldOption {
+func OptFieldTypeTime(quantum types.TimeQuantum, opts ...bool) FieldOption {
 	return func(options *FieldOptions) {
 		options.fieldType = FieldTypeTime
 		options.timeQuantum = quantum
@@ -944,11 +946,11 @@ var (
 // TimeUnitNanos returns the number of nanoseconds in unit.
 func TimeUnitNanos(unit string) int64 {
 	switch unit {
-	case TimeUnitSeconds:
+	case types.TimeUnitSeconds:
 		return int64(time.Second)
-	case TimeUnitMilliseconds:
+	case types.TimeUnitMilliseconds:
 		return int64(time.Millisecond)
-	case TimeUnitMicroseconds:
+	case types.TimeUnitMicroseconds:
 		return int64(time.Microsecond)
 	default:
 		return int64(time.Nanosecond)
@@ -1245,46 +1247,6 @@ const (
 	FieldTypeTimestamp FieldType = "timestamp"
 )
 
-// TimeQuantum type represents valid time quantum values time fields.
-type TimeQuantum string
-
-// TimeQuantum constants
-const (
-	TimeQuantumNone             TimeQuantum = ""
-	TimeQuantumYear             TimeQuantum = "Y"
-	TimeQuantumMonth            TimeQuantum = "M"
-	TimeQuantumDay              TimeQuantum = "D"
-	TimeQuantumHour             TimeQuantum = "H"
-	TimeQuantumYearMonth        TimeQuantum = "YM"
-	TimeQuantumMonthDay         TimeQuantum = "MD"
-	TimeQuantumDayHour          TimeQuantum = "DH"
-	TimeQuantumYearMonthDay     TimeQuantum = "YMD"
-	TimeQuantumMonthDayHour     TimeQuantum = "MDH"
-	TimeQuantumYearMonthDayHour TimeQuantum = "YMDH"
-)
-
-// List of time units.
-const (
-	TimeUnitSeconds      = "s"
-	TimeUnitMilliseconds = "ms"
-	TimeUnitMicroseconds = "µs"
-	TimeUnitNanoseconds  = "ns"
-)
-
-// TimeUnitNano returns the number of nanoseconds in unit.
-func TimeUnitNano(unit string) int64 {
-	switch unit {
-	case TimeUnitSeconds:
-		return int64(time.Second)
-	case TimeUnitMilliseconds:
-		return int64(time.Millisecond)
-	case TimeUnitMicroseconds:
-		return int64(time.Microsecond)
-	default:
-		return int64(time.Nanosecond)
-	}
-}
-
 // CacheType represents cache type for a field
 type CacheType string
 

client/orm_test.go

@@ -14,8 +14,9 @@ import (
 	"testing"
 	"time"
 
-	pilosa "github.com/featurebasedb/featurebase/v3"
+	pilosa "github.com/molecula/featurebase/v3"
-	"github.com/featurebasedb/featurebase/v3/pql"
+	clienttypes "github.com/molecula/featurebase/v3/client/types"
+	"github.com/molecula/featurebase/v3/pql"
 	"github.com/pkg/errors"
 )
 
@@ -102,7 +103,7 @@ func TestORM(t *testing.T) {
 
 	t.Run("NewIndexCopy", func(t *testing.T) {
 		index := schema.Index("my-index-4copy", OptIndexKeys(true))
-		index.Field("my-field-4copy", OptFieldTypeTime(TimeQuantumDayHour))
+		index.Field("my-field-4copy", OptFieldTypeTime(clienttypes.TimeQuantumDayHour))
 		copiedIndex := index.copy()
 		if !reflect.DeepEqual(index, copiedIndex) {
 			t.Fatalf("copied index should be equivalent")
@@ -920,7 +921,7 @@ func TestORM(t *testing.T) {
 		compareFieldOptions(t,
 			field.Options(),
 			FieldTypeSet,
-			TimeQuantumNone,
+			clienttypes.TimeQuantumNone,
 			CacheTypeRanked,
 			9999,
 			pql.NewDecimal(0, 0),
@@ -940,7 +941,7 @@ func TestORM(t *testing.T) {
 		compareFieldOptions(t,
 			field.Options(),
 			FieldTypeInt,
-			TimeQuantumNone,
+			clienttypes.TimeQuantumNone,
 			CacheTypeDefault,
 			0,
 			pql.NewDecimal(-10, 0),
@@ -959,7 +960,7 @@ func TestORM(t *testing.T) {
 		compareFieldOptions(t,
 			field.Options(),
 			FieldTypeInt,
-			TimeQuantumNone,
+			clienttypes.TimeQuantumNone,
 			CacheTypeDefault,
 			0,
 			pql.NewDecimal(-10, 0),
@@ -976,7 +977,7 @@ func TestORM(t *testing.T) {
 		compareFieldOptions(t,
 			field.Options(),
 			FieldTypeInt,
-			TimeQuantumNone,
+			clienttypes.TimeQuantumNone,
 			CacheTypeDefault,
 			0,
 			pql.NewDecimal(math.MinInt64, 0),
@@ -994,7 +995,7 @@ func TestORM(t *testing.T) {
 		compareFieldOptions(t,
 			field.Options(),
 			FieldTypeInt,
-			TimeQuantumNone,
+			clienttypes.TimeQuantumNone,
 			CacheTypeDefault,
 			0,
 			pql.NewDecimal(math.MinInt64, 0),
@@ -1005,7 +1006,7 @@ func TestORM(t *testing.T) {
 	})
 
 	t.Run("TimeFieldOptions", func(t *testing.T) {
-		field := sampleIndex.Field("time-field", OptFieldTypeTime(TimeQuantumDayHour, true))
+		field := sampleIndex.Field("time-field", OptFieldTypeTime(clienttypes.TimeQuantumDayHour, true))
 		if true != field.Opts().NoStandardView() {
 			t.Fatalf("field noStandardView %v != %v", true, field.Opts().NoStandardView())
 		}
@@ -1017,7 +1018,7 @@ func TestORM(t *testing.T) {
 		compareFieldOptions(t,
 			field.Options(),
 			FieldTypeTime,
-			TimeQuantumDayHour,
+			clienttypes.TimeQuantumDayHour,
 			CacheTypeDefault,
 			0,
 			pql.NewDecimal(0, 0),
@@ -1028,7 +1029,7 @@ func TestORM(t *testing.T) {
 	})
 
 	t.Run("TTLOptions", func(t *testing.T) {
-		field := sampleIndex.Field("ttl-field", OptFieldTypeTime(TimeQuantumDayHour, true), OptFieldTTL(0))
+		field := sampleIndex.Field("ttl-field", OptFieldTypeTime(clienttypes.TimeQuantumDayHour, true), OptFieldTTL(0))
 		if true != field.Opts().NoStandardView() {
 			t.Fatalf("field noStandardView %v != %v", true, field.Opts().NoStandardView())
 		}
@@ -1040,7 +1041,7 @@ func TestORM(t *testing.T) {
 		compareFieldOptions(t,
 			field.Options(),
 			FieldTypeTime,
-			TimeQuantumDayHour,
+			clienttypes.TimeQuantumDayHour,
 			CacheTypeDefault,
 			0,
 			pql.NewDecimal(0, 0),
@@ -1060,7 +1061,7 @@ func TestORM(t *testing.T) {
 		compareFieldOptions(t,
 			field.Options(),
 			FieldTypeMutex,
-			TimeQuantumNone,
+			clienttypes.TimeQuantumNone,
 			CacheTypeRanked,
 			9999,
 			pql.NewDecimal(0, 0),
@@ -1080,7 +1081,7 @@ func TestORM(t *testing.T) {
 		compareFieldOptions(t,
 			field.Options(),
 			FieldTypeBool,
-			TimeQuantumNone,
+			clienttypes.TimeQuantumNone,
 			CacheTypeDefault,
 			0,
 			pql.NewDecimal(0, 0),
@@ -1100,7 +1101,7 @@ func TestORM(t *testing.T) {
 		compareFieldOptions(t,
 			field.Options(),
 			FieldTypeDecimal,
-			TimeQuantumNone,
+			clienttypes.TimeQuantumNone,
 			CacheTypeDefault,
 			0,
 			pql.NewDecimal(7, 3),
@@ -1120,7 +1121,7 @@ func TestORM(t *testing.T) {
 		compareFieldOptions(t,
 			field.Options(),
 			FieldTypeDecimal,
-			TimeQuantumNone,
+			clienttypes.TimeQuantumNone,
 			CacheTypeDefault,
 			0,
 			pql.NewDecimal(7, 3),
@@ -1142,7 +1143,7 @@ func TestORM(t *testing.T) {
 		compareFieldOptions(t,
 			field.Options(),
 			FieldTypeTimestamp,
-			TimeQuantumNone,
+			clienttypes.TimeQuantumNone,
 			CacheTypeDefault,
 			0,
 			pql.NewDecimal(MinTimestamp.UnixNano()/TimeUnitNanos(pilosa.TimeUnitSeconds), 0),
@@ -1198,7 +1199,7 @@ func comparePQL(t *testing.T, target string, q PQLQuery) {
 	}
 }
 
-func compareFieldOptions(t *testing.T, opts *FieldOptions, fieldType FieldType, timeQuantum TimeQuantum, cacheType CacheType, cacheSize int, min pql.Decimal, max pql.Decimal, foreignIndex string, timeUnit string, ttl time.Duration) {
+func compareFieldOptions(t *testing.T, opts *FieldOptions, fieldType FieldType, timeQuantum clienttypes.TimeQuantum, cacheType CacheType, cacheSize int, min pql.Decimal, max pql.Decimal, foreignIndex string, timeUnit string, ttl time.Duration) {
 	if fieldType != opts.Type() {
 		t.Fatalf("%s != %s", fieldType, opts.Type())
 	}

client/types/time.go

@@ -0,0 +1,49 @@
+// Copyright 2021 Molecula Corp. All rights reserved.
+// package ctl contains all pilosa subcommands other than 'server'. These are
+// generally administration, testing, and debugging tools.
+
+package types
+
+import (
+	"time"
+)
+
+// TimeQuantum type represents valid time quantum values time fields.
+type TimeQuantum string
+
+// TimeQuantum constants
+const (
+	TimeQuantumNone             TimeQuantum = ""
+	TimeQuantumYear             TimeQuantum = "Y"
+	TimeQuantumMonth            TimeQuantum = "M"
+	TimeQuantumDay              TimeQuantum = "D"
+	TimeQuantumHour             TimeQuantum = "H"
+	TimeQuantumYearMonth        TimeQuantum = "YM"
+	TimeQuantumMonthDay         TimeQuantum = "MD"
+	TimeQuantumDayHour          TimeQuantum = "DH"
+	TimeQuantumYearMonthDay     TimeQuantum = "YMD"
+	TimeQuantumMonthDayHour     TimeQuantum = "MDH"
+	TimeQuantumYearMonthDayHour TimeQuantum = "YMDH"
+)
+
+// List of time units.
+const (
+	TimeUnitSeconds      = "s"
+	TimeUnitMilliseconds = "ms"
+	TimeUnitMicroseconds = "µs"
+	TimeUnitNanoseconds  = "ns"
+)
+
+// TimeUnitNano returns the number of nanoseconds in unit.
+func TimeUnitNano(unit string) int64 {
+	switch unit {
+	case TimeUnitSeconds:
+		return int64(time.Second)
+	case TimeUnitMilliseconds:
+		return int64(time.Millisecond)
+	case TimeUnitMicroseconds:
+		return int64(time.Microsecond)
+	default:
+		return int64(time.Nanosecond)
+	}
+}

cluster.go

@@ -5,12 +5,15 @@ package pilosa
 import (
 	"context"
 	"fmt"
+	"log"
 	"sync"
 	"time"
 
-	"github.com/featurebasedb/featurebase/v3/disco"
+	"github.com/molecula/featurebase/v3/dax"
-	"github.com/featurebasedb/featurebase/v3/logger"
+	"github.com/molecula/featurebase/v3/dax/computer"
-	"github.com/featurebasedb/featurebase/v3/roaring"
+	"github.com/molecula/featurebase/v3/disco"
+	"github.com/molecula/featurebase/v3/logger"
+	"github.com/molecula/featurebase/v3/roaring"
 	"github.com/pkg/errors"
 	"golang.org/x/sync/errgroup"
 )
@@ -68,6 +71,13 @@ type cluster struct { // nolint: maligned
 	confirmDownSleep   time.Duration
 
 	partitionAssigner string
+
+	writeLogWriter computer.WriteLogWriter
+	versionStore   dax.VersionStore
+
+	// isComputeNode is set to true if this node is running as a DAX compute
+	// node.
+	isComputeNode bool
 }
 
 // newCluster returns a new instance of Cluster with defaults.
@@ -90,6 +100,8 @@ func newCluster() *cluster {
 
 		disCo: disco.NopDisCo,
 		noder: disco.NewEmptyLocalNoder(),
+
+		writeLogWriter: computer.NewNopWriteLogWriter(),
 	}
 }
 
@@ -323,8 +335,35 @@ func (c *cluster) createFieldKeys(ctx context.Context, field *Field, keys ...str
 		return nil, errors.Errorf("translating field(%s/%s) keys(%v) - cannot find primary node", field.Index(), field.Name(), keys)
 	}
 	if c.Node.ID == primary.ID {
-		// The local copy is the authoritative copy.
+		translations, err := field.TranslateStore().CreateKeys(keys...)
-		return field.TranslateStore().CreateKeys(keys...)
+		if err != nil {
+			return nil, errors.Errorf("creating field(%s/%s) keys(%v)", field.Index(), field.Name(), keys)
+		}
+
+		// If this is not a DAX compute node, bail early; there's no need to
+		// send data to the write log.
+		if !c.isComputeNode {
+			return translations, nil
+		}
+
+		// Send to write log.
+		tkey := dax.TableKey(field.Index())
+		qtid := tkey.QualifiedTableID()
+		fieldName := dax.FieldName(field.Name())
+
+		// Get the current version for field.
+		version, found, err := c.versionStore.FieldVersion(ctx, qtid, fieldName)
+		if err != nil {
+			return nil, errors.Wrap(err, "getting field version")
+		} else if !found {
+			return nil, errors.Errorf("no version found for table(%s) field(%s)", qtid, fieldName)
+		}
+
+		if err := c.writeLogWriter.CreateFieldKeys(ctx, qtid, fieldName, version, translations); err != nil {
+			return nil, errors.Errorf("logging field(%s/%s) keys(%v)", field.Index(), field.Name(), keys)
+		}
+
+		return translations, nil
 	}
 
 	// Attempt to find the keys locally.
@@ -515,6 +554,13 @@ func (c *cluster) findIndexKeys(ctx context.Context, indexName string, keys ...s
 	for _, key := range keys {
 		partitionID := snap.KeyToKeyPartition(indexName, key)
 		keysByPartition[partitionID] = append(keysByPartition[partitionID], key)
+
+		// This node only handles keys for the partition(s) that it owns.
+		if c.isComputeNode {
+			if !intInPartitions(partitionID, idx.translatePartitions) {
+				return nil, errors.Errorf("cannot find key on this partition: %s, %d", key, partitionID)
+			}
+		}
 	}
 
 	// TODO: use local replicas to short-circuit network traffic
@@ -624,6 +670,14 @@ func (c *cluster) createIndexKeys(ctx context.Context, indexName string, keys ..
 	for _, key := range keys {
 		partitionID := snap.KeyToKeyPartition(indexName, key)
 		keysByPartition[partitionID] = append(keysByPartition[partitionID], key)
+
+		// This node only handles keys for the partition(s) that it owns.
+		if c.isComputeNode {
+			if !intInPartitions(partitionID, idx.translatePartitions) {
+				log.Printf("cannot create key on this partition: %s, %d", key, partitionID)
+				return nil, errors.Errorf("cannot create key on this partition: %s, %d", key, partitionID)
+			}
+		}
 	}
 
 	// TODO: use local replicas to short-circuit network traffic
@@ -689,7 +743,27 @@ func (c *cluster) createIndexKeys(ctx context.Context, indexName string, keys ..
 			}
 
 			translateResults <- translations
-			return nil
+
+			// If this is not a DAX compute node, bail early; there's no need to
+			// send data to the write log.
+			if !c.isComputeNode {
+				return nil
+			}
+
+			// Send to write log.
+			tkey := dax.TableKey(idx.Name())
+			qtid := tkey.QualifiedTableID()
+			partitionNum := dax.PartitionNum(partitionID)
+
+			// Get the current version for partition.
+			version, found, err := c.versionStore.PartitionVersion(ctx, qtid, partitionNum)
+			if err != nil {
+				return errors.Wrap(err, "getting partition version")
+			} else if !found {
+				return errors.Errorf("no version found for table(%s) partition(%d)", qtid, partitionNum)
+			}
+
+			return c.writeLogWriter.CreateTableKeys(ctx, qtid, partitionNum, version, translations)
 		})
 	}
 
@@ -744,6 +818,12 @@ func (c *cluster) translateIndexIDSet(ctx context.Context, indexName string, idS
 	idsByPartition := make(map[int][]uint64, c.partitionN)
 	for id := range idSet {
 		partitionID := snap.IDToShardPartition(indexName, id)
+		// This node only handles keys for the partition(s) that it owns.
+		if c.isComputeNode {
+			if !intInPartitions(partitionID, index.translatePartitions) {
+				return nil, errors.Errorf("cannot find id on this partition: %d, %d", id, partitionID)
+			}
+		}
 		idsByPartition[partitionID] = append(idsByPartition[partitionID], id)
 	}
 
@@ -912,3 +992,12 @@ type TransactionMessage struct {
 	Transaction *Transaction
 	Action      string
 }
+
+func intInPartitions(i int, s dax.Partitions) bool {
+	for _, a := range s {
+		if int(a.Num) == i {
+			return true
+		}
+	}
+	return false
+}

cmd/cli.go

@@ -26,6 +26,14 @@ func newCLICommand(stdin io.Reader, stdout, stderr io.Writer) *cobra.Command {
 	flags := cliCmd.Flags()
 	flags.StringVarP(&cli.Host, "host", "", cli.Host, "hostname of FeatureBase.")
 	flags.StringVarP(&cli.Port, "port", "", cli.Port, "port of FeatureBase.")
+	flags.StringVar(&cli.HistoryPath, "history-path", cli.HistoryPath, "path for history files.")
+	flags.StringVar(&cli.OrganizationID, "org-id", cli.OrganizationID, "OrganizationID.")
+	flags.StringVar(&cli.DatabaseID, "db-id", cli.DatabaseID, "DatabaseID.")
+
+	flags.StringVar(&cli.ClientID, "client-id", cli.ClientID, "Cognito Client ID for FeatureBase Cloud access.")
+	flags.StringVar(&cli.Region, "region", cli.Region, "Cloud region for FeatureBase Cloud access (e.g. us-east-2).")
+	flags.StringVar(&cli.Email, "email", cli.Email, "Email address for FeatureBase Cloud access.")
+	flags.StringVar(&cli.Password, "password", cli.Password, "Password for FeatureBase Cloud access.")
 
 	return cliCmd
 }

cmd/dax.go

@@ -0,0 +1,32 @@
+// Copyright 2021 Molecula Corp. All rights reserved.
+package cmd
+
+import (
+	"io"
+
+	"github.com/molecula/featurebase/v3/ctl"
+	"github.com/molecula/featurebase/v3/dax/server"
+	"github.com/pkg/errors"
+	"github.com/spf13/cobra"
+)
+
+// newDAXCommand runs the FeatureBase CLI subcommand for ingesting bulk data.
+func newDAXCommand(stdin io.Reader, stdout, stderr io.Writer) *cobra.Command {
+	server := server.NewCommand(stdin, stdout, stderr)
+	daxCmd := &cobra.Command{
+		Use:   "dax",
+		Short: "Run a collection of DAX services",
+		Long:  ``,
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if err := server.Start(); err != nil {
+				return errors.Wrap(err, "running server")
+			}
+			return errors.Wrap(server.Wait(), "waiting on Server")
+		},
+	}
+
+	// Attach flags to the command.
+	ctl.BuildDAXFlags(daxCmd, server)
+
+	return daxCmd
+}

cmd/import_test.go

@@ -6,7 +6,7 @@ import (
 	"strings"
 	"testing"
 
-	pilosa "github.com/featurebasedb/featurebase/v3"
+	pilosa "github.com/molecula/featurebase/v3"
 
 	"github.com/featurebasedb/featurebase/v3/cmd"
 	"github.com/featurebasedb/featurebase/v3/pql"

cmd/root.go

@@ -28,6 +28,9 @@ at https://docs.featurebase.com/.
 ` + pilosa.VersionInfo(true) + "\n",
 		PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
 			v := viper.New()
+			if cmd.Use == "dax" {
+				v.Set("future.rename", true) // always use FEATUREBASE env for dax
+			}
 			err := setAllConfig(v, cmd.Flags())
 			if err != nil {
 				return err
@@ -67,6 +70,7 @@ at https://docs.featurebase.com/.
 	rc.AddCommand(newHolderCmd(stdin, stdout, stderr))
 	rc.AddCommand(newKeygenCommand(stdin, stdout, stderr))
 	rc.AddCommand(newCLICommand(stdin, stdout, stderr))
+	rc.AddCommand(newDAXCommand(stdin, stdout, stderr))
 
 	rc.SetOutput(stderr)
 	return rc
@@ -118,10 +122,12 @@ func setAllConfig(v *viper.Viper, flags *pflag.FlagSet) error { // nolint: unpar
 
 		for _, key := range v.AllKeys() {
 			if _, ok := validTags[key]; !ok {
+				if key == "future.rename" {
+					continue
+				}
 				return fmt.Errorf("invalid option in configuration file: %v", key)
 			}
 		}
-
 	}
 
 	// set all values from viper

ctl/cli.go

@@ -1,6 +1,7 @@
 package ctl
 
 import (
+	"bytes"
 	"context"
 	"encoding/json"
 	"fmt"
@@ -14,6 +15,10 @@ import (
 	featurebase "github.com/featurebasedb/featurebase/v3"
 	"github.com/jedib0t/go-pretty/table"
 	"github.com/jedib0t/go-pretty/text"
+	featurebase "github.com/molecula/featurebase/v3"
+	"github.com/molecula/featurebase/v3/dax"
+	queryerhttp "github.com/molecula/featurebase/v3/dax/queryer/http"
+	"github.com/molecula/featurebase/v3/fbcloud"
 	"github.com/pkg/errors"
 )
 
@@ -36,8 +41,19 @@ type CLICommand struct {
 	Port        string `json:"port"`
 	HistoryPath string `json:"history-path"`
 
+	// Cloud Auth
+	ClientID string `json:"client-id"`
+	Region   string `json:"region"`
+	Email    string `json:"email"`
+	Password string `json:"password"`
+
 	// commands holds the list of sql commands to be executed.
 	commands []string
+
+	OrganizationID string `json:"org-id"`
+	DatabaseID     string `json:"db-id"`
+
+	queryer FBQueryer
 }
 
 func NewCLICommand(stdin io.Reader, stdout, stderr io.Writer) *CLICommand {
@@ -56,14 +72,147 @@ func NewCLICommand(stdin io.Reader, stdout, stderr io.Writer) *CLICommand {
 	}
 	return &CLICommand{
 		Host:        "localhost",
-		Port:        "10101",
 		HistoryPath: historyPath,
+
+		OrganizationID: "",
+		DatabaseID:     "",
 	}
 }
 
+// printQualifiers displays the currently set OrganizationID and DatabaseID.
+func (cmd *CLICommand) printQualifiers() {
+	fmt.Printf(" Host: %s\n  Org: %s\n   DB: %s\n",
+		hostPort(cmd.Host, cmd.Port),
+		cmd.OrganizationID,
+		cmd.DatabaseID,
+	)
+}
+
+func (cmd *CLICommand) setupClient() error {
+	if strings.TrimSpace(cmd.Host) == "" {
+		return errors.Errorf("no host provided")
+	}
+
+	if !strings.HasPrefix(cmd.Host, "http") {
+		cmd.Host = "http://" + cmd.Host
+	}
+
+	typ, err := cmd.detectFBType()
+	if err != nil {
+		return errors.Wrap(err, "detecting FeatureBase deployment type")
+	}
+
+	switch typ {
+	case featurebaseTypeStandard:
+		fmt.Println("Detected standard deployment")
+		cmd.queryer = &standardQueryer{
+			Host: cmd.Host,
+			Port: cmd.Port,
+		}
+	case featurebaseTypeDAX:
+		fmt.Println("Detected dax deployment")
+		cmd.queryer = &daxQueryer{
+			Host: cmd.Host,
+			Port: cmd.Port,
+		}
+	case featurebaseTypeCloud:
+		fmt.Println("Detected cloud deployment")
+		cmd.queryer = &fbcloud.Queryer{
+			Host: cmd.Host,
+
+			ClientID: cmd.ClientID,
+			Region:   cmd.Region,
+			Email:    cmd.Email,
+			Password: cmd.Password,
+		}
+	default:
+		return errors.Errorf("unknown type: %s", typ)
+	}
+	return nil
+}
+
+type featurebaseType string
+
+const (
+	featurebaseTypeStandard featurebaseType = "standard"
+	featurebaseTypeDAX      featurebaseType = "dax"
+	featurebaseTypeCloud    featurebaseType = "cloud"
+)
+
+func hostPort(host, port string) string {
+	if port == "" {
+		return host
+	}
+	return host + ":" + port
+}
+
+// detectFBType determines if we're talking to standalone FeatureBase
+// or FeatureBase Cloud
+func (cmd *CLICommand) detectFBType() (featurebaseType, error) {
+	type trial struct {
+		port   string
+		health string
+		typ    featurebaseType
+	}
+
+	// trials is populated with the url/endpoints to try in order to detect if a
+	// process is running there which can support the cli requests.
+	trials := []trial{}
+
+	if cmd.Port != "" {
+		trials = append(trials,
+			// dax
+			trial{
+				port:   cmd.Port,
+				health: "/queryer/health",
+				typ:    featurebaseTypeDAX,
+			},
+			// standard
+			trial{
+				port:   cmd.Port,
+				health: "/status",
+				typ:    featurebaseTypeStandard,
+			},
+		)
+	} else {
+		// Try default ports just in case.
+		trials = append(trials,
+			// dax
+			trial{
+				port:   "8080",
+				health: "/queryer/health",
+				typ:    featurebaseTypeDAX,
+			},
+			// standard
+			trial{
+				port:   "10101",
+				health: "/status",
+				typ:    featurebaseTypeStandard,
+			},
+		)
+	}
+
+	for _, trial := range trials {
+		url := hostPort(cmd.Host, trial.port) + trial.health
+		if resp, err := http.Get(url); err != nil {
+			continue
+		} else if resp.StatusCode/100 == 2 {
+			cmd.Port = trial.port
+			return trial.typ, nil
+		}
+	}
+
+	return featurebaseTypeCloud, nil
+}
+
 func (cmd *CLICommand) Run(ctx context.Context) error {
 	// Print the splash message.
 	fmt.Print(splash)
+	err := cmd.setupClient()
+	if err != nil {
+		return errors.Wrap(err, "setting up client")
+	}
+	cmd.printQualifiers()
 
 	rl, err := readline.NewEx(&readline.Config{
 		Prompt:                 promptBegin,
@@ -76,10 +225,6 @@ func (cmd *CLICommand) Run(ctx context.Context) error {
 	}
 	defer rl.Close()
 
-	if !strings.HasPrefix(cmd.Host, "http") {
-		cmd.Host = "http://" + cmd.Host
-	}
-
 	// partialCommand holds all input prior to receiving a termination
 	// character.
 	var partialCommand string
@@ -174,6 +319,10 @@ func appendCommand(orig string, part string) string {
 	}
 }
 
+type FBQueryer interface {
+	Query(org, db, sql string) (*featurebase.SQLResponse, error)
+}
+
 func (cmd *CLICommand) executeCommands(ctx context.Context) error {
 	// Clear out the buffered commands on any exit from this method.
 	defer func() {
@@ -181,23 +330,19 @@ func (cmd *CLICommand) executeCommands(ctx context.Context) error {
 	}()
 
 	for _, sql := range cmd.commands {
-		resp, err := http.Post(fmt.Sprintf("%s:%s/sql", cmd.Host, cmd.Port), "application/sql", strings.NewReader(sql))
+		// Handle non-sql commands (for example, SET commands).
-		if err != nil {
+		if handled, err := cmd.handleIfNonSQLCommand(ctx, sql); err != nil {
-			return errors.Wrapf(err, "posting query")
+			return errors.Wrapf(err, "handling non-SQL command: %s", sql)
+		} else if handled {
+			continue
 		}
 
-		var sqlResponse response
+		sqlResponse, err := cmd.queryer.Query(cmd.OrganizationID, cmd.DatabaseID, sql)
-		fullbod, err := io.ReadAll(resp.Body)
 		if err != nil {
-			return errors.Wrap(err, "reading response")
+			fmt.Printf("making query: %v\n", err)
+			continue
 		}
-		err = json.Unmarshal(fullbod, &sqlResponse)
+		err = WriteOut(sqlResponse, os.Stdout)
-		if err != nil {
-			fmt.Printf("couldn't decode response: %v\n", err)
-			fmt.Printf("%s\n", fullbod)
-		}
-
-		err = sqlResponse.WriteOut(os.Stdout)
 		if err != nil {
 			return errors.Wrap(err, "writing out response")
 		}
@@ -206,15 +351,58 @@ func (cmd *CLICommand) executeCommands(ctx context.Context) error {
 	return nil
 }
 
-type response struct {
+// handleIfNonSQLCommand will handle special case command like "SET ..." and
-	Schema        featurebase.SQLSchema `json:"schema"`
+// "USE ...". If the sql command matches one of these conditions and is handled,
-	Data          [][]interface{}       `json:"data"`
+// the bool returned will be true;
-	Error         string                `json:"error"`
+func (cmd *CLICommand) handleIfNonSQLCommand(ctx context.Context, sql string) (bool, error) {
-	Warnings      []string              `json:"warnings"`
+	var handled bool
-	ExecutionTime int64                 `json:"exec_time"`
+
+	// Get the first token from the SQL:
+	parts := strings.Split(sql, " ")
+	if len(parts) < 1 {
+		return handled, nil
+	}
+
+	token := strings.ToUpper(parts[0])
+
+	// Supported:
+	// SET ORG acme
+	// SET DB db1
+	// USE db1
+	switch token {
+	case "SET":
+		handled = true
+		switch len(parts) {
+		case 1:
+			// This will fall through and just print the qualifiers.
+		case 3:
+			switch strings.ToUpper(parts[1]) {
+			case "HOST":
+				cmd.Host = parts[2]
+			case "ORG":
+				cmd.OrganizationID = parts[2]
+			case "DB":
+				cmd.DatabaseID = parts[2]
+			}
+		default:
+			return handled, errors.Errorf("SET command takes a name and a value (SET DB db1)")
+		}
+
+	case "USE":
+		handled = true
+		if len(parts) != 2 {
+			return handled, errors.Errorf("USE command takes a single value (USE db1)")
+		}
+		cmd.DatabaseID = parts[1]
+	default:
+		return handled, nil
+	}
+
+	cmd.printQualifiers()
+	return handled, nil
 }
 
-func (r *response) WriteWarnings(w io.Writer) error {
+func WriteWarnings(r *featurebase.SQLResponse, w io.Writer) error {
 	if len(r.Warnings) > 0 {
 		if _, err := w.Write([]byte("\n")); err != nil {
 			return errors.Wrapf(err, "writing warning: %s", r.Error)
@@ -228,12 +416,15 @@ func (r *response) WriteWarnings(w io.Writer) error {
 	return nil
 }
 
-func (r *response) WriteOut(w io.Writer) error {
+func WriteOut(r *featurebase.SQLResponse, w io.Writer) error {
+	if r == nil {
+		return errors.New("attempt to write out nil response")
+	}
 	if r.Error != "" {
 		if _, err := w.Write([]byte("Error: " + r.Error + "\n")); err != nil {
 			return errors.Wrapf(err, "writing error: %s", r.Error)
 		}
-		return r.WriteWarnings(w)
+		return WriteWarnings(r, w)
 	}
 
 	t := table.NewWriter()
@@ -255,7 +446,7 @@ func (r *response) WriteOut(w io.Writer) error {
 	}
 	t.Render()
 
-	err := r.WriteWarnings(w)
+	err := WriteWarnings(r, w)
 	if err != nil {
 		return err
 	}
@@ -282,3 +473,77 @@ func schemaToRow(schema featurebase.SQLSchema) []interface{} {
 	}
 	return ret
 }
+
+// Ensure type implements interface.
+var _ FBQueryer = (*standardQueryer)(nil)
+
+// standardQueryer supports a standard featurebase deployment hitting the /sql
+// endpoint with a payload containing only the sql statement.
+type standardQueryer struct {
+	Host string
+	Port string
+}
+
+func (qryr *standardQueryer) Query(org, db, sql string) (*featurebase.SQLResponse, error) {
+	buf := bytes.Buffer{}
+	url := fmt.Sprintf("%s/sql", hostPort(qryr.Host, qryr.Port))
+
+	buf.Write([]byte(sql))
+
+	resp, err := http.Post(url, "application/json", &buf)
+	if err != nil {
+		return nil, errors.Wrapf(err, "posting query")
+	}
+
+	fullbod, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return nil, errors.Wrap(err, "reading response")
+	}
+	sqlResponse := &featurebase.SQLResponse{}
+	if err := json.Unmarshal(fullbod, sqlResponse); err != nil {
+		return nil, errors.Wrapf(err, "unmarshaling query response, body:\n'%s'\n", fullbod)
+	}
+
+	return sqlResponse, nil
+}
+
+// Ensure type implements interface.
+var _ FBQueryer = (*daxQueryer)(nil)
+
+// daxQueryer is similar to the standardQueryer except that it hits a different
+// endpoint, and its payload is a json object which includes, in addition to the
+// sql statement, things like org and db.
+type daxQueryer struct {
+	Host string
+	Port string
+}
+
+func (qryr *daxQueryer) Query(org, db, sql string) (*featurebase.SQLResponse, error) {
+	buf := bytes.Buffer{}
+	url := fmt.Sprintf("%s/queryer/sql", hostPort(qryr.Host, qryr.Port))
+
+	sqlReq := &queryerhttp.SQLRequest{
+		OrganizationID: dax.OrganizationID(org),
+		DatabaseID:     dax.DatabaseID(db),
+		SQL:            sql,
+	}
+	if err := json.NewEncoder(&buf).Encode(sqlReq); err != nil {
+		return nil, errors.Wrapf(err, "encoding sql request: %s", sql)
+	}
+
+	resp, err := http.Post(url, "application/json", &buf)
+	if err != nil {
+		return nil, errors.Wrapf(err, "posting query")
+	}
+
+	fullbod, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return nil, errors.Wrap(err, "reading response")
+	}
+	sqlResponse := &featurebase.SQLResponse{}
+	if err := json.Unmarshal(fullbod, sqlResponse); err != nil {
+		return nil, errors.Wrapf(err, "unmarshaling query response, body:\n'%s'\n", fullbod)
+	}
+
+	return sqlResponse, nil
+}

ctl/dax.go

@@ -0,0 +1,39 @@
+package ctl
+
+import (
+	"github.com/molecula/featurebase/v3/dax/server"
+	"github.com/spf13/cobra"
+)
+
+// BuildDAXFlags attaches a set of flags to the command for a server instance.
+func BuildDAXFlags(cmd *cobra.Command, srv *server.Command) {
+	flags := cmd.Flags()
+
+	flags.StringVarP(&srv.Config.Bind, "bind", "b", srv.Config.Bind, "Default URI on which this service should listen.")
+	flags.StringVar(&srv.Config.Advertise, "advertise", srv.Config.Advertise, "Address to advertise externally.")
+	flags.BoolVar(&srv.Config.Verbose, "verbose", srv.Config.Verbose, "Enable verbose logging")
+	flags.StringVar(&srv.Config.LogPath, "log-path", srv.Config.LogPath, "Log path")
+
+	flags.StringVar(&srv.Config.StorageMethod, "storage-method", srv.Config.StorageMethod, "Method to use for persistent storage.")
+	flags.StringVar(&srv.Config.StorageDSN, "storage-dsn", srv.Config.StorageDSN, "Datasource Name when using an applicable storage method.")
+
+	// MDS
+	flags.BoolVar(&srv.Config.MDS.Run, "mds.run", srv.Config.MDS.Run, "Run the MDS service in process.")
+	flags.DurationVar(&srv.Config.MDS.Config.RegistrationBatchTimeout, "mds.config.registration-batch-timeout", srv.Config.MDS.Config.RegistrationBatchTimeout, "Timeout for node registration batches.")
+
+	// WriteLogger
+	flags.BoolVar(&srv.Config.WriteLogger.Run, "writelogger.run", srv.Config.WriteLogger.Run, "Run the WriteLogger service in process.")
+	flags.StringVar(&srv.Config.WriteLogger.Config.DataDir, "writelogger.config.data-dir", srv.Config.WriteLogger.Config.DataDir, "WriteLogger directory to use in process.")
+
+	// Snapshotter
+	flags.BoolVar(&srv.Config.Snapshotter.Run, "snapshotter.run", srv.Config.Snapshotter.Run, "Run the Snapshotter service in process.")
+	flags.StringVar(&srv.Config.Snapshotter.Config.DataDir, "snapshotter.config.data-dir", srv.Config.Snapshotter.Config.DataDir, "Snapshotter directory to use in process.")
+
+	// Queryer
+	flags.BoolVar(&srv.Config.Queryer.Run, "queryer.run", srv.Config.Queryer.Run, "Run the Queryer service in process.")
+	flags.StringVar(&srv.Config.Queryer.Config.MDSAddress, "queryer.config.mds-address", srv.Config.Queryer.Config.MDSAddress, "Address of remote MDS process.")
+
+	// Computer
+	flags.BoolVar(&srv.Config.Computer.Run, "computer.run", srv.Config.Computer.Run, "Run the Computer service in process.")
+	flags.AddFlagSet(serverFlagSet(&srv.Config.Computer.Config, "computer.config"))
+}

ctl/server.go

@@ -8,114 +8,146 @@ import (
 	"github.com/featurebasedb/featurebase/v3/server"
 	"github.com/featurebasedb/featurebase/v3/storage"
 	"github.com/spf13/cobra"
+	"github.com/spf13/pflag"
 )
 
+// serverFlagSet returns a pflag.FlagSet. All flags will be prefixed with the
+// given prefix value, and default values come from the provided server.Config.
+func serverFlagSet(srv *server.Config, prefix string) *pflag.FlagSet {
+	// pre applies prefix to s when a prefix is provided.
+	pre := func(s string) string {
+		if prefix == "" {
+			return s
+		}
+		return prefix + "." + s
+	}
+
+	// short will pass through the short flag as long as a prefix is not
+	// specified.
+	short := func(s string) string {
+		if prefix == "" {
+			return s
+		}
+		return ""
+	}
+
+	flags := pflag.NewFlagSet("featurebase", pflag.ExitOnError)
+	flags.StringVar(&srv.Name, pre("name"), srv.Name, "Name of the node in the cluster.")
+	flags.StringVar(&srv.MDSAddress, pre("mds-address"), srv.MDSAddress, "MDS service to register with.")
+	flags.StringVar(&srv.WriteLogger, pre("write-logger"), srv.WriteLogger, "WriteLogger to read/write append logs.")
+	flags.StringVar(&srv.Snapshotter, pre("snapshotter"), srv.Snapshotter, "Snapshotter to read/write snapshots.")
+	flags.StringVarP(&srv.DataDir, pre("data-dir"), short("d"), srv.DataDir, "Directory to store FeatureBase data files.")
+	flags.StringVarP(&srv.Bind, pre("bind"), short("b"), srv.Bind, "Default URI on which FeatureBase should listen.")
+	flags.StringVar(&srv.BindGRPC, pre("bind-grpc"), srv.BindGRPC, "URI on which FeatureBase should listen for gRPC requests.")
+	flags.StringVar(&srv.Advertise, pre("advertise"), srv.Advertise, "Address to advertise externally.")
+	flags.StringVar(&srv.AdvertiseGRPC, pre("advertise-grpc"), srv.AdvertiseGRPC, "Address to advertise externally for gRPC.")
+	flags.IntVar(&srv.MaxWritesPerRequest, pre("max-writes-per-request"), srv.MaxWritesPerRequest, "Number of write commands per request.")
+	flags.StringVar(&srv.LogPath, pre("log-path"), srv.LogPath, "Log path")
+	flags.BoolVar(&srv.Verbose, pre("verbose"), srv.Verbose, "Enable verbose logging")
+	flags.Uint64Var(&srv.MaxMapCount, pre("max-map-count"), srv.MaxMapCount, "Limits the maximum number of active mmaps. FeatureBase will fall back to reading files once this is exhausted. Set below your system's vm.max_map_count.")
+	flags.Uint64Var(&srv.MaxFileCount, pre("max-file-count"), srv.MaxFileCount, "Soft limit on the maximum number of fragment files FeatureBase keeps open simultaneously.")
+	flags.DurationVar((*time.Duration)(&srv.LongQueryTime), pre("long-query-time"), time.Duration(srv.LongQueryTime), "Duration that will trigger log and stat messages for slow queries. Zero to disable.")
+	flags.IntVar(&srv.QueryHistoryLength, pre("query-history-length"), srv.QueryHistoryLength, "Number of queries to remember in history.")
+	flags.Int64Var(&srv.MaxQueryMemory, pre("max-query-memory"), srv.MaxQueryMemory, "Maximum memory allowed per Extract() or SELECT query.")
+
+	// TLS
+	SetTLSConfig(flags, pre(""), &srv.TLS.CertificatePath, &srv.TLS.CertificateKeyPath, &srv.TLS.CACertPath, &srv.TLS.SkipVerify, &srv.TLS.EnableClientVerification)
+
+	// Handler
+	flags.StringSliceVar(&srv.Handler.AllowedOrigins, pre("handler.allowed-origins"), []string{}, "Comma separated list of allowed origin URIs (for CORS/Web UI).")
+
+	// Cluster
+	flags.IntVar(&srv.Cluster.ReplicaN, pre("cluster.replicas"), 1, "Number of hosts each piece of data should be stored on.")
+	flags.DurationVar((*time.Duration)(&srv.Cluster.LongQueryTime), pre("cluster.long-query-time"), time.Duration(srv.Cluster.LongQueryTime), "RENAMED TO 'long-query-time': Duration that will trigger log and stat messages for slow queries.") // negative duration indicates invalid value because 0 is meaningful
+	flags.StringVar(&srv.Cluster.Name, pre("cluster.name"), srv.Cluster.Name, "Human-readable name for the cluster.")
+	flags.StringVar(&srv.Cluster.PartitionToNodeAssignment, pre("cluster.partition-to-node-assignment"), srv.Cluster.PartitionToNodeAssignment, "How to assign partitions to nodes. jmp-hash or modulus")
+
+	// Translation
+	flags.StringVar(&srv.Translation.PrimaryURL, pre("translation.primary-url"), srv.Translation.PrimaryURL, "DEPRECATED: URL for primary translation node for replication.")
+	flags.IntVar(&srv.Translation.MapSize, pre("translation.map-size"), srv.Translation.MapSize, "Size in bytes of mmap to allocate for key translation.")
+
+	// Etcd
+	// Etcd.Name used Config.Name for its value.
+	flags.StringVar(&srv.Etcd.Dir, pre("etcd.dir"), srv.Etcd.Dir, "Directory to store etcd data files. If not provided, a directory will be created under the main data-dir directory.")
+	// Etcd.ClusterName uses Cluster.Name for its value
+	flags.StringVar(&srv.Etcd.LClientURL, pre("etcd.listen-client-address"), srv.Etcd.LClientURL, "Listen client address.")
+	flags.StringVar(&srv.Etcd.AClientURL, pre("etcd.advertise-client-address"), srv.Etcd.AClientURL, "Advertise client address. If not provided, uses the listen client address.")
+	flags.StringVar(&srv.Etcd.LPeerURL, pre("etcd.listen-peer-address"), srv.Etcd.LPeerURL, "Listen peer address.")
+	flags.StringVar(&srv.Etcd.APeerURL, pre("etcd.advertise-peer-address"), srv.Etcd.APeerURL, "Advertise peer address. If not provided, uses the listen peer address.")
+	flags.StringVar(&srv.Etcd.ClusterURL, pre("etcd.cluster-url"), srv.Etcd.ClusterURL, "Cluster URL to join.")
+	flags.StringVar(&srv.Etcd.InitCluster, pre("etcd.initial-cluster"), srv.Etcd.InitCluster, "Initial cluster name1=apurl1,name2=apurl2")
+	flags.Int64Var(&srv.Etcd.HeartbeatTTL, pre("etcd.heartbeat-ttl"), srv.Etcd.HeartbeatTTL, "Timeout used to determine cluster status")
+
+	flags.StringVar(&srv.Etcd.Cluster, "etcd.static-cluster", srv.Etcd.Cluster, "EXPERIMENTAL static featurebase cluster name1=apurl1,name2=apurl2")
+	flags.MarkHidden("etcd.static-cluster")
+	flags.StringVar(&srv.Etcd.EtcdHosts, "etcd.etcd-hosts", srv.Etcd.EtcdHosts, "EXPERIMENTAL etcd server host:port comma separated list")
+	flags.MarkHidden("etcd.etcd-hosts") // TODO (twg) expose when ready for public consumption
+
+	// External postgres database for ExternalLookup
+	flags.StringVar(&srv.LookupDBDSN, pre("lookup-db-dsn"), "", "external (postgres) database DSN to use for ExternalLookup calls")
+
+	// AntiEntropy
+	flags.DurationVar((*time.Duration)(&srv.AntiEntropy.Interval), pre("anti-entropy.interval"), (time.Duration)(srv.AntiEntropy.Interval), "Interval at which to run anti-entropy routine.")
+
+	// Metric
+	flags.StringVar(&srv.Metric.Service, pre("metric.service"), srv.Metric.Service, "Where to send stats: can be expvar (in-memory served at /debug/vars), prometheus, statsd or none.")
+	flags.StringVar(&srv.Metric.Host, pre("metric.host"), srv.Metric.Host, "URI to send metrics when metric.service is statsd.")
+	flags.DurationVar((*time.Duration)(&srv.Metric.PollInterval), pre("metric.poll-interval"), (time.Duration)(srv.Metric.PollInterval), "Polling interval metrics.")
+	flags.BoolVar((&srv.Metric.Diagnostics), pre("metric.diagnostics"), srv.Metric.Diagnostics, "Enabled diagnostics reporting.")
+
+	// Tracing
+	flags.StringVar(&srv.Tracing.AgentHostPort, pre("tracing.agent-host-port"), srv.Tracing.AgentHostPort, "Jaeger agent host:port.")
+	flags.StringVar(&srv.Tracing.SamplerType, pre("tracing.sampler-type"), srv.Tracing.SamplerType, "Jaeger sampler type (remote, const, probabilistic, ratelimiting) or 'off' to disable tracing completely.")
+	flags.Float64Var(&srv.Tracing.SamplerParam, pre("tracing.sampler-param"), srv.Tracing.SamplerParam, "Jaeger sampler parameter.")
+
+	// Profiling
+	flags.IntVar(&srv.Profile.BlockRate, pre("profile.block-rate"), srv.Profile.BlockRate, "Sampling rate for goroutine blocking profiler. One sample per <rate> ns.")
+	flags.IntVar(&srv.Profile.MutexFraction, pre("profile.mutex-fraction"), srv.Profile.MutexFraction, "Sampling fraction for mutex contention profiling. Sample 1/<rate> of events.")
+
+	flags.StringVar(&srv.Storage.Backend, pre("storage.backend"), storage.DefaultBackend, "Storage backend to use: 'rbf' is only supported value.")
+	flags.BoolVar(&srv.Storage.FsyncEnabled, pre("storage.fsync"), true, "enable fsync fully safe flush-to-disk")
+
+	// RBF specific flags. See pilosa/rbf/cfg/cfg.go for definitions.
+	srv.RBFConfig.DefineFlags(flags, prefix)
+
+	flags.BoolVar(&srv.SQL.EndpointEnabled, pre("sql.endpoint-enabled"), srv.SQL.EndpointEnabled, "Enable FeatureBase SQL /sql endpoint (default false)")
+
+	flags.DurationVar(&srv.CheckInInterval, pre("check-in-interval"), srv.CheckInInterval, "Interval between check-ins to MDS")
+
+	// Future flags.
+	flags.BoolVar(&srv.Future.Rename, pre("future.rename"), false, "Present application name as FeatureBase. Defaults to false, will default to true in an upcoming release.")
+
+	// OAuth2.0 identity provider configuration
+	flags.BoolVar(&srv.Auth.Enable, pre("auth.enable"), false, "Enable AuthN/AuthZ of featurebase, disabled by default.")
+	flags.StringVar(&srv.Auth.ClientId, pre("auth.client-id"), srv.Auth.ClientId, "Identity Provider's Application/Client ID.")
+	flags.StringVar(&srv.Auth.ClientSecret, pre("auth.client-secret"), srv.Auth.ClientSecret, "Identity Provider's Client Secret.")
+	flags.StringVar(&srv.Auth.AuthorizeURL, pre("auth.authorize-url"), srv.Auth.AuthorizeURL, "Identity Provider's Authorize URL.")
+	flags.StringVar(&srv.Auth.RedirectBaseURL, pre("auth.redirect-base-url"), srv.Auth.RedirectBaseURL, "Base URL of the featurebase instance used to redirect IDP.")
+	flags.StringVar(&srv.Auth.TokenURL, pre("auth.token-url"), srv.Auth.TokenURL, "Identity Provider's Token URL.")
+	flags.StringVar(&srv.Auth.GroupEndpointURL, pre("auth.group-endpoint-url"), srv.Auth.GroupEndpointURL, "Identity Provider's Group endpoint URL.")
+	flags.StringVar(&srv.Auth.LogoutURL, pre("auth.logout-url"), srv.Auth.LogoutURL, "Identity Provider's Logout URL.")
+	flags.StringSliceVar(&srv.Auth.Scopes, pre("auth.scopes"), srv.Auth.Scopes, "Comma separated list of scopes obtained from IdP")
+	flags.StringVar(&srv.Auth.SecretKey, pre("auth.secret-key"), srv.Auth.SecretKey, "Secret key used for auth.")
+	flags.StringVar(&srv.Auth.PermissionsFile, pre("auth.permissions"), srv.Auth.PermissionsFile, "Permissions' file with group authorization.")
+	flags.StringVar(&srv.Auth.QueryLogPath, pre("auth.query-log-path"), srv.Auth.QueryLogPath, "Path to log user queries")
+	flags.StringSliceVar(&srv.Auth.ConfiguredIPs, pre("auth.configured-ips"), srv.Auth.ConfiguredIPs, "List of configured IPs allowed for ingest")
+
+	flags.BoolVar(&srv.DataDog.Enable, pre("datadog.enable"), false, "enable continuous profiling with DataDog cloud service, Note you must have DataDog agent installed")
+	flags.StringVar(&srv.DataDog.Service, pre("datadog.service"), "default-service", "The Datadog service name, for example my-web-app")
+	flags.StringVar(&srv.DataDog.Env, pre("datadog.env"), "default-env", "The Datadog environment name, for example, production")
+	flags.StringVar(&srv.DataDog.Version, pre("datadog.version"), "default-version", "The version of your application")
+	flags.StringVar(&srv.DataDog.Tags, pre("datadog.tags"), "molecula", "The tags to apply to an uploaded profile. Must be a list of in the format <KEY1>:<VALUE1>,<KEY2>:<VALUE2>")
+	flags.BoolVar(&srv.DataDog.CPUProfile, pre("datadog.cpu-profile"), true, "golang pprof cpu profile ")
+	flags.BoolVar(&srv.DataDog.HeapProfile, pre("datadog.heap-profile"), true, "golang pprof heap profile")
+	flags.BoolVar(&srv.DataDog.MutexProfile, pre("datadog.mutex-profile"), false, "golang pprof mutex profile")
+	flags.BoolVar(&srv.DataDog.GoroutineProfile, pre("datadog.goroutine-profile"), false, "golang pprof goroutine profile")
+	flags.BoolVar(&srv.DataDog.BlockProfile, pre("datadog.block-profile"), false, "golang pprof goroutine ")
+
+	return flags
+}
+
 // BuildServerFlags attaches a set of flags to the command for a server instance.
 func BuildServerFlags(cmd *cobra.Command, srv *server.Command) {
 	flags := cmd.Flags()
-	flags.StringVar(&srv.Config.Name, "name", srv.Config.Name, "Name of the node in the cluster.")
+	flags.AddFlagSet(serverFlagSet(srv.Config, ""))
-	flags.StringVarP(&srv.Config.DataDir, "data-dir", "d", srv.Config.DataDir, "Directory to store FeatureBase data files.")
-	flags.StringVarP(&srv.Config.Bind, "bind", "b", srv.Config.Bind, "Default URI on which FeatureBase should listen.")
-	flags.StringVar(&srv.Config.BindGRPC, "bind-grpc", srv.Config.BindGRPC, "URI on which FeatureBase should listen for gRPC requests.")
-	flags.StringVar(&srv.Config.Advertise, "advertise", srv.Config.Advertise, "Address to advertise externally.")
-	flags.StringVar(&srv.Config.AdvertiseGRPC, "advertise-grpc", srv.Config.AdvertiseGRPC, "Address to advertise externally for gRPC.")
-	flags.IntVar(&srv.Config.MaxWritesPerRequest, "max-writes-per-request", srv.Config.MaxWritesPerRequest, "Number of write commands per request.")
-	flags.StringVar(&srv.Config.LogPath, "log-path", srv.Config.LogPath, "Log path")
-	flags.BoolVar(&srv.Config.Verbose, "verbose", srv.Config.Verbose, "Enable verbose logging")
-	flags.Uint64Var(&srv.Config.MaxMapCount, "max-map-count", srv.Config.MaxMapCount, "Limits the maximum number of active mmaps. FeatureBase will fall back to reading files once this is exhausted. Set below your system's vm.max_map_count.")
-	flags.Uint64Var(&srv.Config.MaxFileCount, "max-file-count", srv.Config.MaxFileCount, "Soft limit on the maximum number of fragment files FeatureBase keeps open simultaneously.")
-	flags.DurationVar((*time.Duration)(&srv.Config.LongQueryTime), "long-query-time", time.Duration(srv.Config.LongQueryTime), "Duration that will trigger log and stat messages for slow queries. Zero to disable.")
-	flags.IntVar(&srv.Config.QueryHistoryLength, "query-history-length", srv.Config.QueryHistoryLength, "Number of queries to remember in history.")
-	flags.Int64Var(&srv.Config.MaxQueryMemory, "max-query-memory", srv.Config.MaxQueryMemory, "Maximum memory allowed per Extract() or SELECT query.")
-
-	// TLS
-	SetTLSConfig(flags, "", &srv.Config.TLS.CertificatePath, &srv.Config.TLS.CertificateKeyPath, &srv.Config.TLS.CACertPath, &srv.Config.TLS.SkipVerify, &srv.Config.TLS.EnableClientVerification)
-
-	// Handler
-	flags.StringSliceVar(&srv.Config.Handler.AllowedOrigins, "handler.allowed-origins", []string{}, "Comma separated list of allowed origin URIs (for CORS/Web UI).")
-
-	// Cluster
-	flags.IntVar(&srv.Config.Cluster.ReplicaN, "cluster.replicas", 1, "Number of hosts each piece of data should be stored on.")
-	flags.DurationVar((*time.Duration)(&srv.Config.Cluster.LongQueryTime), "cluster.long-query-time", time.Duration(srv.Config.Cluster.LongQueryTime), "RENAMED TO 'long-query-time': Duration that will trigger log and stat messages for slow queries.") // negative duration indicates invalid value because 0 is meaningful
-	flags.StringVar(&srv.Config.Cluster.Name, "cluster.name", srv.Config.Cluster.Name, "Human-readable name for the cluster.")
-	flags.StringVar(&srv.Config.Cluster.PartitionToNodeAssignment, "cluster.partition-to-node-assignment", srv.Config.Cluster.PartitionToNodeAssignment, "How to assign partitions to nodes. jmp-hash or modulus")
-
-	// Translation
-	flags.StringVar(&srv.Config.Translation.PrimaryURL, "translation.primary-url", srv.Config.Translation.PrimaryURL, "DEPRECATED: URL for primary translation node for replication.")
-	flags.IntVar(&srv.Config.Translation.MapSize, "translation.map-size", srv.Config.Translation.MapSize, "Size in bytes of mmap to allocate for key translation.")
-
-	// Etcd
-	// Etcd.Name used Config.Name for its value.
-	flags.StringVar(&srv.Config.Etcd.Dir, "etcd.dir", srv.Config.Etcd.Dir, "Directory to store etcd data files. If not provided, a directory will be created under the main data-dir directory.")
-	// Etcd.ClusterName uses Cluster.Name for its value
-	flags.StringVar(&srv.Config.Etcd.LClientURL, "etcd.listen-client-address", srv.Config.Etcd.LClientURL, "Listen client address.")
-	flags.StringVar(&srv.Config.Etcd.AClientURL, "etcd.advertise-client-address", srv.Config.Etcd.AClientURL, "Advertise client address. If not provided, uses the listen client address.")
-	flags.StringVar(&srv.Config.Etcd.LPeerURL, "etcd.listen-peer-address", srv.Config.Etcd.LPeerURL, "Listen peer address.")
-	flags.StringVar(&srv.Config.Etcd.APeerURL, "etcd.advertise-peer-address", srv.Config.Etcd.APeerURL, "Advertise peer address. If not provided, uses the listen peer address.")
-	flags.StringVar(&srv.Config.Etcd.ClusterURL, "etcd.cluster-url", srv.Config.Etcd.ClusterURL, "Cluster URL to join.")
-	flags.StringVar(&srv.Config.Etcd.InitCluster, "etcd.initial-cluster", srv.Config.Etcd.InitCluster, "Initial cluster name1=apurl1,name2=apurl2")
-	flags.Int64Var(&srv.Config.Etcd.HeartbeatTTL, "etcd.heartbeat-ttl", srv.Config.Etcd.HeartbeatTTL, "Timeout used to determine cluster status")
-
-	flags.StringVar(&srv.Config.Etcd.Cluster, "etcd.static-cluster", srv.Config.Etcd.Cluster, "EXPERIMENTAL static featurebase cluster name1=apurl1,name2=apurl2")
-	_ = flags.MarkHidden("etcd.static-cluster")
-	flags.StringVar(&srv.Config.Etcd.EtcdHosts, "etcd.etcd-hosts", srv.Config.Etcd.EtcdHosts, "EXPERIMENTAL etcd server host:port comma separated list")
-	_ = flags.MarkHidden("etcd.etcd-hosts") // TODO (twg) expose when ready for public consumption
-
-	// External postgres database for ExternalLookup
-	flags.StringVar(&srv.Config.LookupDBDSN, "lookup-db-dsn", "", "external (postgres) database DSN to use for ExternalLookup calls")
-
-	// AntiEntropy
-	flags.DurationVar((*time.Duration)(&srv.Config.AntiEntropy.Interval), "anti-entropy.interval", (time.Duration)(srv.Config.AntiEntropy.Interval), "Interval at which to run anti-entropy routine.")
-
-	// Metric
-	flags.StringVar(&srv.Config.Metric.Service, "metric.service", srv.Config.Metric.Service, "Where to send stats: can be expvar (in-memory served at /debug/vars), prometheus, statsd or none.")
-	flags.StringVar(&srv.Config.Metric.Host, "metric.host", srv.Config.Metric.Host, "URI to send metrics when metric.service is statsd.")
-	flags.DurationVar((*time.Duration)(&srv.Config.Metric.PollInterval), "metric.poll-interval", (time.Duration)(srv.Config.Metric.PollInterval), "Polling interval metrics.")
-	flags.BoolVar((&srv.Config.Metric.Diagnostics), "metric.diagnostics", srv.Config.Metric.Diagnostics, "Enabled diagnostics reporting.")
-
-	// Tracing
-	flags.StringVar(&srv.Config.Tracing.AgentHostPort, "tracing.agent-host-port", srv.Config.Tracing.AgentHostPort, "Jaeger agent host:port.")
-	flags.StringVar(&srv.Config.Tracing.SamplerType, "tracing.sampler-type", srv.Config.Tracing.SamplerType, "Jaeger sampler type (remote, const, probabilistic, ratelimiting) or 'off' to disable tracing completely.")
-	flags.Float64Var(&srv.Config.Tracing.SamplerParam, "tracing.sampler-param", srv.Config.Tracing.SamplerParam, "Jaeger sampler parameter.")
-
-	// Profiling
-	flags.IntVar(&srv.Config.Profile.BlockRate, "profile.block-rate", srv.Config.Profile.BlockRate, "Sampling rate for goroutine blocking profiler. One sample per <rate> ns.")
-	flags.IntVar(&srv.Config.Profile.MutexFraction, "profile.mutex-fraction", srv.Config.Profile.MutexFraction, "Sampling fraction for mutex contention profiling. Sample 1/<rate> of events.")
-
-	flags.StringVar(&srv.Config.Storage.Backend, "storage.backend", storage.DefaultBackend, "Storage backend to use: 'rbf' is only supported value.")
-	flags.BoolVar(&srv.Config.Storage.FsyncEnabled, "storage.fsync", true, "enable fsync fully safe flush-to-disk")
-
-	// RBF specific flags. See pilosa/rbf/cfg/cfg.go for definitions.
-	srv.Config.RBFConfig.DefineFlags(flags)
-
-	flags.BoolVar(&srv.Config.SQL.EndpointEnabled, "sql.endpoint-enabled", srv.Config.SQL.EndpointEnabled, "Enable FeatureBase SQL /sql endpoint (default false)")
-
-	// Future flags.
-	flags.BoolVar(&srv.Config.Future.Rename, "future.rename", false, "Present application name as FeatureBase. Defaults to false, will default to true in an upcoming release.")
-
-	// OAuth2.0 identity provider configuration
-	flags.BoolVar(&srv.Config.Auth.Enable, "auth.enable", false, "Enable AuthN/AuthZ of featurebase, disabled by default.")
-	flags.StringVar(&srv.Config.Auth.ClientId, "auth.client-id", srv.Config.Auth.ClientId, "Identity Provider's Application/Client ID.")
-	flags.StringVar(&srv.Config.Auth.ClientSecret, "auth.client-secret", srv.Config.Auth.ClientSecret, "Identity Provider's Client Secret.")
-	flags.StringVar(&srv.Config.Auth.AuthorizeURL, "auth.authorize-url", srv.Config.Auth.AuthorizeURL, "Identity Provider's Authorize URL.")
-	flags.StringVar(&srv.Config.Auth.RedirectBaseURL, "auth.redirect-base-url", srv.Config.Auth.RedirectBaseURL, "Base URL of the featurebase instance used to redirect IDP.")
-	flags.StringVar(&srv.Config.Auth.TokenURL, "auth.token-url", srv.Config.Auth.TokenURL, "Identity Provider's Token URL.")
-	flags.StringVar(&srv.Config.Auth.GroupEndpointURL, "auth.group-endpoint-url", srv.Config.Auth.GroupEndpointURL, "Identity Provider's Group endpoint URL.")
-	flags.StringVar(&srv.Config.Auth.LogoutURL, "auth.logout-url", srv.Config.Auth.LogoutURL, "Identity Provider's Logout URL.")
-	flags.StringSliceVar(&srv.Config.Auth.Scopes, "auth.scopes", srv.Config.Auth.Scopes, "Comma separated list of scopes obtained from IdP")
-	flags.StringVar(&srv.Config.Auth.SecretKey, "auth.secret-key", srv.Config.Auth.SecretKey, "Secret key used for auth.")
-	flags.StringVar(&srv.Config.Auth.PermissionsFile, "auth.permissions", srv.Config.Auth.PermissionsFile, "Permissions' file with group authorization.")
-	flags.StringVar(&srv.Config.Auth.QueryLogPath, "auth.query-log-path", srv.Config.Auth.QueryLogPath, "Path to log user queries")
-	flags.StringSliceVar(&srv.Config.Auth.ConfiguredIPs, "auth.configured-ips", srv.Config.Auth.ConfiguredIPs, "List of configured IPs allowed for ingest")
-
-	flags.BoolVar(&srv.Config.DataDog.Enable, "datadog.enable", false, "enable continuous profiling with DataDog cloud service, Note you must have DataDog agent installed")
-	flags.StringVar(&srv.Config.DataDog.Service, "datadog.service", "default-service", "The Datadog service name, for example my-web-app")
-	flags.StringVar(&srv.Config.DataDog.Env, "datadog.env", "default-env", "The Datadog environment name, for example, production")
-	flags.StringVar(&srv.Config.DataDog.Version, "datadog.version", "default-version", "The version of your application")
-	flags.StringVar(&srv.Config.DataDog.Tags, "datadog.tags", "molecula", "The tags to apply to an uploaded profile. Must be a list of in the format <KEY1>:<VALUE1>,<KEY2>:<VALUE2>")
-	flags.BoolVar(&srv.Config.DataDog.CPUProfile, "datadog.cpu-profile", true, "golang pprof cpu profile ")
-	flags.BoolVar(&srv.Config.DataDog.HeapProfile, "datadog.heap-profile", true, "golang pprof heap profile")
-	flags.BoolVar(&srv.Config.DataDog.MutexProfile, "datadog.mutex-profile", false, "golang pprof mutex profile")
-	flags.BoolVar(&srv.Config.DataDog.GoroutineProfile, "datadog.goroutine-profile", false, "golang pprof goroutine profile")
-	flags.BoolVar(&srv.Config.DataDog.BlockProfile, "datadog.block-profile", false, "golang pprof goroutine ")
 }

dax/Makefile

@@ -0,0 +1,89 @@
+.PHONY: test testv test-integration testv-integration
+
+MCLOUD_ENV ?= sandbox
+MCLOUD_ENV_FILE=.env.$(MCLOUD_ENV)
+-include $(MCLOUD_ENV_FILE)
+
+
+GO=go
+
+test:
+	$(GO) test ./... -short
+
+testv:
+	$(GO) test -v ./... -short
+
+test-integration:
+	mkdir -p ../coverage-from-docker
+	$(GO) test ./test/dax -count 1 -run Integration
+
+testv-integration:
+	$(GO) test -v ./test/dax -count 1 -run Integration
+
+
+
+############################### AWS STUFF ###############################
+
+AWS_REGION  ?=
+AWS_PROFILE ?=
+
+AWS = aws --profile=$(AWS_PROFILE) --region=$(AWS_REGION)
+
+# After pushing new images, use "make redeploy-ecs" to redeploy all DAX services.
+redeploy-ecs: redeploy-svc-mds redeploy-svc-computer redeploy-svc-queryer
+
+redeploy-svc-%:
+	$(AWS) ecs update-service --cluster DAX --service $*-$(MCLOUD_ENV)-ecs-service --force-new-deployment --no-cli-pager
+
+
+# Scale changes the desired count of the computer service. e.g. "make scale N=4"
+scale:
+	$(AWS) ecs update-service --cluster DAX --service computer-$(MCLOUD_ENV)-ecs-service --desired-count=$(N) --no-cli-pager
+
+
+I ?= 0
+# Get a shell on a running contianer. e.g. "make mds-shell", "make datagen-shell", etc.
+%-shell:
+	$(eval TASK_ARN := $(shell $(AWS) ecs list-tasks --cluster=DAX --family=$*-family | jq -r .taskArns[$(I)]))
+	$(AWS) ecs execute-command --cluster=DAX --task=$(TASK_ARN) --command=/bin/sh --interactive
+
+datagen: task-arn-datagen
+	$(eval TASK_ARN := $(shell $(AWS) ecs list-tasks --cluster=DAX --family=$*-family | jq -r .taskArns[$(I)]))
+	$(AWS) ecs run-task --cluster=DAX --task-definition=$(TASK_ARN) --cli-input-json=file://./datagen_task_input.json --no-cli-pager --enable-execute-command
+
+
+####################### docker-compose stuff ##############################3
+
+
+dc-reset: dc-prereqs dc-down
+	rm -rf ./dax-data/{snapshotter,writelogger}/*
+
+dc-build:
+	cd .. && $(MAKE) build-for-quick
+	docker-compose build
+
+dc-up:
+	docker-compose up -d
+
+dc-down:
+	docker-compose down
+
+dc-full-reup: dc-reset dc-build dc-up
+
+dc-logs:
+	docker-compose logs -f
+
+dc-logs-%:
+	docker-compose logs -f $*
+
+dc-prereqs:
+	mkdir -p ../.quick
+
+# This is just an example. For it to work, you'll first need to:
+#   featurebase cli --host localhost --port 8080 --org-id=testorg --db-id=testdb
+#   create table keysidstbl2 (_id string, slice idset);
+dc-datagen:
+	docker-compose run datagen --end-at=500  --pilosa.batch-size=500 --featurebase.table-name=keysidstbl2
+
+dc-exec-%:
+	docker-compose exec $* /bin/sh

dax/README.md

@@ -0,0 +1,64 @@
+# DAX
+
+DAX encapsulates anything which covers all of the services which make up the
+"disaggregation of storage and compute" project. Initially, this will include
+integration tests which pull in things like Metadata Services (MDS), including
+the Controller, as well as FeatureBase and IDK-based ingesters.
+
+## Setting up the tests
+
+The DAX test currently requires docker images for: `featurebase`  and `datagen`.
+
+If at any point you run into problems with go mod failing to reference a private
+repo, make sure that you have `gitlab.com/molecula` in your `GOPRIVATE`
+environment variable.
+
+Note that during the docker image build step, `go mod vendor` is run, which
+creates a `vendor` directory in the root directory, and copies that to docker
+during the build stage. Just be aware of this; you may want to remove that
+vendor directory after you're done building docker images.
+
+#### Possible Configuruation
+The following were relevent when the dax code was in a separate repository.
+These may no longer be relevant.
+
+I needed to but this in my `~/.profile` file:
+
+```export GOPRIVATE=github.com/molecula,gitlab.com/molecula```
+
+And this in my `~/.gitconfig`
+
+```
+[url "ssh://git@github.com/"]
+	insteadOf = https://github.com/
+[url "ssh://git@gitlab.com/"]
+	insteadOf = https://gitlab.com/
+```
+
+Then `make docker` ran successfully.
+
+
+### Build the FeatureBase docker image
+
+- Check out the
+  [dax](https://github.com/molecula/featurebase/tree/dax)
+  branch of the
+  [featurebase](https://github.com/molecula/featurebase) repository.
+- Run `make docker-image-featurebase` to build the docker image
+- You should now have an image in docker named `dax/featurebase` with the tag `latest`.
+
+### Build the Datagen docker image
+
+- `cd <featurebase_repo_root>/idk`
+- Run `make docker-image-datagen` to build the docker image
+- You should now have an image in docker named `dax/datagen` with the tag
+  `latest`.
+
+## Running the tests
+
+- Check out the
+  [dax](https://github.com/molecula/featurebase/tree/dax)
+  branch of the
+  [featurebase](https://github.com/molecula/featurebase) repository.
+- Change into the `dax` directory: `cd dax`
+- Run `make test-integration`.

dax/address.go

@@ -0,0 +1,138 @@
+package dax
+
+import (
+	"context"
+	"fmt"
+	"strconv"
+	"strings"
+)
+
+// Address is a string of the form [scheme]://[host]:[port]
+type Address string
+
+// String returns the Address as a string type.
+func (a Address) String() string {
+	return string(a)
+}
+
+// Scheme returns the [scheme] portion of the Address. This may be an empty
+// string if Address does not contain a scheme.
+func (a Address) Scheme() string {
+	return parse(a).scheme
+}
+
+// HostPort returns the [host]:[port] portion of the Address; in other words,
+// the Address stripped of any scheme.
+func (a Address) HostPort() string {
+	return parse(a).hostPort()
+}
+
+// Host returns the [host] portion of the Address.
+func (a Address) Host() string {
+	return parse(a).host
+}
+
+// Port returns the [port] portion of the Address. If the port values is invalid
+// or does not exist, the returned value will default to 0.
+func (a Address) Port() uint16 {
+	return parse(a).port
+}
+
+// OverrideScheme overrides Address's current scheme with the one provided. If
+// an empty scheme is provided, OverrideScheme will return just the host:port.
+func (a Address) OverrideScheme(scheme string) string {
+	addr := parse(a)
+	if scheme == "" {
+		return addr.hostPort()
+	}
+	return scheme + "://" + addr.hostPort()
+}
+
+// WithScheme ensures that the string returned contains the scheme portion of a
+// URL. Because an Address may not have a scheme (for example, it could be just
+// "host:80"), this method can be applied to an address when it needs to be used
+// as a URL. If the address's existing scheme is blank, the default scheme
+// provided will be used. If address is blank, the default scheme will not be
+// added; i.e. address will remain blank.
+func (a Address) WithScheme(dflt string) string {
+	// If address is empty, don't add a scheme to it.
+	if a == "" {
+		return ""
+	}
+
+	addr := parse(a)
+	if addr.scheme != "" {
+		return a.String()
+	}
+	return dflt + "://" + addr.hostPort()
+}
+
+type addr struct {
+	scheme string
+	host   string
+	port   uint16
+}
+
+// parse breaks the address up into scheme://host:port. It currently assumes
+// that very rigid structure; in other words, if an address does not follow that
+// format, return values may be unexpected.
+func parse(a Address) addr {
+	var scheme string
+	var host string
+	var port uint16
+
+	aStr := string(a)
+
+	var hostPort string
+	if parts := strings.Split(aStr, "://"); len(parts) > 1 {
+		scheme = parts[0]
+		hostPort = parts[1]
+	} else {
+		hostPort = aStr
+	}
+
+	if parts := strings.Split(hostPort, ":"); len(parts) == 2 {
+		host = parts[0]
+		portStr := parts[1]
+		port64, err := strconv.ParseInt(portStr, 10, 32)
+		if err == nil {
+			port = uint16(port64)
+		}
+	} else {
+		host = hostPort
+	}
+
+	return addr{
+		scheme: scheme,
+		host:   host,
+		port:   port,
+	}
+}
+
+func (a addr) hostPort() string {
+	if a.port == 0 {
+		return a.host
+	}
+	return fmt.Sprintf("%s:%d", a.host, a.port)
+}
+
+// AddressManager is an interface for any service which needs to maintain a list
+// of addresses, and receive add/remove address requests from other services.
+type AddressManager interface {
+	AddAddresses(context.Context, ...Address) error
+	RemoveAddresses(context.Context, ...Address) error
+}
+
+// Ensure type implements interface.
+var _ AddressManager = &NopAddressManager{}
+
+// NopAddressManager is a no-op implementation of the AddressManager interface.
+type NopAddressManager struct{}
+
+func NewNopAddressManager() *NopAddressManager {
+	return &NopAddressManager{}
+}
+
+func (a *NopAddressManager) AddAddresses(ctx context.Context, addrs ...Address) error { return nil }
+
+func (a *NopAddressManager) RemoveAddresses(ctx context.Context, addrs ...Address) error { return nil }

dax/address_test.go

@@ -0,0 +1,176 @@
+package dax_test
+
+import (
+	"fmt"
+	"testing"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/stretchr/testify/assert"
+)
+
+func TestAddress(t *testing.T) {
+	t.Run("Address", func(t *testing.T) {
+		tests := []struct {
+			addr        dax.Address
+			expScheme   string
+			expHostPort string
+			expHost     string
+			expPort     uint16
+		}{
+			{
+				// blank address
+				addr:        "",
+				expScheme:   "",
+				expHostPort: "",
+				expHost:     "",
+				expPort:     0,
+			},
+			{
+				// schema://
+				addr:        "http://",
+				expScheme:   "http",
+				expHostPort: "",
+				expHost:     "",
+				expPort:     0,
+			},
+			{
+				// host
+				addr:        "foo",
+				expScheme:   "",
+				expHostPort: "foo",
+				expHost:     "foo",
+				expPort:     0,
+			},
+			{
+				// :port
+				addr:        ":8080",
+				expScheme:   "",
+				expHostPort: ":8080",
+				expHost:     "",
+				expPort:     8080,
+			},
+			{
+				// host:port
+				addr:        "foo:8080",
+				expScheme:   "",
+				expHostPort: "foo:8080",
+				expHost:     "foo",
+				expPort:     8080,
+			},
+			{
+				// schema://host:port
+				addr:        "http://foo:8080",
+				expScheme:   "http",
+				expHostPort: "foo:8080",
+				expHost:     "foo",
+				expPort:     8080,
+			},
+			{
+				// schema://host
+				addr:        "http://foo",
+				expScheme:   "http",
+				expHostPort: "foo",
+				expHost:     "foo",
+				expPort:     0,
+			},
+			{
+				// schema://:port
+				addr:        "http://:8080",
+				expScheme:   "http",
+				expHostPort: ":8080",
+				expHost:     "",
+				expPort:     8080,
+			},
+			{
+				// invalid port
+				addr:        "http://foo:bar",
+				expScheme:   "http",
+				expHostPort: "foo",
+				expHost:     "foo",
+				expPort:     0,
+			},
+			{
+				// :port outside of int16 range
+				addr:        ":53308",
+				expScheme:   "",
+				expHostPort: ":53308",
+				expHost:     "",
+				expPort:     53308,
+			},
+		}
+
+		for i, test := range tests {
+			t.Run(fmt.Sprintf("test-%d", i), func(t *testing.T) {
+				assert.Equal(t, test.expScheme, test.addr.Scheme())
+				assert.Equal(t, test.expHostPort, test.addr.HostPort())
+				assert.Equal(t, test.expHost, test.addr.Host())
+				assert.Equal(t, test.expPort, test.addr.Port())
+			})
+		}
+	})
+
+	t.Run("OverrideScheme", func(t *testing.T) {
+		tests := []struct {
+			addr    dax.Address
+			scheme  string
+			expAddr string
+		}{
+			{
+				addr:    "foo",
+				scheme:  "http",
+				expAddr: "http://foo",
+			},
+			{
+				addr:    "http://foo",
+				scheme:  "grpc",
+				expAddr: "grpc://foo",
+			},
+			{
+				addr:    "http://foo:8080",
+				scheme:  "",
+				expAddr: "foo:8080",
+			},
+		}
+
+		for i, test := range tests {
+			t.Run(fmt.Sprintf("test-%d", i), func(t *testing.T) {
+				assert.Equal(t, test.expAddr, test.addr.OverrideScheme(test.scheme))
+			})
+		}
+	})
+
+	t.Run("WithScheme", func(t *testing.T) {
+		tests := []struct {
+			addr    dax.Address
+			scheme  string
+			expAddr string
+		}{
+			{
+				addr:    "foo",
+				scheme:  "",
+				expAddr: "://foo",
+			},
+			{
+				addr:    "http://foo",
+				scheme:  "grpc",
+				expAddr: "http://foo",
+			},
+			{
+				addr:    "http://foo:8080",
+				scheme:  "",
+				expAddr: "http://foo:8080",
+			},
+			{
+				addr:    "foo:8080",
+				scheme:  "grpc",
+				expAddr: "grpc://foo:8080",
+			},
+		}
+
+		for i, test := range tests {
+			t.Run(fmt.Sprintf("test-%d", i), func(t *testing.T) {
+				assert.Equal(t, test.expAddr, test.addr.WithScheme(test.scheme))
+			})
+		}
+	})
+}

dax/boltdb/boltdb.go

@@ -0,0 +1,151 @@
+// Package boltdb contains the boltdb implementations of the DAX interfaces.
+package boltdb
+
+import (
+	"context"
+	"os"
+	"path/filepath"
+	"strings"
+	"time"
+
+	"github.com/molecula/featurebase/v3/errors"
+	bolt "go.etcd.io/bbolt"
+)
+
+const (
+	ErrFmtBucketNotFound = "boltdb: bucket '%s' not found"
+)
+
+type Bucket []byte
+
+// DB represents the database connection.
+type DB struct {
+	db     *bolt.DB
+	ctx    context.Context // background context
+	cancel func()          // cancel background context
+
+	// Datasource name.
+	DSN string
+
+	// Destination for events to be published.
+	// EventService wtf.EventService
+
+	// Returns the current time. Defaults to time.Now().
+	// Can be mocked for tests.
+	Now func() time.Time
+
+	filePath string
+
+	// bucketQueue contains a list of buckets to create upon Open.
+	bucketQueue []Bucket
+}
+
+// NewDB returns a new instance of DB associated with the given datasource name.
+func NewDB(dsn string) *DB {
+	db := &DB{
+		DSN: dsn,
+		Now: time.Now,
+
+		//EventService: wtf.NopEventService(),
+	}
+	db.ctx, db.cancel = context.WithCancel(context.Background())
+	return db
+}
+
+// NewSvcBolt gets, opens, and creates buckets for a boltDB for a
+// particular named service (the data file will be named after the
+// service).
+func NewSvcBolt(dir, svc string, buckets ...Bucket) (*DB, error) {
+	dir = strings.TrimPrefix(dir, "file:")
+	filename := filepath.Join(dir, svc+".boltdb")
+	db := NewDB("file:" + filename)
+	db.RegisterBuckets(buckets...)
+	err := db.Open()
+	return db, errors.Wrap(err, "opening")
+}
+
+// path returns the file path to the boltdb database file.
+func (db *DB) path() (string, error) {
+	if !strings.HasPrefix(db.DSN, "file:") {
+		return "", errors.New(errors.ErrUncoded, "boltdb package only supports a DSN beginning with `file:`")
+	}
+
+	return db.DSN[5:], nil
+}
+
+// RegisterBuckets queues up the buckets to be created when the database is
+// first opened.
+func (db *DB) RegisterBuckets(buckets ...Bucket) {
+	db.bucketQueue = append(db.bucketQueue, buckets...)
+}
+
+// InitializeBuckets creates the given buckets if they do not already exist.
+func (db *DB) InitializeBuckets(buckets ...Bucket) (err error) {
+	return db.db.Update(func(tx *bolt.Tx) error {
+		for _, bucket := range buckets {
+			if _, err := tx.CreateBucketIfNotExists(bucket); err != nil {
+				return errors.Wrapf(err, "creating bucket: %s", bucket)
+			}
+		}
+		return nil
+	})
+}
+
+// Open opens the database connection.
+func (db *DB) Open() (err error) {
+	path, err := db.path()
+	if err != nil {
+		return errors.Wrap(err, "getting path from DSN")
+	}
+
+	if err := os.MkdirAll(filepath.Dir(path), 0777); err != nil {
+		return errors.Wrapf(err, "mkdir %s", filepath.Dir(path))
+	} else if db.db, err = bolt.Open(path, 0666, &bolt.Options{Timeout: 1 * time.Second}); err != nil {
+		return errors.Wrapf(err, "open file: %s", err)
+	}
+
+	// cache the path in db.filePath.
+	db.filePath = path
+
+	if err := db.InitializeBuckets(db.bucketQueue...); err != nil {
+		return errors.Wrap(err, "initializing buckets")
+	}
+
+	// Reset the bucketQueue.
+	db.bucketQueue = make([]Bucket, 0)
+
+	return nil
+}
+
+// Close closes the database connection.
+func (db *DB) Close() (err error) {
+	return db.db.Close()
+}
+
+// BeginTx starts a transaction and returns a wrapper Tx type. This type
+// provides a reference to the database and a fixed timestamp at the start of
+// the transaction. The timestamp allows us to mock time during tests as well.
+func (db *DB) BeginTx(ctx context.Context, writable bool) (*Tx, error) {
+	tx, err := db.db.Begin(writable)
+	if err != nil {
+		return nil, err
+	}
+
+	// Return wrapper Tx that includes the transaction start time.
+	return &Tx{
+		Tx:  tx,
+		db:  db,
+		now: db.Now().UTC().Truncate(time.Second),
+	}, nil
+}
+
+// Tx wraps the SQL Tx object to provide a timestamp at the start of the transaction.
+type Tx struct {
+	*bolt.Tx
+	db  *DB
+	now time.Time
+}
+
+func (db *DB) Path() string {
+	return db.filePath
+}

dax/boltdb/boltdb_test.go

@@ -0,0 +1,17 @@
+package boltdb_test
+
+import (
+	"testing"
+
+	"github.com/molecula/featurebase/v3/dax/test/boltdb"
+)
+
+// Ensure the test database can open & close.
+func TestDB(t *testing.T) {
+	db := boltdb.MustOpenDB(t)
+	defer boltdb.MustCloseDB(t, db)
+
+	t.Cleanup(func() {
+		boltdb.CleanupDB(t, db.Path())
+	})
+}

dax/boltdb/directiveversion.go

@@ -0,0 +1,66 @@
+package boltdb
+
+import (
+	"context"
+	"encoding/binary"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/errors"
+)
+
+var (
+	bucketDirective     = Bucket("nodeDirective")
+	keyDirectiveVersion = []byte("directiveVersion")
+)
+
+// DirectiveBuckets defines the buckets used by this package. It can be called
+// during setup to create the buckets ahead of time.
+var DirectiveBuckets []Bucket = []Bucket{
+	bucketDirective,
+}
+
+// Ensure type implements interface.
+var _ dax.DirectiveVersion = (*DirectiveVersion)(nil)
+
+type DirectiveVersion struct {
+	db *DB
+}
+
+func NewDirectiveVersion(db *DB) *DirectiveVersion {
+	return &DirectiveVersion{
+		db: db,
+	}
+}
+
+func (d *DirectiveVersion) Increment(ctx context.Context, delta uint64) (uint64, error) {
+	tx, err := d.db.BeginTx(ctx, true)
+	if err != nil {
+		return 0, errors.Wrap(err, "getting transaction")
+	}
+	defer tx.Rollback()
+
+	bkt := tx.Bucket(bucketDirective)
+	if bkt == nil {
+		return 0, errors.Errorf(ErrFmtBucketNotFound, bucketDirective)
+	}
+
+	var nextVersion uint64 = 1 // Start at 1; 0 is an invalid version.
+
+	b := bkt.Get(keyDirectiveVersion)
+	if b != nil {
+		nextVersion = binary.LittleEndian.Uint64(b) + delta
+	}
+
+	vsn := make([]byte, 8)
+	binary.LittleEndian.PutUint64(vsn, nextVersion)
+
+	if err := bkt.Put(keyDirectiveVersion, vsn); err != nil {
+		return 0, errors.Wrap(err, "putting next directive version")
+	}
+
+	if err := tx.Commit(); err != nil {
+		return 0, err
+	}
+
+	return nextVersion, nil
+}

dax/boltdb/node.go

@@ -0,0 +1,157 @@
+package boltdb
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/errors"
+	"github.com/molecula/featurebase/v3/logger"
+)
+
+var (
+	bucketNodes = Bucket("nodeServiceNodes")
+)
+
+// NodeServiceBuckets defines the buckets used by this package. It can be called
+// during setup to create the buckets ahead of time.
+var NodeServiceBuckets []Bucket = []Bucket{
+	bucketNodes,
+}
+
+// Ensure type implements interface.
+var _ dax.NodeService = (*NodeService)(nil)
+
+// NodeService represents a service for managing nodes.
+type NodeService struct {
+	db *DB
+
+	logger logger.Logger
+}
+
+// NewNodeService returns a new instance of NodeService with default values.
+func NewNodeService(db *DB, logger logger.Logger) *NodeService {
+	return &NodeService{
+		db:     db,
+		logger: logger,
+	}
+}
+
+func (s *NodeService) CreateNode(ctx context.Context, addr dax.Address, node *dax.Node) error {
+	tx, err := s.db.BeginTx(ctx, true)
+	if err != nil {
+		return errors.Wrap(err, "getting transaction")
+	}
+	defer tx.Rollback()
+
+	bkt := tx.Bucket(bucketNodes)
+	if bkt == nil {
+		return errors.Errorf(ErrFmtBucketNotFound, bucketNodes)
+	}
+
+	val, err := json.Marshal(node)
+	if err != nil {
+		return errors.Wrap(err, "marshalling node to json")
+	}
+
+	if err := bkt.Put(addressKey(addr), val); err != nil {
+		return errors.Wrap(err, "putting node")
+	}
+
+	return tx.Commit()
+}
+
+func (s *NodeService) ReadNode(ctx context.Context, addr dax.Address) (*dax.Node, error) {
+	tx, err := s.db.BeginTx(ctx, false)
+	if err != nil {
+		return nil, errors.Wrap(err, "beginning tx")
+	}
+	defer tx.Rollback()
+
+	bkt := tx.Bucket(bucketNodes)
+	if bkt == nil {
+		return nil, errors.Errorf(ErrFmtBucketNotFound, bucketNodes)
+	}
+
+	b := bkt.Get(addressKey(addr))
+	if b == nil {
+		return nil, dax.NewErrNodeDoesNotExist(addr)
+	}
+
+	node := &dax.Node{}
+	if err := json.Unmarshal(b, node); err != nil {
+		return nil, errors.Wrap(err, "unmarshalling node json")
+	}
+
+	return node, nil
+}
+
+func (s *NodeService) DeleteNode(ctx context.Context, addr dax.Address) error {
+	tx, err := s.db.BeginTx(ctx, true)
+	if err != nil {
+		return errors.Wrap(err, "beginning tx")
+	}
+	defer tx.Rollback()
+
+	bkt := tx.Bucket(bucketNodes)
+	if bkt == nil {
+		return errors.Errorf(ErrFmtBucketNotFound, bucketNodes)
+	}
+
+	if err := bkt.Delete(addressKey(addr)); err != nil {
+		return errors.Wrapf(err, "deleting node key: %s", addressKey(addr))
+	}
+
+	return tx.Commit()
+}
+
+func (s *NodeService) Nodes(ctx context.Context) ([]*dax.Node, error) {
+	tx, err := s.db.BeginTx(ctx, false)
+	if err != nil {
+		return nil, errors.Wrap(err, "getting tx")
+	}
+	defer tx.Rollback()
+
+	nodes, err := s.getNodes(ctx, tx)
+	if err != nil {
+		return nil, errors.Wrap(err, "getting nodes")
+	}
+
+	return nodes, nil
+}
+
+func (s *NodeService) getNodes(ctx context.Context, tx *Tx) ([]*dax.Node, error) {
+	c := tx.Bucket(bucketNodes).Cursor()
+
+	// Deserialize rows into Node objects.
+	nodes := make([]*dax.Node, 0)
+
+	prefix := []byte(prefixFmtNodes)
+	for k, v := c.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, v = c.Next() {
+		if v == nil {
+			s.logger.Printf("nil value for key: %s", k)
+			continue
+		}
+
+		node := &dax.Node{}
+		if err := json.Unmarshal(v, node); err != nil {
+			return nil, errors.Wrap(err, "unmarshalling node json")
+		}
+
+		nodes = append(nodes, node)
+	}
+
+	return nodes, nil
+}
+
+const (
+	prefixFmtNodes = "nodes/"
+)
+
+// addressKey returns a key based on address.
+func addressKey(addr dax.Address) []byte {
+	key := fmt.Sprintf(prefixFmtNodes+"%s", addr)
+	return []byte(key)
+}

dax/boltdb/node_test.go

@@ -0,0 +1,55 @@
+package boltdb_test
+
+import (
+	"context"
+	"testing"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/boltdb"
+	testbolt "github.com/molecula/featurebase/v3/dax/test/boltdb"
+	"github.com/molecula/featurebase/v3/errors"
+	"github.com/molecula/featurebase/v3/logger"
+	"github.com/stretchr/testify/assert"
+)
+
+func TestNodeService(t *testing.T) {
+	db := testbolt.MustOpenDB(t)
+	defer testbolt.MustCloseDB(t, db)
+
+	t.Cleanup(func() {
+		testbolt.CleanupDB(t, db.Path())
+	})
+
+	ctx := context.Background()
+
+	// Initialize the buckets.
+	assert.NoError(t, db.InitializeBuckets(boltdb.NodeServiceBuckets...))
+
+	t.Run("Nodes", func(t *testing.T) {
+		ns := boltdb.NewNodeService(db, logger.NopLogger)
+
+		node1 := &dax.Node{
+			Address: "localhost:10101",
+			RoleTypes: []dax.RoleType{
+				"compute",
+			},
+		}
+
+		// Create node.
+		assert.NoError(t, ns.CreateNode(ctx, node1.Address, node1))
+
+		// Read node.
+		n, err := ns.ReadNode(ctx, node1.Address)
+		assert.NoError(t, err)
+		assert.Equal(t, node1, n)
+
+		// Delete node.
+		assert.NoError(t, ns.DeleteNode(ctx, node1.Address))
+
+		// Read node.
+		_, err = ns.ReadNode(ctx, node1.Address)
+		if assert.Error(t, err) {
+			assert.True(t, errors.Is(err, dax.ErrNodeDoesNotExist))
+		}
+	})
+}

dax/boltdb/versionstore.go

@@ -0,0 +1,768 @@
+package boltdb
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"fmt"
+	"strconv"
+	"strings"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/inmem"
+	"github.com/molecula/featurebase/v3/errors"
+	"github.com/molecula/featurebase/v3/logger"
+)
+
+var (
+	bucketTables    = Bucket("versionStoreTables")
+	bucketShards    = Bucket("versionStoreShards")
+	bucketTableKeys = Bucket("versionStoreTableKeys")
+	bucketFieldKeys = Bucket("versionStoreFieldKeys")
+)
+
+// VersionStoreBuckets defines the buckets used by this package. It can be
+// called during setup to create the buckets ahead of time.
+var VersionStoreBuckets []Bucket = []Bucket{
+	bucketTables,
+	bucketShards,
+	bucketTableKeys,
+	bucketFieldKeys,
+}
+
+// Ensure type implements interface.
+var _ dax.VersionStore = (*VersionStore)(nil)
+
+// VersionStore manages all version info for shard, table keys, and field keys.
+type VersionStore struct {
+	db *DB
+
+	logger logger.Logger
+}
+
+// NewVersionStore returns a new instance of VersionStore with default values.
+func NewVersionStore(db *DB, logger logger.Logger) *VersionStore {
+	return &VersionStore{
+		db:     db,
+		logger: logger,
+	}
+}
+
+func (s *VersionStore) AddTable(ctx context.Context, qtid dax.QualifiedTableID) error {
+	tx, err := s.db.BeginTx(ctx, true)
+	if err != nil {
+		return errors.Wrap(err, "getting transaction")
+	}
+	defer tx.Rollback()
+
+	bkt := tx.Bucket(bucketTables)
+	if bkt == nil {
+		return errors.Errorf(ErrFmtBucketNotFound, bucketTables)
+	}
+
+	if val := bkt.Get(tableKey(qtid)); val != nil {
+		return dax.NewErrTableIDExists(qtid)
+	}
+
+	// The assumption is that we may store information about the table (other
+	// than just the fact that it exists). So for now, the value is an empty
+	// JSON object.
+	val := []byte("{}")
+
+	if err := bkt.Put(tableKey(qtid), val); err != nil {
+		return errors.Wrap(err, "putting table")
+	}
+
+	// Add the table to the "table index" of the other buckets.
+	//
+	// Shards
+	if bkt := tx.Bucket(bucketShards); bkt == nil {
+		return errors.Errorf(ErrFmtBucketNotFound, bucketShards)
+	} else if err := bkt.Put(tableKey(qtid), val); err != nil {
+		return errors.Wrap(err, "putting table into shards")
+	}
+
+	// TableKeys.
+	if bkt := tx.Bucket(bucketTableKeys); bkt == nil {
+		return errors.Errorf(ErrFmtBucketNotFound, bucketTableKeys)
+	} else if err := bkt.Put(tableKey(qtid), val); err != nil {
+		return errors.Wrap(err, "putting table into table keys")
+	}
+
+	// FieldKeys.
+	if bkt := tx.Bucket(bucketFieldKeys); bkt == nil {
+		return errors.Errorf(ErrFmtBucketNotFound, bucketFieldKeys)
+	} else if err := bkt.Put(tableKey(qtid), val); err != nil {
+		return errors.Wrap(err, "putting table into field keys")
+	}
+
+	return tx.Commit()
+}
+
+func (s *VersionStore) RemoveTable(ctx context.Context, qtid dax.QualifiedTableID) (dax.Shards, dax.Partitions, error) {
+	tx, err := s.db.BeginTx(ctx, true)
+	if err != nil {
+		return nil, nil, err
+	}
+	defer tx.Rollback()
+
+	// Get the shards and partitions before deleting by table.
+	shards, err := s.getShards(ctx, tx, qtid)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	partitions, err := s.getPartitions(ctx, tx, qtid)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	if err := removeTable(ctx, tx, qtid); err != nil {
+		return nil, nil, err
+	}
+
+	if err := tx.Commit(); err != nil {
+		return nil, nil, err
+	}
+
+	return shards, partitions, nil
+}
+
+func removeTable(ctx context.Context, tx *Tx, qtid dax.QualifiedTableID) error {
+	// Tables.
+	if bkt := tx.Bucket(bucketTables); bkt == nil {
+		return errors.Errorf(ErrFmtBucketNotFound, bucketTables)
+	} else if err := bkt.Delete(tableKey(qtid)); err != nil {
+		return errors.Wrap(err, "deleting table")
+	}
+
+	// Shards.
+	if bkt := tx.Bucket(bucketShards); bkt == nil {
+		return errors.Errorf(ErrFmtBucketNotFound, bucketShards)
+	} else if err := bkt.Delete(tableKey(qtid)); err != nil {
+		return errors.Wrap(err, "deleting table in shards")
+	} else if err := deleteByPrefix(tx, bucketShards, []byte(fmt.Sprintf(prefixFmtShards, qtid.OrganizationID, qtid.DatabaseID, qtid.ID))); err != nil {
+		return errors.Wrap(err, "deleting shards for table")
+	}
+
+	// TableKeys.
+	if bkt := tx.Bucket(bucketTableKeys); bkt == nil {
+		return errors.Errorf(ErrFmtBucketNotFound, bucketTableKeys)
+	} else if err := bkt.Delete(tableKey(qtid)); err != nil {
+		return errors.Wrap(err, "deleting table in table keys")
+	} else if err := deleteByPrefix(tx, bucketTableKeys, []byte(fmt.Sprintf(prefixFmtTableKeys, qtid.OrganizationID, qtid.DatabaseID, qtid.ID))); err != nil {
+		return errors.Wrap(err, "deleting table keys for table")
+	}
+
+	// FieldKeys.
+	if bkt := tx.Bucket(bucketFieldKeys); bkt == nil {
+		return errors.Errorf(ErrFmtBucketNotFound, bucketFieldKeys)
+	} else if err := bkt.Delete(tableKey(qtid)); err != nil {
+		return errors.Wrap(err, "deleting table in field keys")
+	} else if err := deleteByPrefix(tx, bucketFieldKeys, []byte(fmt.Sprintf(prefixFmtFieldKeys, qtid.OrganizationID, qtid.DatabaseID, qtid.ID))); err != nil {
+		return errors.Wrap(err, "deleting field keys for table")
+	}
+
+	return nil
+}
+
+func deleteByPrefix(tx *Tx, bucket Bucket, prefix []byte) error {
+	bkt := tx.Bucket(bucket)
+	cursor := bkt.Cursor()
+
+	// Deleting keys within the for loop seems to cause Next() to skip the next
+	// matching key because the Delete() call pops the item and effectively
+	// moves the cursor forward. Then calling Next() skips the item that was
+	// being pointed to after the delete. So, we're going to make a list of keys
+	// to delete, and then delete them outside of the cursor logic.
+	var keysToDelete [][]byte
+	for k, _ := cursor.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, _ = cursor.Next() {
+		keysToDelete = append(keysToDelete, k)
+	}
+
+	for _, k := range keysToDelete {
+		if err := bkt.Delete(k); err != nil {
+			return errors.Wrapf(err, "deleting key: %s", k)
+		}
+	}
+
+	return nil
+}
+
+func (s *VersionStore) AddShards(ctx context.Context, qtid dax.QualifiedTableID, shards ...dax.Shard) error {
+	tx, err := s.db.BeginTx(ctx, true)
+	if err != nil {
+		return errors.Wrap(err, "getting transaction")
+	}
+	defer tx.Rollback()
+
+	for _, shard := range shards {
+		if err := createShard(ctx, tx, qtid, shard); err != nil {
+			return errors.Wrap(err, "creating shard")
+		}
+	}
+
+	return tx.Commit()
+}
+
+func createShard(ctx context.Context, tx *Tx, qtid dax.QualifiedTableID, shard dax.Shard) error {
+	// TODO: validate data more formally
+	if shard.Version < 0 {
+		return errors.New(errors.ErrUncoded, fmt.Sprintf("invalid shard version: %d", shard.Version))
+	}
+
+	bkt := tx.Bucket(bucketShards)
+	if bkt == nil {
+		return errors.Errorf(ErrFmtBucketNotFound, bucketShards)
+	}
+
+	// Ensure the table exists.
+	if val := bkt.Get(tableKey(qtid)); val == nil {
+		return dax.NewErrTableIDDoesNotExist(qtid)
+	}
+
+	vsn := make([]byte, 8)
+	binary.LittleEndian.PutUint64(vsn, uint64(shard.Version))
+
+	return bkt.Put(shardKey(qtid, shard.Num), vsn)
+}
+
+func (s *VersionStore) Shards(ctx context.Context, qtid dax.QualifiedTableID) (dax.Shards, bool, error) {
+	tx, err := s.db.BeginTx(ctx, false)
+	if err != nil {
+		return nil, false, errors.Wrap(err, "getting tx")
+	}
+	defer tx.Rollback()
+
+	shards, err := s.getShards(ctx, tx, qtid)
+	if err != nil {
+		return nil, false, errors.Wrap(err, "getting shards")
+	}
+
+	return shards, true, nil
+}
+
+func (s *VersionStore) getShards(ctx context.Context, tx *Tx, qtid dax.QualifiedTableID) (dax.Shards, error) {
+	c := tx.Bucket(bucketShards).Cursor()
+
+	// Deserialize rows into Shard objects.
+	shards := make(dax.Shards, 0)
+
+	prefix := []byte(fmt.Sprintf(prefixFmtShards, qtid.OrganizationID, qtid.DatabaseID, qtid.ID))
+	for k, v := c.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, v = c.Next() {
+		if v == nil {
+			s.logger.Printf("nil value for key: %s", k)
+			continue
+		}
+
+		var shard dax.Shard
+
+		shardNum, err := keyShardNum(k)
+		if err != nil {
+			return nil, errors.Wrapf(err, "getting shardNum from key: %v", k)
+		}
+
+		shard.Num = shardNum
+		shard.Version = int(binary.LittleEndian.Uint64(v))
+
+		shards = append(shards, shard)
+	}
+
+	return shards, nil
+}
+
+// ShardVersion return the current version for the given table/shardNum.
+// If a version is not being tracked, it returns a bool value of false.
+func (s *VersionStore) ShardVersion(ctx context.Context, qtid dax.QualifiedTableID, shardNum dax.ShardNum) (int, bool, error) {
+	tx, err := s.db.BeginTx(ctx, false)
+	if err != nil {
+		return -1, false, err
+	}
+	defer tx.Rollback()
+
+	return getShardVersion(ctx, tx, qtid, shardNum)
+}
+
+func getShardVersion(ctx context.Context, tx *Tx, qtid dax.QualifiedTableID, shardNum dax.ShardNum) (int, bool, error) {
+	version := -1
+
+	bkt := tx.Bucket(bucketShards)
+	if bkt == nil {
+		return version, false, errors.Errorf(ErrFmtBucketNotFound, bucketShards)
+	}
+
+	b := bkt.Get(shardKey(qtid, shardNum))
+	if b == nil {
+		return version, false, nil
+	}
+	version = int(binary.LittleEndian.Uint64(b))
+
+	return version, true, nil
+}
+
+func (s *VersionStore) ShardTables(ctx context.Context, qual dax.TableQualifier) (dax.TableIDs, error) {
+	tx, err := s.db.BeginTx(ctx, false)
+	if err != nil {
+		return nil, errors.Wrap(err, "beginning tx")
+	}
+	defer tx.Rollback()
+
+	return s.getTableIDs(ctx, tx, qual, bucketShards)
+}
+
+func (s *VersionStore) getTableIDs(ctx context.Context, tx *Tx, qual dax.TableQualifier, bucket Bucket) (dax.TableIDs, error) {
+	c := tx.Bucket(bucket).Cursor()
+
+	// Deserialize rows into Tables objects.
+	tableIDs := make(dax.TableIDs, 0)
+
+	prefix := []byte(fmt.Sprintf(prefixFmtTables, qual.OrganizationID, qual.DatabaseID))
+	for k, v := c.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, v = c.Next() {
+		if v == nil {
+			s.logger.Printf("nil value for key: %s", k)
+			continue
+		}
+
+		var tableID dax.TableID
+
+		tableID, err := keyTableID(k)
+		if err != nil {
+			return nil, errors.Wrapf(err, "getting table name from key: %v", k)
+		}
+
+		tableIDs = append(tableIDs, tableID)
+	}
+
+	return tableIDs, nil
+}
+
+func (s *VersionStore) bucketTables(ctx context.Context, bucket Bucket) ([]dax.QualifiedTableID, error) {
+	tx, err := s.db.BeginTx(ctx, false)
+	if err != nil {
+		return nil, errors.Wrap(err, "beginning tx")
+	}
+	defer tx.Rollback()
+
+	c := tx.Bucket(bucket).Cursor()
+
+	// Deserialize rows into Tables objects.
+	qtids := make([]dax.QualifiedTableID, 0)
+
+	prefix := []byte(prefixTables)
+	for k, v := c.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, v = c.Next() {
+		if v == nil {
+			s.logger.Printf("nil value for key: %s", k)
+			continue
+		}
+
+		qtid, err := keyQualifiedTableID(k)
+		if err != nil {
+			return nil, errors.Wrapf(err, "getting qualified table id from key: %v", k)
+		}
+
+		qtids = append(qtids, qtid)
+	}
+
+	return qtids, nil
+}
+
+// AddPartitions adds new partitions to be managed by VersionStore. It returns
+// the number of partitions added or an error.
+func (s *VersionStore) AddPartitions(ctx context.Context, qtid dax.QualifiedTableID, partitions ...dax.Partition) error {
+	tx, err := s.db.BeginTx(ctx, true)
+	if err != nil {
+		return errors.Wrap(err, "getting transaction")
+	}
+	defer tx.Rollback()
+
+	for _, partition := range partitions {
+		if err := createPartition(ctx, tx, qtid, partition); err != nil {
+			return errors.Wrap(err, "creating partition")
+		}
+	}
+
+	return tx.Commit()
+}
+
+func createPartition(ctx context.Context, tx *Tx, qtid dax.QualifiedTableID, partition dax.Partition) error {
+	// TODO: validate data more formally
+	if partition.Version < 0 {
+		return errors.New(errors.ErrUncoded, fmt.Sprintf("invalid partition version: %d", partition.Version))
+	}
+
+	bkt := tx.Bucket(bucketTableKeys)
+	if bkt == nil {
+		return errors.Errorf(ErrFmtBucketNotFound, bucketTableKeys)
+	}
+
+	// Ensure the table exists.
+	if val := bkt.Get(tableKey(qtid)); val == nil {
+		return dax.NewErrTableIDDoesNotExist(qtid)
+	}
+
+	vsn := make([]byte, 8)
+	binary.LittleEndian.PutUint64(vsn, uint64(partition.Version))
+
+	return bkt.Put(partitionKey(qtid, partition.Num), vsn)
+}
+
+func (s *VersionStore) Partitions(ctx context.Context, qtid dax.QualifiedTableID) (dax.Partitions, bool, error) {
+	tx, err := s.db.BeginTx(ctx, false)
+	if err != nil {
+		return nil, false, errors.Wrap(err, "getting tx")
+	}
+	defer tx.Rollback()
+
+	partitions, err := s.getPartitions(ctx, tx, qtid)
+	if err != nil {
+		return nil, false, errors.Wrap(err, "getting partitions")
+	}
+
+	return partitions, true, nil
+}
+
+func (s *VersionStore) getPartitions(ctx context.Context, tx *Tx, qtid dax.QualifiedTableID) (dax.Partitions, error) {
+	c := tx.Bucket(bucketTableKeys).Cursor()
+
+	// Deserialize rows into Partition objects.
+	partitions := make(dax.Partitions, 0)
+
+	prefix := []byte(fmt.Sprintf(prefixFmtTableKeys, qtid.OrganizationID, qtid.DatabaseID, qtid.ID))
+	for k, v := c.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, v = c.Next() {
+		if v == nil {
+			s.logger.Printf("nil value for key: %s", k)
+			continue
+		}
+
+		var partition dax.Partition
+
+		partitionNum, err := keyPartitionNum(k)
+		if err != nil {
+			return nil, errors.Wrapf(err, "getting partitionNum from key: %v", k)
+		}
+
+		partition.Num = partitionNum
+		partition.Version = int(binary.LittleEndian.Uint64(v))
+
+		partitions = append(partitions, partition)
+	}
+
+	return partitions, nil
+}
+
+func (s *VersionStore) PartitionVersion(ctx context.Context, qtid dax.QualifiedTableID, partitionNum dax.PartitionNum) (int, bool, error) {
+	tx, err := s.db.BeginTx(ctx, false)
+	if err != nil {
+		return -1, false, err
+	}
+	defer tx.Rollback()
+
+	return getPartitionVersion(ctx, tx, qtid, partitionNum)
+}
+
+func getPartitionVersion(ctx context.Context, tx *Tx, qtid dax.QualifiedTableID, partitionNum dax.PartitionNum) (int, bool, error) {
+	version := -1
+
+	bkt := tx.Bucket(bucketTableKeys)
+	if bkt == nil {
+		return version, false, errors.Errorf(ErrFmtBucketNotFound, bucketTableKeys)
+	}
+
+	b := bkt.Get(partitionKey(qtid, partitionNum))
+	if b == nil {
+		return version, false, nil
+	}
+	version = int(binary.LittleEndian.Uint64(b))
+
+	return version, true, nil
+}
+
+func (s *VersionStore) PartitionTables(ctx context.Context, qual dax.TableQualifier) (dax.TableIDs, error) {
+	tx, err := s.db.BeginTx(ctx, false)
+	if err != nil {
+		return nil, errors.Wrap(err, "beginning tx")
+	}
+	defer tx.Rollback()
+
+	return s.getTableIDs(ctx, tx, qual, bucketTableKeys)
+}
+
+// AddFields adds new fields to be managed by VersionStore. It returns the
+// number of fields added or an error.
+func (s *VersionStore) AddFields(ctx context.Context, qtid dax.QualifiedTableID, fields ...dax.FieldVersion) error {
+	tx, err := s.db.BeginTx(ctx, true)
+	if err != nil {
+		return err
+	}
+	defer tx.Rollback()
+
+	for _, field := range fields {
+		if err := createFieldVersion(ctx, tx, qtid, field); err != nil {
+			return errors.Wrap(err, "creating field version")
+		}
+	}
+
+	return tx.Commit()
+}
+
+func createFieldVersion(ctx context.Context, tx *Tx, qtid dax.QualifiedTableID, field dax.FieldVersion) error {
+	// TODO: validate data more formally
+	if field.Version < 0 {
+		return errors.New(errors.ErrUncoded, fmt.Sprintf("invalid field version: %d", field.Version))
+	}
+
+	bkt := tx.Bucket(bucketFieldKeys)
+	if bkt == nil {
+		return errors.Errorf(ErrFmtBucketNotFound, bucketFieldKeys)
+	}
+
+	// Ensure the table exists.
+	if val := bkt.Get(tableKey(qtid)); val == nil {
+		return dax.NewErrTableIDDoesNotExist(qtid)
+	}
+
+	vsn := make([]byte, 8)
+	binary.LittleEndian.PutUint64(vsn, uint64(field.Version))
+
+	return bkt.Put(fieldKey(qtid, field.Name), vsn)
+}
+
+func (s *VersionStore) Fields(ctx context.Context, qtid dax.QualifiedTableID) (dax.FieldVersions, bool, error) {
+	tx, err := s.db.BeginTx(ctx, false)
+	if err != nil {
+		return nil, false, errors.Wrap(err, "getting tx")
+	}
+	defer tx.Rollback()
+
+	fields, err := s.getFields(ctx, tx, qtid)
+	if err != nil {
+		return nil, false, errors.Wrap(err, "getting fields")
+	}
+
+	return fields, true, nil
+}
+
+func (s *VersionStore) getFields(ctx context.Context, tx *Tx, qtid dax.QualifiedTableID) (dax.FieldVersions, error) {
+	c := tx.Bucket(bucketFieldKeys).Cursor()
+
+	// Deserialize rows into FieldVersion objects.
+	fieldVersions := make(dax.FieldVersions, 0)
+
+	prefix := []byte(fmt.Sprintf(prefixFmtFieldKeys, qtid.OrganizationID, qtid.DatabaseID, qtid.ID))
+	for k, v := c.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, v = c.Next() {
+		if v == nil {
+			s.logger.Printf("nil value for key: %s", k)
+			continue
+		}
+
+		var fieldVersion dax.FieldVersion
+
+		fieldName, err := keyFieldName(k)
+		if err != nil {
+			return nil, errors.Wrapf(err, "getting partitionNum from key: %v", k)
+		}
+
+		fieldVersion.Name = fieldName
+		fieldVersion.Version = int(binary.LittleEndian.Uint64(v))
+
+		fieldVersions = append(fieldVersions, fieldVersion)
+	}
+
+	return fieldVersions, nil
+}
+
+func (s *VersionStore) FieldVersion(ctx context.Context, qtid dax.QualifiedTableID, field dax.FieldName) (int, bool, error) {
+	tx, err := s.db.BeginTx(ctx, false)
+	if err != nil {
+		return -1, false, err
+	}
+	defer tx.Rollback()
+
+	return getFieldVersion(ctx, tx, qtid, field)
+}
+
+func getFieldVersion(ctx context.Context, tx *Tx, qtid dax.QualifiedTableID, field dax.FieldName) (int, bool, error) {
+	version := -1
+
+	bkt := tx.Bucket(bucketFieldKeys)
+	if bkt == nil {
+		return version, false, errors.Errorf(ErrFmtBucketNotFound, bucketFieldKeys)
+	}
+
+	b := bkt.Get(fieldKey(qtid, field))
+	if b == nil {
+		return version, false, nil
+	}
+	version = int(binary.LittleEndian.Uint64(b))
+
+	return version, true, nil
+}
+
+func (s *VersionStore) FieldTables(ctx context.Context, qual dax.TableQualifier) (dax.TableIDs, error) {
+	tx, err := s.db.BeginTx(ctx, false)
+	if err != nil {
+		return nil, errors.Wrap(err, "beginning tx")
+	}
+	defer tx.Rollback()
+
+	return s.getTableIDs(ctx, tx, qual, bucketFieldKeys)
+}
+
+// Copy returns an in-memory copy of VersionStore.
+func (s *VersionStore) Copy(ctx context.Context) (dax.VersionStore, error) {
+	new := inmem.NewVersionStore()
+
+	// shards.
+	qtids, err := s.bucketTables(ctx, bucketShards)
+	if err != nil {
+		return nil, errors.Wrap(err, "getting shard tables")
+	}
+	for _, qtid := range qtids {
+		shards, found, err := s.Shards(ctx, qtid)
+		if err != nil {
+			return nil, errors.Wrap(err, "getting shards")
+		} else if !found {
+			continue
+		}
+		_ = new.AddTable(ctx, qtid)
+		new.AddShards(ctx, qtid, shards...)
+	}
+
+	// tableKeys.
+	qtids, err = s.bucketTables(ctx, bucketTableKeys)
+	if err != nil {
+		return nil, errors.Wrap(err, "getting table key tables")
+	}
+	for _, qtid := range qtids {
+		partitions, found, err := s.Partitions(ctx, qtid)
+		if err != nil {
+			return nil, errors.Wrap(err, "getting partitions")
+		} else if !found {
+			continue
+		}
+		_ = new.AddTable(ctx, qtid)
+		new.AddPartitions(ctx, qtid, partitions...)
+	}
+
+	// fieldKeys.
+	qtids, err = s.bucketTables(ctx, bucketFieldKeys)
+	if err != nil {
+		return nil, errors.Wrap(err, "getting field key tables")
+	}
+	for _, qtid := range qtids {
+		fields, found, err := s.Fields(ctx, qtid)
+		if err != nil {
+			return nil, errors.Wrap(err, "getting fields")
+		} else if !found {
+			continue
+		}
+		_ = new.AddTable(ctx, qtid)
+		new.AddFields(ctx, qtid, fields...)
+	}
+
+	return new, nil
+}
+
+/////////////////////////////////////////////////////////
+
+const (
+	prefixShards    = "shards/"
+	prefixFmtShards = prefixShards + "%s/%s/%s/"
+
+	prefixTableKeys    = "tablekeys/"
+	prefixFmtTableKeys = prefixTableKeys + "%s/%s/%s/"
+
+	prefixFieldKeys    = "fieldkeys/"
+	prefixFmtFieldKeys = prefixFieldKeys + "%s/%s/%s/"
+
+	prefixTables    = "tables/"
+	prefixFmtTables = prefixTables + "%s/%s/"
+)
+
+// tableKey returns a key based on table name.
+func tableKey(qtid dax.QualifiedTableID) []byte {
+	qual := qtid.TableQualifier
+	key := fmt.Sprintf(prefixFmtTables+"%s", qual.OrganizationID, qual.DatabaseID, qtid.ID)
+	return []byte(key)
+}
+
+// keyTableID gets the table ID out of the key.
+func keyTableID(key []byte) (dax.TableID, error) {
+	parts := strings.Split(string(key), "/")
+	if len(parts) != 4 {
+		return "", errors.New(errors.ErrUncoded, "table key format expected: `tables/orgID/dbID/tableID`")
+	}
+
+	return dax.TableID(parts[3]), nil
+}
+
+// keyQualifiedTableID gets the qualified table ID out of the key.
+func keyQualifiedTableID(key []byte) (dax.QualifiedTableID, error) {
+	parts := strings.Split(string(key), "/")
+	if len(parts) != 4 {
+		return dax.QualifiedTableID{}, errors.New(errors.ErrUncoded, "table key format expected: `tables/orgID/dbID/tableID`")
+	}
+
+	return dax.NewQualifiedTableID(
+		dax.NewTableQualifier(dax.OrganizationID(parts[1]), dax.DatabaseID(parts[2])),
+		dax.TableID(parts[3]),
+	), nil
+}
+
+// shardKey returns a key based on table and shard.
+func shardKey(qtid dax.QualifiedTableID, shard dax.ShardNum) []byte {
+	key := fmt.Sprintf(prefixFmtShards+"%d", qtid.OrganizationID, qtid.DatabaseID, qtid.ID, shard)
+	return []byte(key)
+}
+
+// keyShardNum gets the shardNum out of the key.
+func keyShardNum(key []byte) (dax.ShardNum, error) {
+	parts := strings.Split(string(key), "/")
+	if len(parts) != 5 {
+		return 0, errors.New(errors.ErrUncoded, "shard key format expected: `shards/orgID/dbID/table/shard`")
+	}
+
+	intVar, err := strconv.Atoi(parts[4])
+	if err != nil {
+		return 0, errors.Wrapf(err, "converting string to shardNum: %s", parts[4])
+	}
+
+	return dax.ShardNum(intVar), nil
+}
+
+// partitionKey returns a key based on table and partition.
+func partitionKey(qtid dax.QualifiedTableID, partition dax.PartitionNum) []byte {
+	key := fmt.Sprintf(prefixFmtTableKeys+"%d", qtid.OrganizationID, qtid.DatabaseID, qtid.ID, partition)
+	return []byte(key)
+}
+
+// keyPartitionNum gets the partitionNum out of the key.
+func keyPartitionNum(key []byte) (dax.PartitionNum, error) {
+	parts := strings.Split(string(key), "/")
+	if len(parts) != 5 {
+		return 0, errors.New(errors.ErrUncoded, "partition key format expected: `tablekeys/orgID/dbID/table/partition`")
+	}
+
+	intVar, err := strconv.Atoi(parts[4])
+	if err != nil {
+		return 0, errors.Wrapf(err, "converting string to partitionNum: %s", parts[4])
+	}
+
+	return dax.PartitionNum(intVar), nil
+}
+
+// fieldKey returns a key based on table and field.
+func fieldKey(qtid dax.QualifiedTableID, field dax.FieldName) []byte {
+	key := fmt.Sprintf(prefixFmtFieldKeys+"%s", qtid.OrganizationID, qtid.DatabaseID, qtid.ID, field)
+	return []byte(key)
+}
+
+// keyFieldName gets the fieldName out of the key.
+func keyFieldName(key []byte) (dax.FieldName, error) {
+	parts := strings.Split(string(key), "/")
+	if len(parts) != 5 {
+		return "", errors.New(errors.ErrUncoded, "field key format expected: `fieldkeys/orgID/dbID/table/field`")
+	}
+
+	return dax.FieldName(parts[4]), nil
+}

dax/boltdb/versionstore_test.go

@@ -0,0 +1,388 @@
+package boltdb_test
+
+import (
+	"context"
+	"fmt"
+	"sort"
+	"testing"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/boltdb"
+	testbolt "github.com/molecula/featurebase/v3/dax/test/boltdb"
+	"github.com/molecula/featurebase/v3/logger"
+	"github.com/stretchr/testify/assert"
+)
+
+func TestVersionStore(t *testing.T) {
+	db := testbolt.MustOpenDB(t)
+	defer testbolt.MustCloseDB(t, db)
+
+	ctx := context.Background()
+
+	t.Cleanup(func() {
+		testbolt.CleanupDB(t, db.Path())
+	})
+
+	orgID := dax.OrganizationID("acme")
+	dbID := dax.DatabaseID("db1")
+
+	qual := dax.NewTableQualifier(orgID, dbID)
+
+	// Initialize the buckets.
+	assert.NoError(t, db.InitializeBuckets(boltdb.VersionStoreBuckets...))
+
+	t.Run("Tables", func(t *testing.T) {
+		vs := boltdb.NewVersionStore(db, logger.NopLogger)
+
+		qtids := newQualifiedTableIDs(t, qual, 3)
+		qtid1 := qtids[0]
+		qtid2 := qtids[1]
+		qtid3 := qtids[2]
+		defer vs.RemoveTable(ctx, qtid1)
+		defer vs.RemoveTable(ctx, qtid2)
+		defer vs.RemoveTable(ctx, qtid3)
+
+		// Add table 1.
+		assert.NoError(t, vs.AddTable(ctx, qtid1))
+
+		// Add table 2.
+		assert.NoError(t, vs.AddTable(ctx, qtid2))
+
+		// Add table 3.
+		assert.NoError(t, vs.AddTable(ctx, qtid3))
+	})
+
+	t.Run("Shards", func(t *testing.T) {
+		vs := boltdb.NewVersionStore(db, logger.NopLogger)
+
+		qtids := newQualifiedTableIDs(t, qual, 3)
+		qtid1 := qtids[0]
+		qtid2 := qtids[1]
+		qtid3 := qtids[2]
+
+		// Add tables.
+		assert.NoError(t, vs.AddTable(ctx, qtid1))
+		assert.NoError(t, vs.AddTable(ctx, qtid2))
+		assert.NoError(t, vs.AddTable(ctx, qtid3))
+		defer vs.RemoveTable(ctx, qtid1)
+		defer vs.RemoveTable(ctx, qtid2)
+		defer vs.RemoveTable(ctx, qtid3)
+
+		// Create some shards to insert into the table.
+		shards := make(dax.Shards, 3)
+		for i := range shards {
+			shards[i] = dax.Shard{
+				Num:     dax.ShardNum(i),
+				Version: i * 2,
+			}
+		}
+
+		// Add shards to table 1.
+		{
+			err := vs.AddShards(ctx, qtid1, shards...)
+			assert.NoError(t, err)
+		}
+
+		// Add shards to table 2.
+		{
+			err := vs.AddShards(ctx, qtid2, shards...)
+			assert.NoError(t, err)
+		}
+
+		// Fetch a shard and compare.
+		{
+			version, found, err := vs.ShardVersion(ctx, qtid1, 2)
+			assert.NoError(t, err)
+			assert.True(t, found)
+			assert.Equal(t, 4, version)
+		}
+
+		// Fetch all shards and compare.
+		{
+			shrds, found, err := vs.Shards(ctx, qtid1)
+			assert.NoError(t, err)
+			assert.True(t, found)
+			assert.Equal(t, shards, shrds)
+		}
+
+		// Fetch tables.
+		{
+			tblIDs, err := vs.ShardTables(ctx, qual)
+			assert.NoError(t, err)
+			exp := dax.TableIDs{qtid1.ID, qtid2.ID, qtid3.ID}
+			assert.Equal(t, exp, tblIDs)
+		}
+
+		// Remove table 1.
+		{
+			shards, partitions, err := vs.RemoveTable(ctx, qtid1)
+			assert.NoError(t, err)
+			assert.Equal(t, shards, shards)
+			assert.Equal(t, dax.Partitions{}, partitions)
+		}
+
+		// Fetch all shards and compare.
+		{
+			shrds, found, err := vs.Shards(ctx, qtid1)
+			assert.NoError(t, err)
+			assert.True(t, found)
+			assert.Equal(t, dax.Shards{}, shrds)
+		}
+	})
+
+	t.Run("Partitions", func(t *testing.T) {
+		vs := boltdb.NewVersionStore(db, logger.NopLogger)
+
+		// Create some partitions to insert into the table.
+		partitions := make(dax.Partitions, 3)
+		for i := range partitions {
+			partitions[i] = dax.Partition{
+				Num:     dax.PartitionNum(i),
+				Version: i * 2,
+			}
+		}
+
+		qtids := newQualifiedTableIDs(t, qual, 2)
+		qtid1 := qtids[0]
+		qtid2 := qtids[1]
+
+		// Add tables.
+		assert.NoError(t, vs.AddTable(ctx, qtid1))
+		assert.NoError(t, vs.AddTable(ctx, qtid2))
+		defer vs.RemoveTable(ctx, qtid1)
+		defer vs.RemoveTable(ctx, qtid2)
+
+		// Add partitions to table 1.
+		{
+			err := vs.AddPartitions(ctx, qtid1, partitions...)
+			assert.NoError(t, err)
+		}
+
+		// Add partitions to table 2.
+		{
+			err := vs.AddPartitions(ctx, qtid2, partitions...)
+			assert.NoError(t, err)
+		}
+
+		// Fetch a partition and compare.
+		{
+			version, found, err := vs.PartitionVersion(ctx, qtid1, 2)
+			assert.NoError(t, err)
+			assert.True(t, found)
+			assert.Equal(t, 4, version)
+		}
+
+		// Fetch all partitions and compare.
+		{
+			parts, found, err := vs.Partitions(ctx, qtid1)
+			assert.NoError(t, err)
+			assert.True(t, found)
+			assert.Equal(t, partitions, parts)
+		}
+
+		// Fetch tables.
+		{
+			tblIDs, err := vs.PartitionTables(ctx, qual)
+			assert.NoError(t, err)
+			exp := dax.TableIDs{qtid1.ID, qtid2.ID}
+			assert.Equal(t, exp, tblIDs)
+		}
+
+		// Remove table 1.
+		{
+			shards, partitions, err := vs.RemoveTable(ctx, qtid1)
+			assert.NoError(t, err)
+			assert.Equal(t, dax.Shards{}, shards)
+			assert.Equal(t, partitions, partitions)
+		}
+
+		// Fetch all partitions and compare.
+		{
+			parts, found, err := vs.Partitions(ctx, qtid1)
+			assert.NoError(t, err)
+			assert.True(t, found)
+			assert.Equal(t, dax.Partitions{}, parts)
+		}
+	})
+
+	t.Run("FieldVersions", func(t *testing.T) {
+		vs := boltdb.NewVersionStore(db, logger.NopLogger)
+
+		qtids := newQualifiedTableIDs(t, qual, 2)
+		qtid1 := qtids[0]
+		qtid2 := qtids[1]
+
+		// Add tables.
+		assert.NoError(t, vs.AddTable(ctx, qtid1))
+		assert.NoError(t, vs.AddTable(ctx, qtid2))
+		defer vs.RemoveTable(ctx, qtid1)
+		defer vs.RemoveTable(ctx, qtid2)
+
+		// Create some fieldVersions to insert into the table.
+		fieldVersions := make(dax.FieldVersions, 3)
+		for i := range fieldVersions {
+			fieldVersions[i] = dax.FieldVersion{
+				Name:    dax.FieldName(fmt.Sprintf("fld-%d", i)),
+				Version: i * 2,
+			}
+		}
+
+		// Add fieldVersions to table 1.
+		{
+			err := vs.AddFields(ctx, qtid1, fieldVersions...)
+			assert.NoError(t, err)
+		}
+
+		// Add fieldVersions to table 2.
+		{
+			err := vs.AddFields(ctx, qtid2, fieldVersions...)
+			assert.NoError(t, err)
+		}
+
+		// Fetch a fieldVersion and compare.
+		{
+			version, found, err := vs.FieldVersion(ctx, qtid1, dax.FieldName("fld-2"))
+			assert.NoError(t, err)
+			assert.True(t, found)
+			assert.Equal(t, 4, version)
+		}
+
+		// Fetch all fieldVersions and compare.
+		{
+			flds, found, err := vs.Fields(ctx, qtid1)
+			assert.NoError(t, err)
+			assert.True(t, found)
+			assert.Equal(t, fieldVersions, flds)
+		}
+
+		// Fetch tables.
+		{
+			tblIDs, err := vs.FieldTables(ctx, qual)
+			assert.NoError(t, err)
+			exp := dax.TableIDs{qtid1.ID, qtid2.ID}
+			assert.Equal(t, exp, tblIDs)
+		}
+
+		// Remove table 1.
+		{
+			shards, partitions, err := vs.RemoveTable(ctx, qtid1)
+			assert.NoError(t, err)
+			assert.Equal(t, dax.Shards{}, shards)
+			assert.Equal(t, dax.Partitions{}, partitions)
+		}
+
+		// Fetch all fieldVersions and compare.
+		{
+			flds, found, err := vs.Fields(ctx, qtid1)
+			assert.NoError(t, err)
+			assert.True(t, found)
+			assert.Equal(t, dax.FieldVersions{}, flds)
+		}
+	})
+
+	t.Run("Copy", func(t *testing.T) {
+		vs := boltdb.NewVersionStore(db, logger.NopLogger)
+
+		qtids := newQualifiedTableIDs(t, qual, 1)
+		qtid1 := qtids[0]
+
+		// Add tables.
+		assert.NoError(t, vs.AddTable(ctx, qtid1))
+		defer vs.RemoveTable(ctx, qtid1)
+
+		// Create some shards to insert into the table.
+		shards := make(dax.Shards, 3)
+		for i := range shards {
+			shards[i] = dax.Shard{
+				Num:     dax.ShardNum(i),
+				Version: i * 2,
+			}
+		}
+
+		// Create some partitions to insert into the table.
+		partitions := make(dax.Partitions, 3)
+		for i := range partitions {
+			partitions[i] = dax.Partition{
+				Num:     dax.PartitionNum(i),
+				Version: i * 2,
+			}
+		}
+
+		// Create some fieldVersions to insert into the table.
+		fieldVersions := make(dax.FieldVersions, 3)
+		for i := range fieldVersions {
+			fieldVersions[i] = dax.FieldVersion{
+				Name:    dax.FieldName(fmt.Sprintf("fld-%d", i)),
+				Version: i * 2,
+			}
+		}
+
+		// Add shards to table 1.
+		{
+			err := vs.AddShards(ctx, qtid1, shards...)
+			assert.NoError(t, err)
+		}
+
+		// Add partitions to table 1.
+		{
+			err := vs.AddPartitions(ctx, qtid1, partitions...)
+			assert.NoError(t, err)
+		}
+
+		// Add fieldVersions to table 1.
+		{
+			err := vs.AddFields(ctx, qtid1, fieldVersions...)
+			assert.NoError(t, err)
+		}
+
+		copy, err := vs.Copy(ctx)
+		assert.NoError(t, err)
+
+		// Fetch a shard and compare.
+		{
+			version, found, err := copy.ShardVersion(ctx, qtid1, 2)
+			assert.NoError(t, err)
+			assert.True(t, found)
+			assert.Equal(t, 4, version)
+		}
+
+		// Fetch all partitions and compare.
+		{
+			parts, found, err := copy.Partitions(ctx, qtid1)
+			assert.NoError(t, err)
+			assert.True(t, found)
+			assert.Equal(t, partitions, parts)
+		}
+
+		// Fetch all fieldVersions and compare.
+		{
+			flds, found, err := copy.Fields(ctx, qtid1)
+			assert.NoError(t, err)
+			assert.True(t, found)
+			assert.Equal(t, fieldVersions, flds)
+		}
+	})
+}
+
+// newQualifiedTableIDs is a test helper function which generates a slice of n
+// qtid. The entries in the slice will be ordered by TableID.
+func newQualifiedTableIDs(t *testing.T, qual dax.TableQualifier, n int) []dax.QualifiedTableID {
+	t.Helper()
+
+	qtids := make([]dax.QualifiedTableID, n)
+	for i := range qtids {
+		tbl := dax.NewTable("testvstore")
+		tbl.CreateID()
+		qtids[i] = dax.NewQualifiedTableID(
+			qual,
+			tbl.ID,
+		)
+	}
+
+	// sort the qtids by ID
+	sort.Slice(qtids, func(i, j int) bool {
+		return qtids[i].ID < qtids[j].ID
+	})
+
+	return qtids
+}

dax/computer/alpha/alpha.go

@@ -0,0 +1,24 @@
+package alpha
+
+import (
+	"fmt"
+	"path"
+
+	"github.com/molecula/featurebase/v3/dax"
+)
+
+const (
+	keysFileName = "keys"
+)
+
+func partitionBucket(table dax.TableKey, partition dax.PartitionNum) string {
+	return path.Join(string(table), "partition", fmt.Sprintf("%d", partition))
+}
+
+func shardKey(shard dax.ShardNum) string {
+	return path.Join("shard", fmt.Sprintf("%d", shard))
+}
+
+func fieldBucket(table dax.TableKey, field dax.FieldName) string {
+	return path.Join(string(table), "field", string(field))
+}

dax/computer/alpha/snapshot.go

@@ -0,0 +1,104 @@
+// Package alpha contains an implementation of the SnapshotReadWriter interface.
+// In the case where a sub-service (such as snapshotter) implements these
+// interfaces directly with both its service and its http client, then we don't
+// need this middle implementation layer. But in this case, the Snapshotter
+// operates as a third-party service might, meaning its API methods don't align
+// with what FeatureBase needs to call. So this implementation acts as a
+// translation later between the featurebase-to-snapshotter interface, and the
+// third-party Snapshotter service.
+package alpha
+
+import (
+	"context"
+	"io"
+
+	featurebase "github.com/molecula/featurebase/v3"
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/computer"
+	"github.com/molecula/featurebase/v3/errors"
+)
+
+// Ensure type implements interface.
+var _ computer.SnapshotReadWriter = &alphaSnapshot{}
+
+// alphaSnapshot uses a Snapshotter implementation (which could be, for
+// example, an http client or a locally running sub-service) to store its
+// snapshots.
+type alphaSnapshot struct {
+	ss featurebase.Snapshotter
+}
+
+func NewAlphaSnapshot(sser featurebase.Snapshotter) *alphaSnapshot {
+	return &alphaSnapshot{
+		ss: sser,
+	}
+}
+
+func (s *alphaSnapshot) WriteShardData(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum, version int, rc io.ReadCloser) error {
+	bucket := partitionBucket(qtid.Key(), partition)
+	key := shardKey(shard)
+
+	if err := s.ss.Write(bucket, key, version, rc); err != nil {
+		return errors.Wrapf(err, "writing shard data: %s, %d", key, version)
+	}
+
+	return nil
+}
+
+func (s *alphaSnapshot) ReadShardData(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum, version int) (io.ReadCloser, error) {
+	bucket := partitionBucket(qtid.Key(), partition)
+	key := shardKey(shard)
+
+	rc, err := s.ss.Read(bucket, key, version)
+	if err != nil {
+		return nil, errors.Wrapf(err, "reading shard data: %s, %s, %d", bucket, key, version)
+	}
+
+	return rc, nil
+}
+
+func (s *alphaSnapshot) WriteTableKeys(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, version int, wrTo io.WriterTo) error {
+	bucket := partitionBucket(qtid.Key(), partition)
+	key := keysFileName
+
+	if err := s.ss.WriteTo(bucket, key, version, wrTo); err != nil {
+		return errors.Wrapf(err, "writing table keys: %s, %d", key, version)
+	}
+
+	return nil
+}
+
+func (s *alphaSnapshot) ReadTableKeys(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, version int) (io.ReadCloser, error) {
+	bucket := partitionBucket(qtid.Key(), partition)
+	key := keysFileName
+
+	rc, err := s.ss.Read(bucket, key, version)
+	if err != nil {
+		return nil, errors.Wrapf(err, "reading table keys: %s, %s, %d", bucket, key, version)
+	}
+
+	return rc, nil
+}
+
+func (s *alphaSnapshot) WriteFieldKeys(ctx context.Context, qtid dax.QualifiedTableID, field dax.FieldName, version int, wrTo io.WriterTo) error {
+	bucket := fieldBucket(qtid.Key(), field)
+	key := keysFileName
+
+	if err := s.ss.WriteTo(bucket, key, version, wrTo); err != nil {
+		return errors.Wrapf(err, "writing field keys: %s, %d", key, version)
+	}
+
+	return nil
+}
+
+func (s *alphaSnapshot) ReadFieldKeys(ctx context.Context, qtid dax.QualifiedTableID, field dax.FieldName, version int) (io.ReadCloser, error) {
+	bucket := fieldBucket(qtid.Key(), field)
+	key := keysFileName
+
+	rc, err := s.ss.Read(bucket, key, version)
+	if err != nil {
+		return nil, errors.Wrapf(err, "reading field keys: %s, %s, %d", bucket, key, version)
+	}
+
+	return rc, nil
+}

dax/computer/alpha/writelog.go

@@ -0,0 +1,331 @@
+// Package alpha contains an implementation of the WriteLogReader and
+// WriteLogWriter interfaces. In the case where a sub-service (such as
+// writelogger) implements these interfaces directly with both its service and
+// its http client, then we don't need this middle implementation layer. But in
+// this case, the WriteLogger operates as a third-party service might, meaning
+// its API methods don't align with what FeatureBase needs to call. So this
+// implementation acts as a translation later between the
+// featurebase-to-writelogger interface, and the third-party WriteLogger
+// service.
+package alpha
+
+import (
+	"bufio"
+	"context"
+	"encoding/json"
+	"io"
+
+	featurebase "github.com/molecula/featurebase/v3"
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/computer"
+	"github.com/molecula/featurebase/v3/errors"
+)
+
+// Ensure type implements interface.
+var _ computer.WriteLogReader = &alphaWriteLog{}
+var _ computer.WriteLogWriter = &alphaWriteLog{}
+
+// alphaWriteLog uses a WLer implementation (which could be, for example, an
+// http client or a locally running sub-service) to store its log messages.
+type alphaWriteLog struct {
+	wl featurebase.WriteLogger
+}
+
+func NewAlphaWriteLog(wler featurebase.WriteLogger) *alphaWriteLog {
+	return &alphaWriteLog{
+		wl: wler,
+	}
+}
+
+func (w *alphaWriteLog) CreateTableKeys(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, version int, m map[string]uint64) error {
+	msg := computer.PartitionKeyMap{
+		TableKey:   qtid.Key(),
+		Partition:  partition,
+		StringToID: m,
+	}
+
+	b, err := json.Marshal(msg)
+	if err != nil {
+		return errors.Wrap(err, "marshalling partition key map to json")
+	}
+
+	bucket := partitionBucket(qtid.Key(), partition)
+
+	if err := w.wl.AppendMessage(bucket, keysFileName, version, b); err != nil {
+		return errors.Wrapf(err, "appending partition key message: %s, %d", keysFileName, version)
+	}
+
+	return nil
+}
+
+func (w *alphaWriteLog) DeleteTableKeys(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, version int) error {
+	bucket := partitionBucket(qtid.Key(), partition)
+	return w.wl.DeleteLog(bucket, keysFileName, version)
+}
+
+func (w *alphaWriteLog) CreateFieldKeys(ctx context.Context, qtid dax.QualifiedTableID, field dax.FieldName, version int, m map[string]uint64) error {
+	msg := computer.FieldKeyMap{
+		TableKey:   qtid.Key(),
+		Field:      field,
+		StringToID: m,
+	}
+
+	b, err := json.Marshal(msg)
+	if err != nil {
+		return errors.Wrap(err, "marshalling field key map to json")
+	}
+
+	bucket := fieldBucket(qtid.Key(), field)
+
+	if err := w.wl.AppendMessage(bucket, keysFileName, version, b); err != nil {
+		return errors.Wrapf(err, "appending field key message: %s, %d", keysFileName, version)
+	}
+
+	return nil
+}
+
+func (w *alphaWriteLog) DeleteFieldKeys(ctx context.Context, qtid dax.QualifiedTableID, field dax.FieldName, version int) error {
+	bucket := fieldBucket(qtid.Key(), field)
+	return w.wl.DeleteLog(bucket, keysFileName, version)
+}
+
+func (w *alphaWriteLog) WriteShard(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum, version int, msg computer.LogMessage) error {
+	b, err := computer.MarshalLogMessage(msg)
+	if err != nil {
+		return errors.Wrap(err, "marshalling log message")
+	}
+
+	bucket := partitionBucket(qtid.Key(), partition)
+	shardKey := shardKey(shard)
+
+	if err := w.wl.AppendMessage(bucket, shardKey, version, b); err != nil {
+		return errors.Wrapf(err, "appending shard key message: %s, %d", shardKey, version)
+	}
+
+	return nil
+}
+
+func (w *alphaWriteLog) DeleteShard(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum, version int) error {
+	bucket := partitionBucket(qtid.Key(), partition)
+	shardKey := shardKey(shard)
+
+	return w.wl.DeleteLog(bucket, shardKey, version)
+}
+
+////////////////////////////////////////////////
+
+func (w *alphaWriteLog) TableKeyReader(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, version int) computer.TableKeyReader {
+	return newTableKeyReader(w.wl, qtid, partition, version)
+}
+
+type tableKeyReader struct {
+	wl        featurebase.WriteLogger
+	table     dax.TableKey
+	partition dax.PartitionNum
+	version   int
+	scanner   *bufio.Scanner
+	closer    io.Closer
+}
+
+func newTableKeyReader(wl featurebase.WriteLogger, qtid dax.QualifiedTableID, partition dax.PartitionNum, version int) *tableKeyReader {
+	r := &tableKeyReader{
+		wl:        wl,
+		table:     qtid.Key(),
+		partition: partition,
+		version:   version,
+	}
+
+	return r
+}
+
+func (r *tableKeyReader) Open() error {
+	bucket := partitionBucket(r.table, r.partition)
+
+	reader, closer, err := r.wl.LogReader(bucket, keysFileName, r.version)
+	if err != nil {
+		return errors.Wrapf(err, "getting log reader: %s, %s, %d", bucket, keysFileName, r.version)
+	}
+
+	r.closer = closer
+	r.scanner = bufio.NewScanner(reader)
+
+	return nil
+}
+
+func (r *tableKeyReader) Read() (computer.PartitionKeyMap, error) {
+	if r.scanner == nil {
+		return computer.PartitionKeyMap{}, io.EOF
+	}
+
+	var b []byte
+	var out computer.PartitionKeyMap
+
+	if r.scanner.Scan() {
+		b = r.scanner.Bytes()
+		if err := json.Unmarshal(b, &out); err != nil {
+			return out, err
+		}
+		return out, nil
+	}
+	if err := r.scanner.Err(); err != nil {
+		return out, err
+	}
+
+	return out, io.EOF
+}
+
+func (r *tableKeyReader) Close() error {
+	if r.closer != nil {
+		return r.closer.Close()
+	}
+	return nil
+}
+
+////////////////////////////////////////////////
+
+func (w *alphaWriteLog) FieldKeyReader(ctx context.Context, qtid dax.QualifiedTableID, field dax.FieldName, version int) computer.FieldKeyReader {
+	return newFieldKeyReader(w.wl, qtid, field, version)
+}
+
+type fieldKeyReader struct {
+	wl      featurebase.WriteLogger
+	table   dax.TableKey
+	field   dax.FieldName
+	version int
+	scanner *bufio.Scanner
+	closer  io.Closer
+}
+
+func newFieldKeyReader(wl featurebase.WriteLogger, qtid dax.QualifiedTableID, field dax.FieldName, version int) *fieldKeyReader {
+	r := &fieldKeyReader{
+		wl:      wl,
+		table:   qtid.Key(),
+		field:   field,
+		version: version,
+	}
+
+	return r
+}
+
+func (r *fieldKeyReader) Open() error {
+	bucket := fieldBucket(r.table, r.field)
+
+	reader, closer, err := r.wl.LogReader(bucket, keysFileName, r.version)
+	if err != nil {
+		return errors.Wrapf(err, "getting log reader: %s, %s, %d", bucket, keysFileName, r.version)
+	}
+
+	r.closer = closer
+	r.scanner = bufio.NewScanner(reader)
+
+	return nil
+}
+
+func (r *fieldKeyReader) Read() (computer.FieldKeyMap, error) {
+	if r.scanner == nil {
+		return computer.FieldKeyMap{}, io.EOF
+	}
+
+	var b []byte
+	var out computer.FieldKeyMap
+
+	if r.scanner.Scan() {
+		b = r.scanner.Bytes()
+		if err := json.Unmarshal(b, &out); err != nil {
+			return out, err
+		}
+		return out, nil
+	}
+	if err := r.scanner.Err(); err != nil {
+		return out, err
+	}
+
+	return out, io.EOF
+}
+
+func (r *fieldKeyReader) Close() error {
+	if r.closer != nil {
+		return r.closer.Close()
+	}
+	return nil
+}
+
+////////////////////////////////////////////////
+
+func (w *alphaWriteLog) ShardReader(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum, version int) computer.ShardReader {
+	return newShardReader(w.wl, qtid, partition, shard, version)
+}
+
+type shardReader struct {
+	wl        featurebase.WriteLogger
+	table     dax.TableKey
+	partition dax.PartitionNum
+	shard     dax.ShardNum
+	version   int
+	scanner   *bufio.Scanner
+	closer    io.Closer
+}
+
+func newShardReader(wl featurebase.WriteLogger, qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum, version int) *shardReader {
+	r := &shardReader{
+		wl:        wl,
+		table:     qtid.Key(),
+		partition: partition,
+		shard:     shard,
+		version:   version,
+	}
+
+	return r
+}
+
+func (r *shardReader) Open() error {
+	bucket := partitionBucket(r.table, r.partition)
+	shardKey := shardKey(r.shard)
+
+	reader, closer, err := r.wl.LogReader(bucket, shardKey, r.version)
+	if err != nil {
+		return errors.Wrapf(err, "getting log reader: %s, %s, %d", bucket, shardKey, r.version)
+	}
+
+	r.closer = closer
+	r.scanner = bufio.NewScanner(reader)
+
+	return nil
+}
+
+func (r *shardReader) Read() (computer.LogMessage, error) {
+	if r.scanner == nil {
+		return nil, io.EOF
+	}
+
+	if r.scanner.Scan() {
+		b := r.scanner.Bytes()
+
+		if len(b) == 0 {
+			return nil, errors.New(errors.ErrUncoded, "empty log record")
+		}
+		logMessageType := b[0]
+
+		msg, err := computer.LogMessageByType(logMessageType)
+		if err != nil {
+			return nil, errors.Wrap(err, "getting log message by type")
+		}
+
+		if err := json.Unmarshal(b[1:], &msg); err != nil {
+			return nil, errors.Wrap(err, "unmarshaling log message")
+		}
+		return msg, nil
+	}
+	if err := r.scanner.Err(); err != nil {
+		return nil, err
+	}
+
+	return nil, io.EOF
+}
+
+func (r *shardReader) Close() error {
+	if r.closer != nil {
+		return r.closer.Close()
+	}
+	return nil
+}

dax/computer/api/openapi.yaml

@@ -0,0 +1,235 @@
+openapi: 3.0.3
+
+info:
+  title: Computer
+  description: The dax-related API for the Computer service.
+  version: 0.0.0
+
+paths:
+  /computer/health:
+    get:
+      summary: Health check endpoint.
+      description: Provides an endpoint to check the overall health of the Computer service.
+      operationId: GetHealth
+      responses:
+        200:
+          description: Service is healthy.
+
+  /computer/directive:
+    post:
+      summary: Post Directive to compute node.
+      description: Post a Directive to the compute node.
+      operationId: PostDirective
+      requestBody:
+        content:
+          application/json:
+            schema:
+              $ref: '#/components/schemas/Directive'
+      responses:
+        200:
+          description: Directive was applied successfully.
+
+  /computer/snapshot/shard-data:
+    post:
+      summary: Request to snapshot shard data.
+      description: Request to snapshot shard data.
+      operationId: PostSnapshotShardData
+      requestBody:
+        content:
+          application/json:
+            schema:
+              type: object
+              properties:
+                address:
+                  type: string
+                table:
+                  type: string
+                shard:
+                  type: integer
+                  format: int64
+                fromVersion:
+                  type: integer
+                  format: int64
+                toVersion:
+                  type: integer
+                  format: int64
+                directive:
+                  $ref: '#/components/schemas/Directive'
+                  
+      responses:
+        200:
+          description: Shard snapshot was successful.
+
+  /computer/snapshot/table-keys:
+    post:
+      summary: Request to snapshot table keys.
+      description: Request to snapshot table keys.
+      operationId: PostSnapshotTableKeys
+      requestBody:
+        content:
+          application/json:
+            schema:
+              type: object
+              properties:
+                address:
+                  type: string
+                table:
+                  type: string
+                partition:
+                  type: integer
+                  format: int32
+                fromVersion:
+                  type: integer
+                  format: int64
+                toVersion:
+                  type: integer
+                  format: int64
+                directive:
+                  $ref: '#/components/schemas/Directive'
+                  
+      responses:
+        200:
+          description: Table keys snapshot was successful.
+
+  /computer/snapshot/field-keys:
+    post:
+      summary: Request to snapshot field keys.
+      description: Request to snapshot field keys.
+      operationId: PostSnapshotFieldKeys
+      requestBody:
+        content:
+          application/json:
+            schema:
+              type: object
+              properties:
+                address:
+                  type: string
+                table:
+                  type: string
+                field:
+                  type: string
+                fromVersion:
+                  type: integer
+                  format: int64
+                toVersion:
+                  type: integer
+                  format: int64
+                directive:
+                  $ref: '#/components/schemas/Directive'
+                  
+      responses:
+        200:
+          description: Field keys snapshot was successful.
+
+components:
+  responses:
+    Directive:
+      description: Directive response.
+      content:
+        application/json:
+          schema:
+            $ref: '#/components/schemas/Directive'
+
+  schemas:
+    Directive:
+      type: object
+      properties:
+        address:
+          type: string
+        tables:
+          type: array
+          items:
+            $ref: '#/components/schemas/Table'
+        computeRoles:
+          type: array
+          items:
+            type: object
+            properties:
+              table:
+                type: string
+              shards:
+                type: array
+                items:
+                  type: integer
+                  format: int64
+        translateRoles:
+          type: array
+          items:
+            type: object
+            properties:
+              table:
+                type: string
+              partitions:
+                type: array
+                items:
+                  type: integer
+                  format: int32
+              fields:
+                type: array
+                items:
+                  type: string
+        version:
+          type: integer
+          format: int64
+
+    # This is copied from /mds/api/openapi.yaml. TODO: share schemas across yaml files.
+    Table:
+      type: object
+      properties:
+        name:
+          type: string
+        fields:
+          type: array
+          items:
+            $ref: '#/components/schemas/Field'
+        partitionN:
+          type: integer
+          format: int32
+
+    Field:
+      type: object
+      properties:
+        name:
+          type: string
+        type:
+          type: string
+          enum:
+          - bool
+          - decimal
+          - id
+          - idset
+          - int
+          - string
+          - stringset
+          - timestamp
+        options:
+          type: object
+          properties:
+            min:
+              type: integer
+              format: int64
+            max:
+              type: integer
+              format: int64
+            scale:
+              type: integer
+              format: int64
+              minimum: 0
+            noStandardView:
+              type: boolean
+            cacheType:
+              type: string
+            cacheSize:
+              type: integer
+              format: int32
+            timeUnit:
+              type: string
+            epoch:
+              type: string
+              format: date-time
+            timeQuantum:
+              type: string
+            ttl:
+              type: string
+            foreignIndex:
+              type: string

dax/computer/computer.go

@@ -0,0 +1,5 @@
+// Package computer contains the compute-specific portions of the DAX
+// architecture. In general, this is a dumb FeatureBase node (or service) which
+// essentially contains the Executor and its interaction with the underlying
+// data.
+package computer

dax/computer/snapshot.go

@@ -0,0 +1,61 @@
+package computer
+
+import (
+	"context"
+	"io"
+
+	"github.com/molecula/featurebase/v3/dax"
+)
+
+// SnapshotReadWriter provides the interface for all snapshot read and writes in
+// FeatureBase.
+type SnapshotReadWriter interface {
+	WriteShardData(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum, version int, rc io.ReadCloser) error
+	ReadShardData(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum, version int) (io.ReadCloser, error)
+
+	WriteTableKeys(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, version int, wrTo io.WriterTo) error
+	ReadTableKeys(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, version int) (io.ReadCloser, error)
+
+	WriteFieldKeys(ctx context.Context, qtid dax.QualifiedTableID, field dax.FieldName, version int, wrTo io.WriterTo) error
+	ReadFieldKeys(ctx context.Context, qtid dax.QualifiedTableID, field dax.FieldName, version int) (io.ReadCloser, error)
+}
+
+// Ensure type implements interface.
+var _ SnapshotReadWriter = &NopSnapshotReadWriter{}
+
+// NopSnapshotReadWriter is a no-op implementation of the SnapshotReadWriter
+// interface.
+type NopSnapshotReadWriter struct{}
+
+func NewNopSnapshotReadWriter() *NopSnapshotReadWriter {
+	return &NopSnapshotReadWriter{}
+}
+
+func (w *NopSnapshotReadWriter) WriteShardData(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum, version int, rc io.ReadCloser) error {
+	return nil
+}
+
+func (w *NopSnapshotReadWriter) ReadShardData(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum, version int) (io.ReadCloser, error) {
+	return &nopReadCloser{}, nil
+}
+
+func (w *NopSnapshotReadWriter) WriteTableKeys(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, version int, wrTo io.WriterTo) error {
+	return nil
+}
+
+func (w *NopSnapshotReadWriter) ReadTableKeys(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, version int) (io.ReadCloser, error) {
+	return &nopReadCloser{}, nil
+}
+
+func (w *NopSnapshotReadWriter) WriteFieldKeys(ctx context.Context, qtid dax.QualifiedTableID, field dax.FieldName, version int, wrTo io.WriterTo) error {
+	return nil
+}
+
+func (w *NopSnapshotReadWriter) ReadFieldKeys(ctx context.Context, qtid dax.QualifiedTableID, field dax.FieldName, version int) (io.ReadCloser, error) {
+	return &nopReadCloser{}, nil
+}
+
+type nopReadCloser struct{}
+
+func (n *nopReadCloser) Read([]byte) (int, error) { return 0, nil }
+func (n *nopReadCloser) Close() error             { return nil }

dax/computer/writelog.go

@@ -0,0 +1,307 @@
+package computer
+
+import (
+	"context"
+	"encoding/json"
+	"io"
+	"time"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/errors"
+)
+
+// WriteLogWriter provides the interface for all data writes to FeatureBase. After
+// data has been written to the local FeatureBase node, the respective interface
+// method(s) will be called.
+type WriteLogWriter interface {
+	// CreateTableKeys sends a map of string key to uint64 ID for the table and
+	// partition provided.
+	CreateTableKeys(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, version int, _ map[string]uint64) error
+	DeleteTableKeys(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, version int) error
+
+	// CreateFieldKeys sends a map of string key to uint64 ID for the table and
+	// field provided.
+	CreateFieldKeys(ctx context.Context, qtid dax.QualifiedTableID, field dax.FieldName, version int, _ map[string]uint64) error
+	DeleteFieldKeys(ctx context.Context, qtid dax.QualifiedTableID, field dax.FieldName, version int) error
+
+	WriteShard(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum, version int, msg LogMessage) error
+	DeleteShard(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum, version int) error
+}
+
+// Ensure type implements interface.
+var _ WriteLogWriter = (*NopWriteLogWriter)(nil)
+
+// NopWriteLogWriter is a no-op implementation of the WriteLogWriter interface.
+type NopWriteLogWriter struct{}
+
+func NewNopWriteLogWriter() *NopWriteLogWriter {
+	return &NopWriteLogWriter{}
+}
+
+func (w *NopWriteLogWriter) CreateTableKeys(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, version int, m map[string]uint64) error {
+	return nil
+}
+
+func (w *NopWriteLogWriter) DeleteTableKeys(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, version int) error {
+	return nil
+}
+
+func (w *NopWriteLogWriter) CreateFieldKeys(ctx context.Context, qtid dax.QualifiedTableID, field dax.FieldName, version int, m map[string]uint64) error {
+	return nil
+}
+
+func (w *NopWriteLogWriter) DeleteFieldKeys(ctx context.Context, qtid dax.QualifiedTableID, field dax.FieldName, version int) error {
+	return nil
+}
+
+func (w *NopWriteLogWriter) WriteShard(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum, version int, msg LogMessage) error {
+	return nil
+}
+
+func (w *NopWriteLogWriter) DeleteShard(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum, version int) error {
+	return nil
+}
+
+// WriteLogReader provides the interface for all reads from the write log.
+type WriteLogReader interface {
+	ShardReader(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum, version int) ShardReader
+	TableKeyReader(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, version int) TableKeyReader
+	FieldKeyReader(ctx context.Context, qtid dax.QualifiedTableID, field dax.FieldName, version int) FieldKeyReader
+}
+
+// Ensure type implements interface.
+var _ WriteLogReader = (*NopWriteLogReader)(nil)
+
+// NopWriteLogReader is a no-op implementation of the WriteLogReader interface.
+type NopWriteLogReader struct{}
+
+func NewNopWriteLogReader() *NopWriteLogReader {
+	return &NopWriteLogReader{}
+}
+
+func (w *NopWriteLogReader) TableKeyReader(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, version int) TableKeyReader {
+	return NewNopTableKeyReader()
+}
+
+func (w *NopWriteLogReader) FieldKeyReader(ctx context.Context, qtid dax.QualifiedTableID, field dax.FieldName, version int) FieldKeyReader {
+	return NewNopFieldKeyReader()
+}
+
+func (w *NopWriteLogReader) ShardReader(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum, version int) ShardReader {
+	return NewNopShardReader()
+}
+
+////////////////////////////////////////////////
+
+type TableKeyReader interface {
+	Open() error
+	Read() (PartitionKeyMap, error)
+	Close() error
+}
+
+// Ensure type implements interface.
+var _ TableKeyReader = &NopTableKeyReader{}
+
+// NopTableKeyReader is a no-op implementation of the TableKeyReader
+// interface.
+type NopTableKeyReader struct{}
+
+func NewNopTableKeyReader() *NopTableKeyReader {
+	return &NopTableKeyReader{}
+}
+
+func (r *NopTableKeyReader) Open() error { return nil }
+func (r *NopTableKeyReader) Read() (PartitionKeyMap, error) {
+	return PartitionKeyMap{}, io.EOF
+}
+func (r *NopTableKeyReader) Close() error { return nil }
+
+////////////////////////////////////////////////
+
+type FieldKeyReader interface {
+	Open() error
+	Read() (FieldKeyMap, error)
+	Close() error
+}
+
+// Ensure type implements interface.
+var _ FieldKeyReader = &NopFieldKeyReader{}
+
+// NopFieldKeyReader is a no-op implementation of the FieldKeyReader
+// interface.
+type NopFieldKeyReader struct{}
+
+func NewNopFieldKeyReader() *NopFieldKeyReader {
+	return &NopFieldKeyReader{}
+}
+
+func (r *NopFieldKeyReader) Open() error { return nil }
+func (r *NopFieldKeyReader) Read() (FieldKeyMap, error) {
+	return FieldKeyMap{}, io.EOF
+}
+func (r *NopFieldKeyReader) Close() error { return nil }
+
+////////////////////////////////////////////////
+
+type ShardReader interface {
+	Open() error
+	Read() (LogMessage, error)
+	Close() error
+}
+
+// Ensure type implements interface.
+var _ ShardReader = &NopShardReader{}
+
+// NopShardReader is a no-op implementation of the ShardReader interface.
+type NopShardReader struct{}
+
+func NewNopShardReader() *NopShardReader {
+	return &NopShardReader{}
+}
+
+func (r *NopShardReader) Open() error { return nil }
+func (r *NopShardReader) Read() (LogMessage, error) {
+	return nil, io.EOF
+}
+func (r *NopShardReader) Close() error { return nil }
+
+//////////////// Messages ///////////////////////
+
+type PartitionKeyMap struct {
+	TableKey   dax.TableKey      `json:"table-key"`
+	Partition  dax.PartitionNum  `json:"partition"`
+	StringToID map[string]uint64 `json:"string-to-id"`
+}
+
+type FieldKeyMap struct {
+	TableKey   dax.TableKey      `json:"table-key"`
+	Field      dax.FieldName     `json:"field"`
+	StringToID map[string]uint64 `json:"string-to-id"`
+}
+
+const (
+	logMessageTypeImportRoaring = iota
+	logMessageTypeImport
+	logMessageTypeImportValue
+	logMessageTypeImportRoaringShard
+)
+
+type LogMessage interface{}
+
+// MarshalLogMessage serializes the log message and adds log message type info.
+func MarshalLogMessage(msg LogMessage) ([]byte, error) {
+	typ, err := getLogMessageType(msg)
+	if err != nil {
+		return nil, errors.Wrap(err, "getting log message type")
+	}
+
+	buf, err := json.Marshal(msg)
+	if err != nil {
+		return nil, errors.Wrap(err, "marshaling log message")
+	}
+	return append([]byte{typ}, buf...), nil
+}
+
+func LogMessageByType(typ byte) (LogMessage, error) {
+	switch typ {
+	case logMessageTypeImportRoaring:
+		return &ImportRoaringMessage{}, nil
+	case logMessageTypeImport:
+		return &ImportMessage{}, nil
+	case logMessageTypeImportValue:
+		return &ImportValueMessage{}, nil
+	case logMessageTypeImportRoaringShard:
+		return &ImportRoaringShardMessage{}, nil
+	default:
+		return nil, errors.Errorf("unknown message type %d", typ)
+	}
+}
+
+func getLogMessageType(m LogMessage) (byte, error) {
+	switch m.(type) {
+	case *ImportRoaringMessage:
+		return logMessageTypeImportRoaring, nil
+	case *ImportMessage:
+		return logMessageTypeImport, nil
+	case *ImportValueMessage:
+		return logMessageTypeImportValue, nil
+	case *ImportRoaringShardMessage:
+		return logMessageTypeImportRoaringShard, nil
+	default:
+		return 0, errors.Errorf("don't have type for message %#v", m)
+	}
+}
+
+type ImportRoaringMessage struct {
+	LogMessage `json:"-"`
+
+	Table           string            `json:"table"`
+	Field           string            `json:"field"`
+	Partition       int               `json:"partition"`
+	Shard           uint64            `json:"shard"`
+	Clear           bool              `json:"clear"`
+	Action          string            `json:"action"` // [set, clear, overwrite]
+	Block           int               `json:"block"`
+	Views           map[string][]byte `json:"views"`
+	UpdateExistence bool              `json:"update-existence"`
+}
+
+type ImportMessage struct {
+	LogMessage `json:"-"`
+
+	Table      string   `json:"table"`
+	Field      string   `json:"field"`
+	Partition  int      `json:"partition"`
+	Shard      uint64   `json:"shard"`
+	RowIDs     []uint64 `json:"row-ids"`
+	ColumnIDs  []uint64 `json:"column-ids"`
+	RowKeys    []string `json:"row-keys"`
+	ColumnKeys []string `json:"column-keys"`
+	Timestamps []int64  `json:"timestamps"`
+	Clear      bool     `json:"clear"`
+
+	// options
+	IgnoreKeyCheck bool `json:"ignore-key-check"`
+	Presorted      bool `json:"presorted"`
+}
+
+type ImportValueMessage struct {
+	LogMessage `json:"-"`
+
+	Table           string      `json:"table"`
+	Field           string      `json:"field"`
+	Partition       int         `json:"partition"`
+	Shard           uint64      `json:"shard"`
+	ColumnIDs       []uint64    `json:"column-ids"`
+	ColumnKeys      []string    `json:"column-keys"`
+	Values          []int64     `json:"values"`
+	FloatValues     []float64   `json:"float-values"`
+	TimestampValues []time.Time `json:"timestamp-values"`
+	StringValues    []string    `json:"string-values"`
+	Clear           bool        `json:"clear"`
+
+	// options
+	IgnoreKeyCheck bool `json:"ignore-key-check"`
+	Presorted      bool `json:"presorted"`
+}
+
+type ImportRoaringShardMessage struct {
+	LogMessage `json:"-"`
+
+	Table     string          `json:"table"`
+	Partition int             `json:"partition"`
+	Shard     uint64          `json:"shard"`
+	Views     []RoaringUpdate `json:"views"`
+}
+
+// RoaringUpdate is identical to featurebase.RoaringUpdate, but we
+// can't import it due to import cycles. TODO featurebase top level
+// shouldn't import dax stuff... all the types it needs should just be
+// in the top level.
+type RoaringUpdate struct {
+	Field        string `json:"field"`
+	View         string `json:"view"`
+	Clear        []byte `json:"clear"`
+	Set          []byte `json:"set"`
+	ClearRecords bool   `json:"clear-records"`
+}

dax/dax.go

@@ -0,0 +1,11 @@
+// Package dax defines DAX domain level types.
+package dax
+
+// ServicePrefixes are used as the service prefix value in http handlers.
+const (
+	ServicePrefixComputer    = "computer"
+	ServicePrefixMDS         = "mds"
+	ServicePrefixQueryer     = "queryer"
+	ServicePrefixSnapshotter = "snapshotter"
+	ServicePrefixWriteLogger = "writelogger"
+)

dax/directive.go

@@ -0,0 +1,175 @@
+package dax
+
+// Directive contains the instructions, sent from MDS, which a compute node is
+// to follow. A Directive is typically JSON-encoded and POSTed to a compute
+// node's `/directive` endpoint.
+type Directive struct {
+	Address Address `json:"address"`
+
+	// Method describes how the compute node should handle the Directive. See
+	// the different constants of type DirectiveMethod for how this value is
+	// handled.
+	Method DirectiveMethod `json:"method"`
+
+	Tables []*QualifiedTable `json:"schema"`
+
+	ComputeRoles   []ComputeRole   `json:"compute-roles"`
+	TranslateRoles []TranslateRole `json:"translate-roles"`
+
+	Version uint64 `json:"version"`
+}
+
+// DirectiveMethod is used to tell the compute node how it should handle the
+// Directive.
+type DirectiveMethod string
+
+const (
+	// DirectiveMethodDiff tells the compute node to diff the Directive with its
+	// local, cached Directive and only apply the differences.
+	DirectiveMethodDiff DirectiveMethod = "diff"
+
+	// DirectiveMethodReset tells the compute node to delete all of its existing
+	// data before applying the directive.
+	DirectiveMethodReset DirectiveMethod = "reset"
+
+	// DirectiveMethodSnapshot tells the compute node that the incoming
+	// Directive should only contain data version updates related to a snapshot
+	// request.
+	DirectiveMethodSnapshot DirectiveMethod = "snapshot"
+)
+
+// Table returns the ID'd table from the Directive's Tables list. If it's not
+// found, it returns nil and a non-nil error. A nil error guarantees that the
+// returned table is non-nil.
+func (d *Directive) Table(qtid QualifiedTableID) (*QualifiedTable, error) {
+	for _, qtbl := range d.Tables {
+		// We can't do qtbl.QualifiedID() == qtid because the value of qtid.Name
+		// is empty and causes the equality check to fail. Hence the .Equals()
+		// method.
+		if qtbl.QualifiedID().Equals(qtid) {
+			return qtbl, nil
+		}
+	}
+	return nil, NewErrTableIDDoesNotExist(qtid)
+}
+
+// ComputeShards returns the list of shards, for the given table, for which this
+// compute node is responsible. It assumes that the Directive does not contain
+// more than one ComputeRole for the same table; in that case, we would need to
+// return the union of Shards.
+func (d *Directive) ComputeShards(tbl TableKey) Shards {
+	if d == nil || d.ComputeRoles == nil {
+		return Shards{}
+	}
+
+	for _, cr := range d.ComputeRoles {
+		if cr.TableKey == tbl {
+			return cr.Shards
+		}
+	}
+
+	return Shards{}
+}
+
+// ComputeShardsMap returns a map of table to shards. It assumes that the
+// Directive does not contain more than one ComputeRole for the same table; in
+// that case, we would need to return the union of Shards.
+func (d *Directive) ComputeShardsMap() map[TableKey]Shards {
+	m := make(map[TableKey]Shards)
+	if d == nil || d.ComputeRoles == nil {
+		return m
+	}
+
+	for _, cr := range d.ComputeRoles {
+		m[cr.TableKey] = cr.Shards
+	}
+
+	return m
+}
+
+// TranslatePartitions returns the list of partitions, for the given table, for
+// which this translate node is responsible. It assumes that the Directive does
+// not contain more than one TranslateRole for the same table; in that case, we
+// would need to return the union of Shards.
+func (d *Directive) TranslatePartitions(tbl TableKey) Partitions {
+	if d == nil || d.TranslateRoles == nil {
+		return Partitions{}
+	}
+
+	for _, tr := range d.TranslateRoles {
+		if tr.TableKey == tbl {
+			return tr.Partitions
+		}
+	}
+	return Partitions{}
+}
+
+// TranslatePartitionsMap returns a map of table to partitions. It assumes that
+// the Directive does not contain more than one TranslateRole for the same
+// table; in that case, we would need to return the union of Partitions.
+func (d *Directive) TranslatePartitionsMap() map[TableKey]Partitions {
+	m := make(map[TableKey]Partitions)
+	if d == nil || d.TranslateRoles == nil {
+		return m
+	}
+
+	for _, tr := range d.TranslateRoles {
+		// Since we added FieldVersions to the TranslateRole, it's possible for
+		// a TranslateRole to have an empty Partitions list. In that case, we
+		// want to exclude that from the map.
+		if len(tr.Partitions) == 0 {
+			continue
+		}
+		m[tr.TableKey] = tr.Partitions
+	}
+
+	return m
+}
+
+// TranslateFieldsMap returns a map of table to fields. It assumes that
+// the Directive does not contain more than one TranslateRole for the same
+// table; in that case, we would need to return the union of FieldValues.
+func (d *Directive) TranslateFieldsMap() map[TableKey]FieldVersions {
+	m := make(map[TableKey]FieldVersions)
+	if d == nil || d.TranslateRoles == nil {
+		return m
+	}
+
+	for _, tr := range d.TranslateRoles {
+		if len(tr.Fields) == 0 {
+			continue
+		}
+		m[tr.TableKey] = tr.Fields
+	}
+
+	return m
+}
+
+// IsEmpty tells whether a directive is assigning actual responsibilty
+// to a node or not. If the directive does not assign responsibility
+// for any shard or partition then it is considered empty. This is
+// used to determine whether we can ignore an error received from
+// applying this directive (an empty directive is often sent to a node
+// which is already down).
+func (d *Directive) IsEmpty() bool {
+	for _, role := range d.ComputeRoles {
+		if len(role.Shards) > 0 {
+			return false
+		}
+	}
+
+	for _, role := range d.TranslateRoles {
+		if len(role.Partitions) > 0 {
+			return false
+		}
+	}
+
+	return true
+}
+
+// Directives is a sortable slice of Directive.
+type Directives []*Directive
+
+func (d Directives) Len() int           { return len(d) }
+func (d Directives) Less(i, j int) bool { return d[i].Address.String() < d[j].Address.String() }
+func (d Directives) Swap(i, j int)      { d[i], d[j] = d[j], d[i] }

dax/docker-compose.yml

@@ -0,0 +1,66 @@
+version: '3'
+
+services:
+  mds:
+    build:
+      context: ../.quick
+      dockerfile: ../Dockerfile-dax-quick
+    environment:
+      FEATUREBASE_BIND: 0.0.0.0:8080
+      FEATUREBASE_VERBOSE: "true"
+      FEATUREBASE_STORAGE_METHOD: boltdb
+      FEATUREBASE_STORAGE_DSN: file:/dax-data/mds.boldtb
+      FEATUREBASE_MDS_RUN: "true"
+    ports:
+      - "8081:8080"
+
+  queryer:
+    build:
+      context: ../.quick
+      dockerfile: ../Dockerfile-dax-quick
+    environment:
+      FEATUREBASE_BIND: 0.0.0.0:8080
+      FEATUREBASE_VERBOSE: "true"
+      FEATUREBASE_QUERYER_RUN: "true"
+      FEATUREBASE_QUERYER_CONFIG_MDS_ADDRESS: "mds:8080"
+    depends_on:
+      - mds
+    ports:
+      - "8080:8080"
+
+  computer:
+    build:
+      context: ../.quick
+      dockerfile: ../Dockerfile-dax-quick
+    environment:
+      FEATUREBASE_COMPUTER_RUN: "true"
+      FEATUREBASE_COMPUTER_CONFIG_MDS_ADDRESS: "mds:8080"
+      FEATUREBASE_COMPUTER_CONFIG_DATA_DIR: /dax-data/computer
+      FEATUREBASE_BIND: 0.0.0.0:8080
+      FEATUREBASE_VERBOSE: "true"
+      FEATUREBASE_STORAGE_METHOD: boltdb
+      FEATUREBASE_WRITELOGGER_RUN: "true"
+      FEATUREBASE_WRITELOGGER_CONFIG_DATA_DIR: "/dax-data/writelogger"
+      FEATUREBASE_SNAPSHOTTER_RUN: "true"
+      FEATUREBASE_SNAPSHOTTER_CONFIG_DATA_DIR: "/dax-data/snapshotter"
+    volumes:
+      - "./dax-data/writelogger:/dax-data/writelogger"
+      - "./dax-data/snapshotter:/dax-data/snapshotter"
+    depends_on:
+      - mds
+    deploy:
+      replicas: 1
+
+  datagen:
+    build:
+      context: ..
+      dockerfile: Dockerfile-datagen
+    profiles: [ "datagen" ]
+    environment:
+      GEN_CUSTOM_CONFIG: "/testdata/keys_ids.yaml"
+      GEN_FEATUREBASE_ORG_ID: "testorg"
+      GEN_FEATUREBASE_DB_ID: "testdb"
+      GEN_USE_SHARD_TRANSACTIONAL_ENDPOINT: "true"
+      GEN_SOURCE: "custom"
+      GEN_TARGET: "mds"
+      GEN_MDS_ADDRESS: "mds:8080"

dax/errors.go

@@ -0,0 +1,80 @@
+package dax
+
+import (
+	"fmt"
+
+	"github.com/molecula/featurebase/v3/errors"
+)
+
+const (
+	ErrTableIDExists         errors.Code = "TableIDExists"
+	ErrTableKeyExists        errors.Code = "TableKeyExists"
+	ErrTableNameExists       errors.Code = "TableNameExists"
+	ErrTableIDDoesNotExist   errors.Code = "TableIDDoesNotExist"
+	ErrTableKeyDoesNotExist  errors.Code = "TableKeyDoesNotExist"
+	ErrTableNameDoesNotExist errors.Code = "TableNameDoesNotExist"
+
+	ErrFieldExists       errors.Code = "FieldExists"
+	ErrFieldDoesNotExist errors.Code = "FieldDoesNotExist"
+
+	ErrUnimplemented errors.Code = "Unimplemented"
+)
+
+// The following are helper functions for constructing coded errors containing
+// relevant information about the specific error.
+
+func NewErrTableIDDoesNotExist(qtid QualifiedTableID) error {
+	return errors.New(
+		ErrTableIDDoesNotExist,
+		fmt.Sprintf("table ID '%s' does not exist", qtid),
+	)
+}
+
+func NewErrTableKeyDoesNotExist(tkey TableKey) error {
+	return errors.New(
+		ErrTableKeyDoesNotExist,
+		fmt.Sprintf("table key '%s' does not exist", tkey),
+	)
+}
+
+func NewErrTableNameDoesNotExist(tableName TableName) error {
+	return errors.New(
+		ErrTableNameDoesNotExist,
+		fmt.Sprintf("table name '%s' does not exist", tableName),
+	)
+}
+
+func NewErrTableIDExists(qtid QualifiedTableID) error {
+	return errors.New(
+		ErrTableIDExists,
+		fmt.Sprintf("table ID '%s' already exists", qtid),
+	)
+}
+
+func NewErrTableKeyExists(tkey TableKey) error {
+	return errors.New(
+		ErrTableKeyExists,
+		fmt.Sprintf("table key '%s' already exists", tkey),
+	)
+}
+
+func NewErrTableNameExists(tableName TableName) error {
+	return errors.New(
+		ErrTableNameExists,
+		fmt.Sprintf("table name '%s' already exists", tableName),
+	)
+}
+
+func NewErrFieldDoesNotExist(fieldName FieldName) error {
+	return errors.New(
+		ErrFieldDoesNotExist,
+		fmt.Sprintf("field '%s' does not exist", fieldName),
+	)
+}
+
+func NewErrFieldExists(fieldName FieldName) error {
+	return errors.New(
+		ErrFieldExists,
+		fmt.Sprintf("field '%s' already exists", fieldName),
+	)
+}

dax/fieldversion.go

@@ -0,0 +1,32 @@
+package dax
+
+import "fmt"
+
+// FieldVersion is used in a similar way to Shard and Partition in that they all
+// contain a snapshot version. It would have been confusing to use the Field
+// type which already exists, because versioning that would mean something else.
+// This is really snapshot specific (as are Shard and Partition).
+type FieldVersion struct {
+	Name    FieldName `json:"name"`
+	Version int       `json:"version"`
+}
+
+// String returns the FieldVersion (i.e. its Name and Version) as a string.
+func (f FieldVersion) String() string {
+	return fmt.Sprintf("%s.%d", f.Name, f.Version)
+}
+
+// NewFieldVersion returns a FieldVersion with the provided name and version.
+func NewFieldVersion(name FieldName, version int) FieldVersion {
+	return FieldVersion{
+		Name:    name,
+		Version: version,
+	}
+}
+
+// FieldVersions is a sortable slice of FieldVersion.
+type FieldVersions []FieldVersion
+
+func (f FieldVersions) Len() int           { return len(f) }
+func (f FieldVersions) Less(i, j int) bool { return f[i].Name < f[j].Name }
+func (f FieldVersions) Swap(i, j int)      { f[i], f[j] = f[j], f[i] }

dax/http/handler.go

@@ -0,0 +1,219 @@
+package http
+
+import (
+	"context"
+	"net"
+	"net/http"
+	"runtime/debug"
+	"time"
+
+	"github.com/gorilla/mux"
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/mds"
+	mdshttp "github.com/molecula/featurebase/v3/dax/mds/http"
+	"github.com/molecula/featurebase/v3/dax/queryer"
+	queryerhttp "github.com/molecula/featurebase/v3/dax/queryer/http"
+	"github.com/molecula/featurebase/v3/dax/snapshotter"
+	snapshotterhttp "github.com/molecula/featurebase/v3/dax/snapshotter/http"
+	"github.com/molecula/featurebase/v3/dax/writelogger"
+	writeloggerhttp "github.com/molecula/featurebase/v3/dax/writelogger/http"
+	"github.com/molecula/featurebase/v3/errors"
+	"github.com/molecula/featurebase/v3/logger"
+)
+
+// Handler represents an HTTP handler.
+type Handler struct {
+	Handler http.Handler
+
+	bind string
+
+	ln net.Listener
+	// url is used to hold the advertise bind address for printing a log during startup.
+	url string
+
+	closeTimeout time.Duration
+
+	server *http.Server
+
+	mds         *mds.MDS
+	writeLogger *writelogger.WriteLogger
+	snapshotter *snapshotter.Snapshotter
+	queryer     *queryer.Queryer
+
+	computer http.Handler
+
+	logger logger.Logger
+}
+
+// HandlerOption is a functional option type for Handler
+type HandlerOption func(s *Handler) error
+
+func OptHandlerBind(b string) HandlerOption {
+	return func(h *Handler) error {
+		h.bind = b
+		return nil
+	}
+}
+
+func OptHandlerMDS(m *mds.MDS) HandlerOption {
+	return func(h *Handler) error {
+		h.mds = m
+		return nil
+	}
+}
+
+func OptHandlerWriteLogger(w *writelogger.WriteLogger) HandlerOption {
+	return func(h *Handler) error {
+		h.writeLogger = w
+		return nil
+	}
+}
+
+func OptHandlerSnapshotter(s *snapshotter.Snapshotter) HandlerOption {
+	return func(h *Handler) error {
+		h.snapshotter = s
+		return nil
+	}
+}
+
+func OptHandlerQueryer(q *queryer.Queryer) HandlerOption {
+	return func(h *Handler) error {
+		h.queryer = q
+		return nil
+	}
+}
+
+func OptHandlerLogger(l logger.Logger) HandlerOption {
+	return func(h *Handler) error {
+		h.logger = l
+		return nil
+	}
+}
+
+// OptHandlerCloseTimeout controls how long to wait for the http Server to
+// shutdown cleanly before forcibly destroying it. Default is 30 seconds.
+func OptHandlerCloseTimeout(d time.Duration) HandlerOption {
+	return func(h *Handler) error {
+		h.closeTimeout = d
+		return nil
+	}
+}
+
+// OptHandlerListener set the listener that will be used by the HTTP server.
+// Url must be the advertised URL. It will be used to show a log to the user
+// about where the Web UI is. This option is mandatory.
+func OptHandlerListener(ln net.Listener, url string) HandlerOption {
+	return func(h *Handler) error {
+		h.ln = ln
+		h.url = url
+		return nil
+	}
+}
+
+func OptHandlerComputer(handler http.Handler) HandlerOption {
+	return func(h *Handler) error {
+		h.computer = handler
+		return nil
+	}
+}
+
+// NewHandler returns a new instance of Handler with a default logger.
+func NewHandler(opts ...HandlerOption) (*Handler, error) {
+	handler := &Handler{
+		logger:       logger.NopLogger,
+		closeTimeout: time.Second * 30,
+	}
+
+	for _, opt := range opts {
+		err := opt(handler)
+		if err != nil {
+			return nil, errors.Wrap(err, "applying option")
+		}
+	}
+
+	handler.Handler = newRouter(handler)
+
+	handler.server = &http.Server{Handler: handler}
+
+	return handler, nil
+}
+
+func (h *Handler) Serve() error {
+	err := h.server.Serve(h.ln)
+	if err != nil && err.Error() != "http: Server closed" {
+		h.logger.Errorf("HTTP handler terminated with error: %s\n", err)
+		return errors.Wrap(err, "serve http")
+	}
+	return nil
+}
+
+// Close tries to cleanly shutdown the HTTP server, and failing that, after a
+// timeout, calls Server.Close.
+func (h *Handler) Close() error {
+	deadlineCtx, cancelFunc := context.WithDeadline(context.Background(), time.Now().Add(h.closeTimeout))
+	defer cancelFunc()
+	err := h.server.Shutdown(deadlineCtx)
+	if err != nil {
+		err = h.server.Close()
+	}
+	return errors.Wrap(err, "shutdown/close http server")
+}
+
+// newRouter creates a new mux http router.
+func newRouter(handler *Handler) http.Handler {
+	router := mux.NewRouter()
+
+	router.HandleFunc("/health", handler.handleGetHealth).Methods("GET").Name("GetHealth")
+
+	if handler.mds != nil {
+		pre := "/" + dax.ServicePrefixMDS
+		router.PathPrefix(pre).Handler(
+			http.StripPrefix(pre, mdshttp.Handler(handler.mds)))
+	}
+
+	if handler.writeLogger != nil {
+		pre := "/" + dax.ServicePrefixWriteLogger
+		router.PathPrefix(pre).Handler(
+			http.StripPrefix(pre, writeloggerhttp.Handler(handler.writeLogger, handler.logger)))
+	}
+
+	if handler.snapshotter != nil {
+		pre := "/" + dax.ServicePrefixSnapshotter
+		router.PathPrefix(pre).Handler(
+			http.StripPrefix(pre, snapshotterhttp.Handler(handler.snapshotter)))
+	}
+
+	if handler.queryer != nil {
+		pre := "/" + dax.ServicePrefixQueryer
+		router.PathPrefix(pre).Handler(
+			http.StripPrefix(pre, queryerhttp.Handler(handler.queryer)))
+	}
+
+	if handler.computer != nil {
+		pre := "/" + dax.ServicePrefixComputer
+		router.PathPrefix(pre).Handler(
+			http.StripPrefix(pre, handler.computer))
+	}
+
+	var h http.Handler = router
+
+	return h
+}
+
+// ServeHTTP handles an HTTP request.
+func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+	defer func() {
+		if err := recover(); err != nil {
+			w.WriteHeader(http.StatusInternalServerError)
+			stack := debug.Stack()
+			h.logger.Printf("PANIC: %s\n%s", err, stack)
+		}
+	}()
+
+	h.Handler.ServeHTTP(w, r)
+}
+
+// GET /health
+func (h *Handler) handleGetHealth(w http.ResponseWriter, r *http.Request) {
+	w.WriteHeader(http.StatusOK)
+}

dax/inmem/inmem.go

@@ -0,0 +1,2 @@
+// Package inmem contains the in-memory implementation of the dax interfaces.
+package inmem

dax/inmem/versionstore.go

@@ -0,0 +1,430 @@
+package inmem
+
+import (
+	"context"
+	"sort"
+	"sync"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/errors"
+)
+
+// Ensure type implements interface.
+var _ dax.VersionStore = (*VersionStore)(nil)
+
+// VersionStore manages all version info for shard, table keys, and field keys.
+type VersionStore struct {
+	mu sync.RWMutex
+
+	// shards is a map of all shards, by table, by shard number, known to
+	// contain data.
+	shards map[dax.TableQualifierKey]map[dax.TableID]map[dax.ShardNum]dax.Shard
+
+	// tableKeys is a map of all partitions, by table, by partition number,
+	// known to contain key data.
+	tableKeys map[dax.TableQualifierKey]map[dax.TableID]map[dax.PartitionNum]int
+
+	// fieldKeys is a map of all fields, by table, known to contain key data.
+	fieldKeys map[dax.TableQualifierKey]map[dax.TableID]map[dax.FieldName]int
+}
+
+// NewVersionStore returns a new instance of VersionStore with default values.
+func NewVersionStore() *VersionStore {
+	return &VersionStore{
+		shards:    make(map[dax.TableQualifierKey]map[dax.TableID]map[dax.ShardNum]dax.Shard),
+		tableKeys: make(map[dax.TableQualifierKey]map[dax.TableID]map[dax.PartitionNum]int),
+		fieldKeys: make(map[dax.TableQualifierKey]map[dax.TableID]map[dax.FieldName]int),
+	}
+}
+
+// AddTable adds a table to be managed by VersionStore.
+func (s *VersionStore) AddTable(ctx context.Context, qtid dax.QualifiedTableID) error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	// This check is clunky; three maps contain the table, but we only check for
+	// existence in one of them. It also seems weird to check all three, because
+	// if we get in a state where one of the maps doesn't contain a table that
+	// the other maps do contain, the state of the data is in question.
+	if _, found := s.shards[qtid.TableQualifier.Key()][qtid.ID]; found {
+		return dax.NewErrTableIDExists(qtid)
+	}
+
+	// Initialize the maps in case VersionStore wasn't created with NewVersionStore().
+	if s.shards == nil {
+		s.shards = make(map[dax.TableQualifierKey]map[dax.TableID]map[dax.ShardNum]dax.Shard)
+	}
+	if s.tableKeys == nil {
+		s.tableKeys = make(map[dax.TableQualifierKey]map[dax.TableID]map[dax.PartitionNum]int)
+	}
+	if s.fieldKeys == nil {
+		s.fieldKeys = make(map[dax.TableQualifierKey]map[dax.TableID]map[dax.FieldName]int)
+	}
+
+	// shards.
+	if _, ok := s.shards[qtid.TableQualifier.Key()]; !ok {
+		s.shards[qtid.TableQualifier.Key()] = make(map[dax.TableID]map[dax.ShardNum]dax.Shard, 0)
+	}
+	if _, ok := s.shards[qtid.TableQualifier.Key()][qtid.ID]; !ok {
+		s.shards[qtid.TableQualifier.Key()][qtid.ID] = make(map[dax.ShardNum]dax.Shard, 0)
+	}
+
+	// tableKeys.
+	if _, ok := s.tableKeys[qtid.TableQualifier.Key()]; !ok {
+		s.tableKeys[qtid.TableQualifier.Key()] = make(map[dax.TableID]map[dax.PartitionNum]int, 0)
+	}
+	if _, ok := s.tableKeys[qtid.TableQualifier.Key()][qtid.ID]; !ok {
+		s.tableKeys[qtid.TableQualifier.Key()][qtid.ID] = make(map[dax.PartitionNum]int, 0)
+	}
+
+	// fieldKeys.
+	if _, ok := s.fieldKeys[qtid.TableQualifier.Key()]; !ok {
+		s.fieldKeys[qtid.TableQualifier.Key()] = make(map[dax.TableID]map[dax.FieldName]int, 0)
+	}
+	if _, ok := s.fieldKeys[qtid.TableQualifier.Key()][qtid.ID]; !ok {
+		s.fieldKeys[qtid.TableQualifier.Key()][qtid.ID] = make(map[dax.FieldName]int, 0)
+	}
+
+	return nil
+}
+
+// RemoveTable removes the given table. An error will be returned if the table
+// does not exist.
+func (s *VersionStore) RemoveTable(ctx context.Context, qtid dax.QualifiedTableID) (dax.Shards, dax.Partitions, error) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	var foundTable bool
+	var shards dax.Shards
+	var partitions dax.Partitions
+	var err error
+
+	// Remove shards for table.
+	if s.shards != nil {
+		if _, ok := s.shards[qtid.TableQualifier.Key()][qtid.ID]; ok {
+			foundTable = true
+
+			// Get the shards to return before deleting from map.
+			shards, _, err = s.shardSlice(qtid)
+			if err != nil {
+				return nil, nil, errors.Wrapf(err, "getting shard slice: %s", qtid)
+			}
+
+			// Remove the shards.
+			delete(s.shards[qtid.TableQualifier.Key()], qtid.ID)
+		}
+	}
+
+	// Remove tableKeys for table.
+	if s.tableKeys != nil {
+		if _, ok := s.tableKeys[qtid.TableQualifier.Key()][qtid.ID]; ok {
+			foundTable = true
+
+			// Get the partitions to return before deleting from map.
+			partitions, _, err = s.partitionSlice(qtid)
+			if err != nil {
+				return nil, nil, errors.Wrapf(err, "getting partition slice: %s", qtid)
+			}
+
+			// Remove the tableKeys.
+			delete(s.tableKeys[qtid.TableQualifier.Key()], qtid.ID)
+		}
+	}
+
+	// Remove fieldKeys for table.
+	if s.fieldKeys != nil {
+		if _, ok := s.fieldKeys[qtid.TableQualifier.Key()][qtid.ID]; ok {
+			foundTable = true
+
+			// Remove the fieldKeys.
+			delete(s.fieldKeys[qtid.TableQualifier.Key()], qtid.ID)
+		}
+	}
+
+	if !foundTable {
+		return nil, nil, dax.NewErrTableIDDoesNotExist(qtid)
+	}
+
+	return shards, partitions, nil
+}
+
+// AddShards adds new shards to be managed by VersionStore. It returns the
+// number of shards added or an error.
+func (s *VersionStore) AddShards(ctx context.Context, qtid dax.QualifiedTableID, shards ...dax.Shard) error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	sh, ok := s.shards[qtid.TableQualifier.Key()][qtid.ID]
+	if !ok {
+		return dax.NewErrTableIDDoesNotExist(qtid)
+	}
+
+	var n int
+
+	for _, shard := range shards {
+		if _, ok := sh[shard.Num]; !ok {
+			n++ // TODO: this isn't considering a shard that exists, but the version changes.
+		}
+		sh[shard.Num] = shard
+	}
+
+	return nil
+}
+
+// Shards returns the list of shards available for the give table. It returns
+// false if the table does not exist.
+func (s *VersionStore) Shards(ctx context.Context, qtid dax.QualifiedTableID) (dax.Shards, bool, error) {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+
+	return s.shardSlice(qtid)
+}
+
+// shardSlice is an unprotected version of Shards().
+func (s *VersionStore) shardSlice(qtid dax.QualifiedTableID) (dax.Shards, bool, error) {
+	if s.shards == nil {
+		return nil, false, nil
+	}
+
+	if shardNumMap, ok := s.shards[qtid.TableQualifier.Key()][qtid.ID]; ok {
+		rtn := make(dax.Shards, 0, len(shardNumMap))
+		for _, shard := range shardNumMap {
+			rtn = append(rtn, shard)
+		}
+		sort.Sort(rtn)
+		return rtn, true, nil
+	}
+
+	return nil, false, nil
+}
+
+// ShardVersion return the current version for the given table/shardNum.
+// If a version is not being tracked, it returns a bool value of false.
+func (s *VersionStore) ShardVersion(ctx context.Context, qtid dax.QualifiedTableID, shardNum dax.ShardNum) (int, bool, error) {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+
+	t, ok := s.shards[qtid.TableQualifier.Key()][qtid.ID]
+	if !ok {
+		return -1, false, nil
+	}
+
+	v, ok := t[shardNum]
+	if !ok {
+		return -1, false, nil
+	}
+	return v.Version, true, nil
+}
+
+func (s *VersionStore) ShardTables(ctx context.Context, qual dax.TableQualifier) (dax.TableIDs, error) {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+
+	qual.Key()
+
+	tableIDs := make(dax.TableIDs, 0, len(s.shards[qual.Key()]))
+
+	for tableID := range s.shards[qual.Key()] {
+		tableIDs = append(tableIDs, tableID)
+	}
+
+	return tableIDs, nil
+}
+
+// AddPartitions adds new partitions to be managed by VersionStore. It returns
+// the number of partitions added or an error.
+func (s *VersionStore) AddPartitions(ctx context.Context, qtid dax.QualifiedTableID, partitions ...dax.Partition) error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	tk, ok := s.tableKeys[qtid.TableQualifier.Key()][qtid.ID]
+	if !ok {
+		return dax.NewErrTableIDDoesNotExist(qtid)
+	}
+
+	for _, partition := range partitions {
+		tk[partition.Num] = partition.Version
+	}
+
+	return nil
+}
+
+// Partitions returns the list of partitions available for the give table. It
+// returns false if the table does not exist.
+func (s *VersionStore) Partitions(ctx context.Context, qtid dax.QualifiedTableID) (dax.Partitions, bool, error) {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+
+	return s.partitionSlice(qtid)
+}
+
+// partitionSlice is an unprotected version of Partitions().
+func (s *VersionStore) partitionSlice(qtid dax.QualifiedTableID) (dax.Partitions, bool, error) {
+	if s.tableKeys == nil {
+		return nil, false, nil
+	}
+
+	if partitionNumMap, ok := s.tableKeys[qtid.TableQualifier.Key()][qtid.ID]; ok {
+		rtn := make(dax.Partitions, 0, len(partitionNumMap))
+		for partitionNum, version := range partitionNumMap {
+			rtn = append(rtn, dax.NewPartition(partitionNum, version))
+		}
+		sort.Sort(rtn)
+		return rtn, true, nil
+	}
+
+	return nil, false, nil
+}
+
+// PartitionVersion return the current version for the given table/partitionNum.
+// If a version is not being tracked, it returns a bool value of false.
+func (s *VersionStore) PartitionVersion(ctx context.Context, qtid dax.QualifiedTableID, partitionNum dax.PartitionNum) (int, bool, error) {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+
+	t, ok := s.tableKeys[qtid.TableQualifier.Key()][qtid.ID]
+	if !ok {
+		return -1, false, nil
+	}
+
+	v, ok := t[partitionNum]
+	if !ok {
+		return -1, false, nil
+	}
+	return v, true, nil
+}
+
+func (s *VersionStore) PartitionTables(ctx context.Context, qual dax.TableQualifier) (dax.TableIDs, error) {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+
+	tableIDs := make(dax.TableIDs, 0, len(s.tableKeys[qual.Key()]))
+
+	for tableName := range s.tableKeys[qual.Key()] {
+		tableIDs = append(tableIDs, tableName)
+	}
+
+	return tableIDs, nil
+}
+
+// AddFields adds new fields to be managed by VersionStore. It returns the
+// number of fields added or an error.
+func (s *VersionStore) AddFields(ctx context.Context, qtid dax.QualifiedTableID, fields ...dax.FieldVersion) error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	fk, ok := s.fieldKeys[qtid.TableQualifier.Key()][qtid.ID]
+	if !ok {
+		return dax.NewErrTableIDDoesNotExist(qtid)
+	}
+
+	for _, field := range fields {
+		fk[field.Name] = field.Version
+	}
+
+	return nil
+}
+
+// Fields returns the list of fields available for the give table. It returns
+// false if the table does not exist.
+func (s *VersionStore) Fields(ctx context.Context, qtid dax.QualifiedTableID) (dax.FieldVersions, bool, error) {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+
+	return s.fieldSlice(qtid)
+}
+
+// fieldSlice is an unprotected version of Fields().
+func (s *VersionStore) fieldSlice(qtid dax.QualifiedTableID) (dax.FieldVersions, bool, error) {
+	if s.fieldKeys == nil {
+		return nil, false, nil
+	}
+
+	if fieldNameMap, ok := s.fieldKeys[qtid.TableQualifier.Key()][qtid.ID]; ok {
+		rtn := make(dax.FieldVersions, 0, len(fieldNameMap))
+		for fieldName, version := range fieldNameMap {
+			rtn = append(rtn, dax.NewFieldVersion(fieldName, version))
+		}
+		sort.Sort(rtn)
+		return rtn, true, nil
+	}
+
+	return nil, false, nil
+}
+
+// FieldVersion return the current version for the given table/field.
+// If a version is not being tracked, it returns a bool value of false.
+func (s *VersionStore) FieldVersion(ctx context.Context, qtid dax.QualifiedTableID, field dax.FieldName) (int, bool, error) {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+
+	t, ok := s.fieldKeys[qtid.TableQualifier.Key()][qtid.ID]
+	if !ok {
+		return -1, false, nil
+	}
+
+	v, ok := t[field]
+	if !ok {
+		return -1, false, nil
+	}
+	return v, true, nil
+}
+
+func (s *VersionStore) FieldTables(ctx context.Context, qual dax.TableQualifier) (dax.TableIDs, error) {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+
+	tableIDs := make(dax.TableIDs, 0, len(s.fieldKeys[qual.Key()]))
+
+	for tableID := range s.fieldKeys[qual.Key()] {
+		tableIDs = append(tableIDs, tableID)
+	}
+
+	return tableIDs, nil
+}
+
+// Copy returns a new copy of VersionStore.
+func (s *VersionStore) Copy(ctx context.Context) (dax.VersionStore, error) {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+
+	new := NewVersionStore()
+
+	// shards.
+	for qkey, tableIDs := range s.shards {
+		for tableID, shards := range tableIDs {
+			qual := dax.NewTableQualifier(qkey.OrganizationID(), qkey.DatabaseID())
+			qtid := dax.NewQualifiedTableID(qual, tableID)
+			_ = new.AddTable(ctx, qtid)
+			for shardNum, shard := range shards {
+				new.shards[qual.Key()][tableID][shardNum] = shard
+			}
+		}
+	}
+
+	// tableKeys.
+	for qkey, tableIDs := range s.tableKeys {
+		for tableID, partitions := range tableIDs {
+			qual := dax.NewTableQualifier(qkey.OrganizationID(), qkey.DatabaseID())
+			qtid := dax.NewQualifiedTableID(qual, tableID)
+			_ = new.AddTable(ctx, qtid)
+			for partitionNum, version := range partitions {
+				new.tableKeys[qual.Key()][tableID][partitionNum] = version
+			}
+		}
+	}
+
+	// fieldKeys.
+	for qkey, tableIDs := range s.fieldKeys {
+		for tableID, fields := range tableIDs {
+			qual := dax.NewTableQualifier(qkey.OrganizationID(), qkey.DatabaseID())
+			qtid := dax.NewQualifiedTableID(qual, tableID)
+			_ = new.AddTable(ctx, qtid)
+			for field, version := range fields {
+				new.fieldKeys[qual.Key()][tableID][field] = version
+			}
+		}
+	}
+
+	return new, nil
+}

dax/inmem/versionstore_test.go

@@ -0,0 +1,166 @@
+package inmem_test
+
+import (
+	"context"
+	"testing"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/inmem"
+	"github.com/molecula/featurebase/v3/errors"
+	"github.com/stretchr/testify/assert"
+)
+
+func TestVersionStore(t *testing.T) {
+	orgID := dax.OrganizationID("acme")
+	dbID := dax.DatabaseID("db1")
+
+	tableID := dax.TableID("0000000000000001")
+
+	qual := dax.NewTableQualifier(orgID, dbID)
+	qtid := dax.NewQualifiedTableID(qual, tableID)
+
+	invalidQtid := dax.NewQualifiedTableID(qual, dax.TableID("0000000000000000"))
+
+	ctx := context.Background()
+
+	// Ensure that when using a Schemar not initiated with NewSchemar, the error
+	// handling works as expected.
+	t.Run("EmptyVersionStore", func(t *testing.T) {
+		s := inmem.VersionStore{}
+
+		t.Run("GetShardsInvalid", func(t *testing.T) {
+			sh, ok, err := s.Shards(ctx, invalidQtid)
+			assert.NoError(t, err)
+			assert.False(t, ok)
+			assert.Nil(t, sh)
+		})
+
+		// Add new table.
+		assert.NoError(t, s.AddTable(ctx, qtid))
+	})
+
+	t.Run("NewVersionStore", func(t *testing.T) {
+		s := inmem.NewVersionStore()
+
+		// Add new table.
+		assert.NoError(t, s.AddTable(ctx, qtid))
+
+		t.Run("AddTableAgain", func(t *testing.T) {
+			err := s.AddTable(ctx, qtid)
+			if assert.Error(t, err) {
+				assert.True(t, errors.Is(err, dax.ErrTableIDExists))
+			}
+		})
+
+		t.Run("AddShards", func(t *testing.T) {
+			err := s.AddShards(ctx, invalidQtid, dax.NewShard(1, 0))
+			if assert.Error(t, err) {
+				assert.True(t, errors.Is(err, dax.ErrTableIDDoesNotExist))
+			}
+
+			{
+				_, ok, err := s.Shards(ctx, invalidQtid)
+				assert.NoError(t, err)
+				assert.False(t, ok)
+			}
+
+			// Shards is empty if no shards have been added.
+			{
+				sh, ok, err := s.Shards(ctx, qtid)
+				assert.NoError(t, err)
+				assert.True(t, ok)
+				assert.Equal(t, sh, dax.Shards{})
+			}
+
+			// Add the first set of shards (with a duplicate (8)).
+			{
+				err := s.AddShards(ctx, qtid,
+					dax.NewShard(8, 0),
+					dax.NewShard(9, 0),
+					dax.NewShard(8, 0),
+					dax.NewShard(10, 0),
+				)
+				assert.NoError(t, err)
+			}
+
+			{
+				sh, ok, err := s.Shards(ctx, qtid)
+				assert.NoError(t, err)
+				assert.True(t, ok)
+				assert.Equal(t, dax.Shards{
+					dax.NewShard(8, 0),
+					dax.NewShard(9, 0),
+					dax.NewShard(10, 0),
+				}, sh)
+			}
+
+			// Add another set of shards (with one duplicate (11) and one
+			// existing (10)).
+			{
+				err := s.AddShards(ctx, qtid,
+					dax.NewShard(10, 0),
+					dax.NewShard(11, 0),
+					dax.NewShard(12, 0),
+					dax.NewShard(11, 0),
+				)
+				assert.NoError(t, err)
+			}
+
+			{
+				sh, ok, err := s.Shards(ctx, qtid)
+				assert.NoError(t, err)
+				assert.True(t, ok)
+				assert.Equal(t, dax.Shards{
+					dax.NewShard(8, 0),
+					dax.NewShard(9, 0),
+					dax.NewShard(10, 0),
+					dax.NewShard(11, 0),
+					dax.NewShard(12, 0),
+				}, sh)
+			}
+		})
+
+		t.Run("RemoveTable", func(t *testing.T) {
+			shards, partitions, err := s.RemoveTable(ctx, qtid)
+			assert.NoError(t, err)
+			assert.Equal(t, dax.Partitions{}, partitions)
+			assert.Equal(t, dax.Shards{
+				dax.NewShard(8, 0),
+				dax.NewShard(9, 0),
+				dax.NewShard(10, 0),
+				dax.NewShard(11, 0),
+				dax.NewShard(12, 0),
+			}, shards)
+
+			// Make sure the table was removed.
+			shards, ok, err := s.Shards(ctx, qtid)
+			assert.NoError(t, err)
+			assert.False(t, ok)
+			assert.Nil(t, shards)
+		})
+	})
+
+	t.Run("ErrorConditions", func(t *testing.T) {
+		t.Run("JustSchemar", func(t *testing.T) {
+			s := inmem.VersionStore{}
+
+			shards, partitions, err := s.RemoveTable(ctx, qtid)
+			assert.Nil(t, shards)
+			assert.Nil(t, partitions)
+			if assert.Error(t, err) {
+				assert.True(t, errors.Is(err, dax.ErrTableIDDoesNotExist))
+			}
+		})
+
+		t.Run("NewSchemar", func(t *testing.T) {
+			s := inmem.NewVersionStore()
+
+			shards, partitions, err := s.RemoveTable(ctx, qtid)
+			assert.Nil(t, shards)
+			assert.Nil(t, partitions)
+			if assert.Error(t, err) {
+				assert.True(t, errors.Is(err, dax.ErrTableIDDoesNotExist))
+			}
+		})
+	})
+}

dax/mds/api/openapi.yaml

@@ -0,0 +1,548 @@
+openapi: 3.0.3
+
+info:
+  title: MDS
+  description: Metadata Services.
+  version: 0.0.0
+
+paths:
+  /mds/health:
+    get:
+      summary: Health check endpoint.
+      description: Provides an endpoint to check the overall health of the MDS service.
+      operationId: GetHealth
+      responses:
+        200:
+          description: Service is healthy.
+
+  /mds/create-table:
+    post:
+      summary: Create a table.
+      description: Create a table based on the provided schema.
+      operationId: PostCreateTable
+      requestBody:
+        content:
+          application/json:
+            examples:
+              table:
+                $ref: '#/components/examples/Table'
+            schema:
+              $ref: '#/components/schemas/Table'
+      responses:
+        200:
+          $ref: '#/components/responses/CreateTableResponse'
+
+  /mds/drop-table:
+    post:
+      summary: Drop a table.
+      description: Drop a table based on the provided table name.
+      operationId: PostDropTable
+      requestBody:
+        content:
+          application/json:
+            example:
+              name: tbl
+            schema:
+              type: object
+              properties:
+                name:
+                  type: string
+      responses:
+        200:
+          description: Table was dropped.
+
+  /mds/create-field:
+    post:
+      summary: Create a field.
+      description: Create a field based on the provided table and schema.
+      operationId: PostCreateField
+      requestBody:
+        content:
+          application/json:
+            example:
+              table: tbl
+              field:
+                name: a_string
+                type: string
+                options:
+                  cacheType: ranked
+                  cacheSize: 50000
+            schema:
+              $ref: '#/components/schemas/TableField'
+      responses:
+        200:
+          description: Field was created.
+
+  /mds/drop-field:
+    post:
+      summary: Drop a field.
+      description: Drop a field based on the provided table and field name.
+      operationId: PostDropField
+      requestBody:
+        content:
+          application/json:
+            example:
+              table: tbl
+              field: fld
+            schema:
+              type: object
+              properties:
+                table:
+                  type: string
+                field:
+                  type: string
+      responses:
+        200:
+          description: Field was dropped.
+
+  /mds/table:
+    post:
+      summary: Get a table.
+      description: Get a table based on the provided table name.
+      operationId: PostTable
+      requestBody:
+        content:
+          application/json:
+            example:
+              name: tbl
+            schema:
+              type: object
+              properties:
+                name:
+                  type: string
+      responses:
+        200:
+          $ref: '#/components/responses/Table'
+
+  /mds/tables:
+    post:
+      summary: Get a list of table.
+      description: Get a list of tables. If a filter is provided, only those tables will be included in the result.
+      operationId: PostTables
+      requestBody:
+        content:
+          application/json:
+            example:
+              names:
+              - tbl1
+              - tbl2
+            schema:
+              type: object
+              properties:
+                names:
+                  type: array
+                  items:
+                    type: string
+      responses:
+        200:
+          $ref: '#/components/responses/Tables'
+
+  /mds/ingest-partition:
+    post:
+      summary: Request to ingest partition data.
+      description: Request to ingest (write) partition data. The address of the compute node responsible is returned.
+      operationId: PostIngestPartition
+      requestBody:
+        content:
+          application/json:
+            example:
+              table: tbl
+              partition: 7
+            schema:
+              type: object
+              properties:
+                table:
+                  type: string
+                partition:
+                  type: integer
+                  format: int32
+      responses:
+        200:
+          $ref: '#/components/responses/Address'
+
+  /mds/ingest-shard:
+    post:
+      summary: Request to ingest shard data.
+      description: Request to ingest (write) shard data. The address of the compute node responsible is returned.
+      operationId: PostIngestShard
+      requestBody:
+        content:
+          application/json:
+            example:
+              table: tbl
+              shard: 12
+            schema:
+              type: object
+              properties:
+                table:
+                  type: string
+                shard:
+                  type: integer
+                  format: int64
+      responses:
+        200:
+          $ref: '#/components/responses/Address'
+
+  /mds/snapshot/shard-data:
+    post:
+      summary: Request to snapshot shard data.
+      description: Request to snapshot shard data.
+      operationId: PostSnapshotShardData
+      requestBody:
+        content:
+          application/json:
+            example:
+              table: tbl
+              shard: 12
+            schema:
+              type: object
+              properties:
+                table:
+                  type: string
+                shard:
+                  type: integer
+                  format: int64
+      responses:
+        200:
+          description: Shard snapshot was successful.
+
+  /mds/snapshot/table-keys:
+    post:
+      summary: Request to snapshot table keys.
+      description: Request to snapshot table keys.
+      operationId: PostSnapshotTableKeys
+      requestBody:
+        content:
+          application/json:
+            example:
+              table: tbl
+              partition: 7
+            schema:
+              type: object
+              properties:
+                table:
+                  type: string
+                partition:
+                  type: integer
+                  format: int32
+      responses:
+        200:
+          description: Table keys snapshot was successful.
+
+  /mds/snapshot/field-keys:
+    post:
+      summary: Request to snapshot field keys.
+      description: Request to snapshot field keys.
+      operationId: PostSnapshotFieldKeys
+      requestBody:
+        content:
+          application/json:
+            example:
+              table: tbl
+              field: fld
+            schema:
+              type: object
+              properties:
+                table:
+                  type: string
+                field:
+                  type: string
+      responses:
+        200:
+          description: Field keys snapshot was successful.
+
+  /mds/register-node:
+    post:
+      summary: Register node.
+      description: Register a node with MDS.
+      operationId: PostRegisterNode
+      requestBody:
+        content:
+          application/json:
+            example:
+              address: 10.0.0.1:8000
+              roleTypes:
+              - compute
+              - translate
+            schema:
+              type: object
+              properties:
+                address:
+                  type: string
+                roleTypes:
+                  type: array
+                  items:
+                    types: string
+      responses:
+        200:
+          description: Node registration was successful.
+
+  /mds/deregister-nodes:
+    post:
+      summary: Deregister nodes.
+      description: Deregister nodes with MDS.
+      operationId: PostDeregisterNodes
+      requestBody:
+        content:
+          application/json:
+            example:
+              address: 10.0.0.1:8000
+            schema:
+              type: object
+              properties:
+                address:
+                  type: string
+      responses:
+        200:
+          description: Node deregistration was successful.
+
+  /mds/compute-nodes:
+    post:
+      summary: Get compute nodes.
+      description: Get the compute nodes responsible for the given shards.
+      operationId: PostComputeNodes
+      requestBody:
+        content:
+          application/json:
+            example:
+              table: tbl
+              shards:
+              - 10
+              - 11
+              - 12
+              isWrite: false
+            schema:
+              type: object
+              properties:
+                table:
+                  type: string
+                shards:
+                  type: array
+                  items:
+                    type: integer
+                    format: int64
+                isWrite:
+                  type: boolean  
+      responses:
+        200:
+          $ref: '#/components/responses/ComputeNodes'
+
+  /mds/translate-nodes:
+    post:
+      summary: Get translate nodes.
+      description: Get the translate nodes responsible for the given partitions.
+      operationId: PostTranslateNodes
+      requestBody:
+        content:
+          application/json:
+            example:
+              table: tbl
+              partitions:
+              - 6
+              - 7
+              isWrite: false
+            schema:
+              type: object
+              properties:
+                table:
+                  type: string
+                partitions:
+                  type: array
+                  items:
+                    type: integer
+                    format: int32
+                isWrite:
+                  type: boolean  
+      responses:
+        200:
+          $ref: '#/components/responses/TranslateNodes'
+
+components:
+  responses:
+    CreateTableResponse:
+      description: Placeholder response.
+      content:
+        application/json:
+          schema:
+            type: object
+
+    Address:
+      description: Single node address.
+      content:
+        application/json:
+          schema:
+            type: object
+            properties:
+              address:
+                type: string
+
+    Table:
+      description: Table response.
+      content:
+        application/json:
+          schema:
+            $ref: '#/components/schemas/Table'
+
+    Tables:
+      description: Tables response.
+      content:
+        application/json:
+          schema:
+            type: array
+            items:
+              $ref: '#/components/schemas/Table'
+
+    ComputeNodes:
+      description: Compute nodes response.
+      content:
+        application/json:
+          schema:
+            type: array
+            items:
+              $ref: '#/components/schemas/ComputeNode'
+
+    TranslateNodes:
+      description: Translate nodes response.
+      content:
+        application/json:
+          schema:
+            type: array
+            items:
+              $ref: '#/components/schemas/TranslateNode'
+
+  schemas:
+    ComputeNode:
+      type: object
+      properties:
+        address:
+          type: string
+        table:
+          type: string
+        shards:
+          type: array
+          items:
+            type: integer
+            format: int64
+
+    TranslateNode:
+      type: object
+      properties:
+        address:
+          type: string
+        table:
+          type: string
+        partitions:
+          type: array
+          items:
+            type: integer
+            format: int32
+
+    Table:
+      type: object
+      properties:
+        name:
+          type: string
+        fields:
+          type: array
+          items:
+            $ref: '#/components/schemas/Field'
+        partitionN:
+          type: integer
+          format: int32
+
+    TableField:
+      type: object
+      properties:
+        table:
+          type: string
+        field:
+          $ref: '#/components/schemas/Field'
+
+    Field:
+      type: object
+      properties:
+        name:
+          type: string
+        type:
+          type: string
+          enum:
+          - bool
+          - decimal
+          - id
+          - idset
+          - int
+          - string
+          - stringset
+          - timestamp
+        options:
+          type: object
+          properties:
+            min:
+              type: integer
+              format: int64
+            max:
+              type: integer
+              format: int64
+            scale:
+              type: integer
+              format: int64
+              minimum: 0
+            noStandardView:
+              type: boolean
+            cacheType:
+              type: string
+            cacheSize:
+              type: integer
+              format: int32
+            timeUnit:
+              type: string
+            epoch:
+              type: string
+              format: date-time
+            timeQuantum:
+              type: string
+            ttl:
+              type: string
+            foreignIndex:
+              type: string
+
+  examples:
+    Table:
+      name: tbl
+      fields:
+      - name: _id
+        type: string
+      - name: a_bool
+        type: bool
+      - name: an_id
+        type: id
+        options:
+          cacheType: ranked
+          cacheSize: 50000
+      - name: an_id_set
+        type: idset
+        options:
+          cacheType: ranked
+          cacheSize: 50000
+      - name: a_string
+        type: string
+        options:
+          cacheType: ranked
+          cacheSize: 50000
+      - name: a_string_set
+        type: stringset
+        options:
+          cacheType: ranked
+          cacheSize: 50000
+      - name: an_int
+        type: int
+        options:
+          min: -100
+          max: 500
+      - name: a_decimal
+        type: decimal
+        options:
+          min: -10.24
+          max: 50.75
+          scale: 2
+      partitionN: 16

dax/mds/client/client.go

@@ -0,0 +1,478 @@
+// Package client is an HTTP client for MDS.
+package client
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"log"
+	"net/http"
+
+	fb "github.com/molecula/featurebase/v3"
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/mds/controller"
+	mdshttp "github.com/molecula/featurebase/v3/dax/mds/http"
+	"github.com/molecula/featurebase/v3/errors"
+)
+
+const (
+	defaultScheme = "http"
+	defaultPath   = "/mds"
+)
+
+// Ensure type implements interface.
+var _ fb.MDS = (*Client)(nil)
+
+// Client is an HTTP client that operates on the MDS endpoints exposed by the
+// main MDS service.
+type Client struct {
+	address dax.Address
+}
+
+// New returns a new instance of Client.
+func New(address dax.Address) *Client {
+	return &Client{
+		address: address,
+	}
+}
+
+// Health returns true if the client address returns status OK at its /health
+// endpoint.
+func (c *Client) Health() bool {
+	url := fmt.Sprintf("%s%s/health", c.address.WithScheme(defaultScheme), defaultPath)
+
+	if resp, err := http.Get(url); err != nil {
+		return false
+	} else if resp.StatusCode != http.StatusOK {
+		return false
+	}
+
+	return true
+}
+
+func (c *Client) Table(ctx context.Context, qtid dax.QualifiedTableID) (*dax.QualifiedTable, error) {
+	url := fmt.Sprintf("%s%s/table", c.address.WithScheme(defaultScheme), defaultPath)
+
+	// Encode the request.
+	postBody, err := json.Marshal(qtid)
+	if err != nil {
+		return nil, errors.Wrap(err, "marshalling post request")
+	}
+	responseBody := bytes.NewBuffer(postBody)
+
+	// Post the request.
+	log.Printf("POST table request: url: %s", url)
+	resp, err := http.Post(url, "application/json", responseBody)
+	if err != nil {
+		return nil, errors.Wrap(err, "posting table request")
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return nil, errors.Errorf("status code: %d: %s", resp.StatusCode, b)
+	}
+
+	var qtable *dax.QualifiedTable
+	if err := json.NewDecoder(resp.Body).Decode(&qtable); err != nil {
+		return nil, errors.Wrap(err, "reading response body")
+	}
+
+	return qtable, nil
+}
+
+func (c *Client) TableID(ctx context.Context, qual dax.TableQualifier, name dax.TableName) (dax.QualifiedTableID, error) {
+	url := fmt.Sprintf("%s%s/table-id", c.address.WithScheme(defaultScheme), defaultPath)
+
+	dflt := dax.QualifiedTableID{}
+
+	req := dax.QualifiedTableID{
+		TableQualifier: qual,
+		Name:           name,
+	}
+
+	// Encode the request.
+	postBody, err := json.Marshal(req)
+	if err != nil {
+		return dflt, errors.Wrap(err, "marshalling post request")
+	}
+	requestBody := bytes.NewBuffer(postBody)
+
+	// Post the request.
+	resp, err := http.Post(url, "application/json", requestBody)
+	if err != nil {
+		return dflt, errors.Wrap(err, "posting table-id request")
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return dflt, errors.Errorf("status code: %d: %s", resp.StatusCode, b)
+	}
+
+	var qtid dax.QualifiedTableID
+	if err := json.NewDecoder(resp.Body).Decode(&qtid); err != nil {
+		return dflt, errors.Wrap(err, "reading response body")
+	}
+
+	return qtid, nil
+}
+
+func (c *Client) Tables(ctx context.Context, qual dax.TableQualifier, ids ...dax.TableID) ([]*dax.QualifiedTable, error) {
+	url := fmt.Sprintf("%s%s/tables", c.address.WithScheme(defaultScheme), defaultPath)
+
+	req := mdshttp.TablesRequest{
+		OrganizationID: qual.OrganizationID,
+		DatabaseID:     qual.DatabaseID,
+		TableIDs:       ids,
+	}
+
+	// Encode the request.
+	postBody, err := json.Marshal(req)
+	if err != nil {
+		return nil, errors.Wrap(err, "marshalling post request")
+	}
+	responseBody := bytes.NewBuffer(postBody)
+
+	// Post the request.
+	resp, err := http.Post(url, "application/json", responseBody)
+	if err != nil {
+		return nil, errors.Wrap(err, "posting tables request")
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return nil, errors.Errorf("status code: %d: %s", resp.StatusCode, b)
+	}
+
+	var qtables []*dax.QualifiedTable
+	if err := json.NewDecoder(resp.Body).Decode(&qtables); err != nil {
+		return nil, errors.Wrap(err, "reading response body")
+	}
+
+	return qtables, nil
+}
+
+func (c *Client) CreateTable(ctx context.Context, qtbl *dax.QualifiedTable) error {
+	url := fmt.Sprintf("%s%s/create-table", c.address.WithScheme(defaultScheme), defaultPath)
+
+	// Encode the request.
+	postBody, err := json.Marshal(qtbl)
+	if err != nil {
+		return errors.Wrap(err, "marshalling post request")
+	}
+	responseBody := bytes.NewBuffer(postBody)
+
+	// Post the request.
+	resp, err := http.Post(url, "application/json", responseBody)
+	if err != nil {
+		return errors.Wrap(err, "posting create table request")
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return errors.Errorf("status code: %d: %s", resp.StatusCode, b)
+	}
+
+	return nil
+}
+
+func (c *Client) DropTable(ctx context.Context, qtid dax.QualifiedTableID) error {
+	url := fmt.Sprintf("%s%s/drop-table", c.address.WithScheme(defaultScheme), defaultPath)
+
+	// Encode the request.
+	postBody, err := json.Marshal(qtid)
+	if err != nil {
+		return errors.Wrap(err, "marshalling post request")
+	}
+	responseBody := bytes.NewBuffer(postBody)
+
+	// Post the request.
+	resp, err := http.Post(url, "application/json", responseBody)
+	if err != nil {
+		return errors.Wrap(err, "posting drop table request")
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return errors.Errorf("status code: %d: %s", resp.StatusCode, b)
+	}
+
+	return nil
+}
+
+func (c *Client) CreateField(ctx context.Context, qtid dax.QualifiedTableID, fld *dax.Field) error {
+	url := fmt.Sprintf("%s%s/create-field", c.address.WithScheme(defaultScheme), defaultPath)
+
+	req := mdshttp.CreateFieldRequest{
+		TableKey: qtid.Key(),
+		Field:    fld,
+	}
+	// Encode the request.
+	postBody, err := json.Marshal(req)
+	if err != nil {
+		return errors.Wrap(err, "marshalling post request")
+	}
+	responseBody := bytes.NewBuffer(postBody)
+
+	// Post the request.
+	resp, err := http.Post(url, "application/json", responseBody)
+	if err != nil {
+		return errors.Wrap(err, "posting create field request")
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return errors.Errorf("status code: %d: %s", resp.StatusCode, b)
+	}
+
+	return nil
+}
+
+func (c *Client) DropField(ctx context.Context, qtid dax.QualifiedTableID, fldName dax.FieldName) error {
+	url := fmt.Sprintf("%s%s/drop-field", c.address.WithScheme(defaultScheme), defaultPath)
+
+	// Encode the request.
+	req := mdshttp.DropFieldRequest{
+		Table: qtid,
+		Field: fldName,
+	}
+
+	postBody, err := json.Marshal(req)
+	if err != nil {
+		return errors.Wrap(err, "marshalling post request")
+	}
+	responseBody := bytes.NewBuffer(postBody)
+
+	// Post the request.
+	resp, err := http.Post(url, "application/json", responseBody)
+	if err != nil {
+		return errors.Wrap(err, "posting drop field request")
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return errors.Errorf("status code: %d: %s", resp.StatusCode, b)
+	}
+
+	return nil
+}
+
+func (c *Client) IngestShard(ctx context.Context, qtid dax.QualifiedTableID, shard dax.ShardNum) (dax.Address, error) {
+	url := fmt.Sprintf("%s%s/ingest-shard", c.address.WithScheme(defaultScheme), defaultPath)
+
+	var host dax.Address
+
+	req := &mdshttp.IngestShardRequest{
+		Table: qtid,
+		Shard: shard,
+	}
+
+	// Encode the request.
+	postBody, err := json.Marshal(req)
+	if err != nil {
+		return host, errors.Wrap(err, "marshalling post request")
+	}
+	responseBody := bytes.NewBuffer(postBody)
+
+	// Post the request.
+	resp, err := http.Post(url, "application/json", responseBody)
+	if err != nil {
+		return host, errors.Wrap(err, "posting ingest-shard request")
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return host, errors.Errorf("status code: %d: %s", resp.StatusCode, b)
+	}
+
+	var isr *mdshttp.IngestShardResponse
+	if err := json.NewDecoder(resp.Body).Decode(&isr); err != nil {
+		return host, errors.Wrap(err, "reading response body")
+	}
+
+	return isr.Address, nil
+}
+
+func (c *Client) IngestPartition(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum) (dax.Address, error) {
+	url := fmt.Sprintf("%s%s/ingest-partition", c.address.WithScheme(defaultScheme), defaultPath)
+
+	var host dax.Address
+
+	req := &mdshttp.IngestPartitionRequest{
+		Table:     qtid,
+		Partition: partition,
+	}
+
+	// Encode the request.
+	postBody, err := json.Marshal(req)
+	if err != nil {
+		return host, errors.Wrap(err, "marshalling post request")
+	}
+	responseBody := bytes.NewBuffer(postBody)
+
+	// Post the request.
+	resp, err := http.Post(url, "application/json", responseBody)
+	if err != nil {
+		return host, errors.Wrap(err, "posting ingest-partition request")
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return host, errors.Errorf("status code: %d: %s", resp.StatusCode, b)
+	}
+
+	var isr *mdshttp.IngestPartitionResponse
+	if err := json.NewDecoder(resp.Body).Decode(&isr); err != nil {
+		return host, errors.Wrap(err, "reading response body")
+	}
+
+	return isr.Address, nil
+}
+
+func (c *Client) ComputeNodes(ctx context.Context, qtid dax.QualifiedTableID, shards ...dax.ShardNum) ([]controller.ComputeNode, error) {
+	url := fmt.Sprintf("%s%s/compute-nodes", c.address.WithScheme(defaultScheme), defaultPath)
+	log.Printf("ComputeNodes url: %s", url)
+
+	var nodes []controller.ComputeNode
+
+	req := &mdshttp.ComputeNodesRequest{
+		Table:  qtid,
+		Shards: shards,
+	}
+
+	// Encode the request.
+	postBody, err := json.Marshal(req)
+	if err != nil {
+		return nodes, errors.Wrap(err, "marshalling post request")
+	}
+	responseBody := bytes.NewBuffer(postBody)
+
+	// Post the request.
+	resp, err := http.Post(url, "application/json", responseBody)
+	if err != nil {
+		return nodes, errors.Wrap(err, "posting compute-nodes request")
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return nodes, errors.Errorf("status code: %d: %s", resp.StatusCode, b)
+	}
+
+	var cnr *mdshttp.ComputeNodesResponse
+	if err := json.NewDecoder(resp.Body).Decode(&cnr); err != nil {
+		return nodes, errors.Wrap(err, "reading response body")
+	}
+
+	return cnr.ComputeNodes, nil
+}
+
+func (c *Client) TranslateNodes(ctx context.Context, qtid dax.QualifiedTableID, partitions ...dax.PartitionNum) ([]controller.TranslateNode, error) {
+	url := fmt.Sprintf("%s%s/translate-nodes", c.address.WithScheme(defaultScheme), defaultPath)
+	log.Printf("TranslateNodes url: %s", url)
+
+	var nodes []controller.TranslateNode
+
+	req := &mdshttp.TranslateNodesRequest{
+		Table:      qtid,
+		Partitions: partitions,
+	}
+
+	// Encode the request.
+	postBody, err := json.Marshal(req)
+	if err != nil {
+		return nodes, errors.Wrap(err, "marshalling post request")
+	}
+	responseBody := bytes.NewBuffer(postBody)
+
+	// Post the request.
+	resp, err := http.Post(url, "application/json", responseBody)
+	if err != nil {
+		return nodes, errors.Wrap(err, "posting translate-nodes request")
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return nodes, errors.Errorf("status code: %d: %s", resp.StatusCode, b)
+	}
+
+	var cnr *mdshttp.TranslateNodesResponse
+	if err := json.NewDecoder(resp.Body).Decode(&cnr); err != nil {
+		return nodes, errors.Wrap(err, "reading response body")
+	}
+
+	return cnr.TranslateNodes, nil
+}
+
+func (c *Client) RegisterNode(ctx context.Context, node *dax.Node) error {
+	url := fmt.Sprintf("%s%s/register-node", c.address.WithScheme(defaultScheme), defaultPath)
+	log.Printf("RegisterNode url: %s", url)
+
+	req := &mdshttp.RegisterNodeRequest{
+		Address:   node.Address,
+		RoleTypes: node.RoleTypes,
+	}
+
+	// Encode the request.
+	postBody, err := json.Marshal(req)
+	if err != nil {
+		return errors.Wrap(err, "marshalling post request")
+	}
+	responseBody := bytes.NewBuffer(postBody)
+
+	// Post the request.
+	resp, err := http.Post(url, "application/json", responseBody)
+	if err != nil {
+		return errors.Wrap(err, "posting translate-nodes request")
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return errors.Errorf("status code: %d: %s", resp.StatusCode, b)
+	}
+
+	return nil
+}
+
+func (c *Client) CheckInNode(ctx context.Context, node *dax.Node) error {
+	url := fmt.Sprintf("%s%s/check-in-node", c.address.WithScheme(defaultScheme), defaultPath)
+	log.Printf("CheckInNode url: %s", url)
+
+	req := &mdshttp.CheckInNodeRequest{
+		Address:   node.Address,
+		RoleTypes: node.RoleTypes,
+	}
+
+	// Encode the request.
+	postBody, err := json.Marshal(req)
+	if err != nil {
+		return errors.Wrap(err, "marshalling post request")
+	}
+	responseBody := bytes.NewBuffer(postBody)
+
+	// Post the request.
+	resp, err := http.Post(url, "application/json", responseBody)
+	if err != nil {
+		return errors.Wrap(err, "posting translate-nodes request")
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return errors.Errorf("status code: %d: %s", resp.StatusCode, b)
+	}
+
+	return nil
+}

dax/mds/controller/alpha/director.go

@@ -0,0 +1,125 @@
+// Package alpha contains inter-service implemenations of interfaces.
+package alpha
+
+import (
+	"context"
+	"encoding/json"
+
+	featurebase "github.com/molecula/featurebase/v3"
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/mds/controller"
+	"github.com/molecula/featurebase/v3/errors"
+	featurebaseserver "github.com/molecula/featurebase/v3/server"
+)
+
+// Ensure type implements interface.
+var _ controller.Director = (*Director)(nil)
+
+// Director is a direct, service-to-service implementation of the Director
+// interface.
+type Director struct {
+	computers map[dax.Address]*featurebaseserver.Command
+}
+
+func NewDirector() *Director {
+	return &Director{
+		computers: make(map[dax.Address]*featurebaseserver.Command),
+	}
+}
+
+func (d *Director) AddCmd(addr dax.Address, cmd *featurebaseserver.Command) error {
+	if cmd == nil {
+		return errors.New(errors.ErrUncoded, "cannot add nil cmd to director")
+	}
+	d.computers[addr] = cmd
+	return nil
+}
+
+func (d *Director) api(addr dax.Address) (*featurebase.API, error) {
+	cmd, found := d.computers[addr]
+	if !found {
+		// Address not registered with the Director.
+		return nil, errors.New(errors.ErrUncoded, "cmd not registered with director")
+	}
+
+	api := cmd.API
+	if api == nil {
+		// Command does not have an API.
+		return nil, errors.New(errors.ErrUncoded, "cmd does not have an api")
+	}
+
+	return api, nil
+}
+
+func (d *Director) SendDirective(ctx context.Context, dir *dax.Directive) error {
+	api, err := d.api(dir.Address)
+	if err != nil {
+		return errors.Wrap(err, "getting api from director")
+	}
+
+	ndir, err := marshalUnmarshal(dir)
+	if err != nil {
+		return errors.Wrap(err, "marshalUnmarshal")
+	}
+
+	return api.Directive(ctx, ndir)
+}
+
+func (d *Director) SendSnapshotShardDataRequest(ctx context.Context, req *dax.SnapshotShardDataRequest) error {
+	api, err := d.api(req.Address)
+	if err != nil {
+		return errors.Wrap(err, "getting api from director")
+	}
+
+	nreq, err := marshalUnmarshal(req)
+	if err != nil {
+		return errors.Wrap(err, "marshalUnmarshal")
+	}
+
+	return api.SnapshotShardData(ctx, nreq)
+}
+
+func (d *Director) SendSnapshotTableKeysRequest(ctx context.Context, req *dax.SnapshotTableKeysRequest) error {
+	api, err := d.api(req.Address)
+	if err != nil {
+		return errors.Wrap(err, "getting api from director")
+	}
+
+	nreq, err := marshalUnmarshal(req)
+	if err != nil {
+		return errors.Wrap(err, "marshalUnmarshal")
+	}
+
+	return api.SnapshotTableKeys(ctx, nreq)
+}
+
+func (d *Director) SendSnapshotFieldKeysRequest(ctx context.Context, req *dax.SnapshotFieldKeysRequest) error {
+	api, err := d.api(req.Address)
+	if err != nil {
+		return errors.Wrap(err, "getting api from director")
+	}
+
+	nreq, err := marshalUnmarshal(req)
+	if err != nil {
+		return errors.Wrap(err, "marshalUnmarshal")
+	}
+
+	return api.SnapshotFieldKeys(ctx, nreq)
+}
+
+// marshalUnmarshal simply marshals anything to json, and then
+// unmarshals it. This might seem a bit silly. The reason it exists is
+// to exercise the same encode/decode logic that we'd need to if we
+// were traversing the network, and guarantee that we aren't sharing
+// pointers across API boundaries.
+func marshalUnmarshal[K any](a K) (K, error) {
+	var newA K
+	abytes, err := json.Marshal(a)
+	if err != nil {
+		return newA, errors.Wrap(err, "marshaling directive")
+	}
+	if err := json.Unmarshal(abytes, &newA); err != nil {
+		return newA, errors.Wrap(err, "unmarshaling directive")
+	}
+	return newA, nil
+}

dax/mds/controller/balancer.go

@@ -0,0 +1,69 @@
+package controller
+
+import (
+	"context"
+	"fmt"
+
+	"github.com/molecula/featurebase/v3/dax"
+)
+
+type Balancer interface {
+	AddWorker(ctx context.Context, worker fmt.Stringer) ([]dax.WorkerDiff, error)
+	RemoveWorker(ctx context.Context, worker fmt.Stringer) ([]dax.WorkerDiff, error)
+	AddJob(ctx context.Context, job fmt.Stringer) ([]dax.WorkerDiff, error)
+	RemoveJob(ctx context.Context, job fmt.Stringer) ([]dax.WorkerDiff, error)
+	Balance(ctx context.Context) ([]dax.WorkerDiff, error)
+	CurrentState(ctx context.Context) ([]dax.WorkerInfo, error)
+	WorkerState(ctx context.Context, worker dax.Worker) (dax.WorkerInfo, error)
+	WorkersForJobs(ctx context.Context, jobs []dax.Job) ([]dax.WorkerInfo, error)
+
+	// WorkersForJobPrefix returns all workers and their job
+	// assignments which start with `prefix` for all jobs that start
+	// with `prefix`. If there are free jobs that start with `prefix`
+	// an error is returned.
+	//
+	// The motivating use case is getting all workers for a particular
+	// table so we can execute a query that will hit every shard in a
+	// table. If there are jobs representing shards in that table
+	// which are not assigned to any worker, that means the query
+	// would return incomplete data, so we want to error.
+	WorkersForJobPrefix(ctx context.Context, prefix string) ([]dax.WorkerInfo, error)
+}
+
+// Ensure type implements interface.
+var _ Balancer = (*NopBalancer)(nil)
+
+// NopBalancer is a no-op implementation of the Balancer interface.
+type NopBalancer struct{}
+
+func NewNopBalancer() *NopBalancer {
+	return &NopBalancer{}
+}
+
+func (b *NopBalancer) AddWorker(ctx context.Context, worker fmt.Stringer) ([]dax.WorkerDiff, error) {
+	return []dax.WorkerDiff{}, nil
+}
+func (b *NopBalancer) RemoveWorker(ctx context.Context, worker fmt.Stringer) ([]dax.WorkerDiff, error) {
+	return []dax.WorkerDiff{}, nil
+}
+func (b *NopBalancer) AddJob(ctx context.Context, job fmt.Stringer) ([]dax.WorkerDiff, error) {
+	return []dax.WorkerDiff{}, nil
+}
+func (b *NopBalancer) RemoveJob(ctx context.Context, job fmt.Stringer) ([]dax.WorkerDiff, error) {
+	return []dax.WorkerDiff{}, nil
+}
+func (b *NopBalancer) Balance(ctx context.Context) ([]dax.WorkerDiff, error) {
+	return []dax.WorkerDiff{}, nil
+}
+func (b *NopBalancer) CurrentState(ctx context.Context) ([]dax.WorkerInfo, error) {
+	return []dax.WorkerInfo{}, nil
+}
+func (b *NopBalancer) WorkerState(ctx context.Context, worker dax.Worker) (dax.WorkerInfo, error) {
+	return dax.WorkerInfo{}, nil
+}
+func (b *NopBalancer) WorkersForJobs(ctx context.Context, jobs []dax.Job) ([]dax.WorkerInfo, error) {
+	return []dax.WorkerInfo{}, nil
+}
+func (b *NopBalancer) WorkersForJobPrefix(ctx context.Context, prefix string) ([]dax.WorkerInfo, error) {
+	return []dax.WorkerInfo{}, nil
+}

dax/mds/controller/config.go

@@ -0,0 +1,29 @@
+package controller
+
+import (
+	"time"
+
+	"github.com/molecula/featurebase/v3/dax/boltdb"
+	"github.com/molecula/featurebase/v3/dax/mds/schemar"
+	"github.com/molecula/featurebase/v3/logger"
+)
+
+type NewBalancerFn func(string, logger.Logger) Balancer
+
+type Config struct {
+	Director          Director
+	Schemar           schemar.Schemar
+	ComputeBalancer   Balancer
+	TranslateBalancer Balancer
+
+	StorageMethod string
+	BoltDB        *boltdb.DB
+
+	// RegistrationBatchTimeout is the time that the controller will
+	// wait after a node registers itself to see if any more nodes
+	// will register before sending out directives to all nodes which
+	// have been registered.
+	RegistrationBatchTimeout time.Duration
+
+	Logger logger.Logger
+}

dax/mds/controller/director.go

@@ -0,0 +1,40 @@
+package controller
+
+import (
+	"context"
+
+	"github.com/molecula/featurebase/v3/dax"
+)
+
+type Director interface {
+	SendDirective(ctx context.Context, dir *dax.Directive) error
+	SendSnapshotShardDataRequest(ctx context.Context, req *dax.SnapshotShardDataRequest) error
+	SendSnapshotTableKeysRequest(ctx context.Context, req *dax.SnapshotTableKeysRequest) error
+	SendSnapshotFieldKeysRequest(ctx context.Context, req *dax.SnapshotFieldKeysRequest) error
+}
+
+// Ensure type implements interface.
+var _ Director = &NopDirector{}
+
+// NopDirector is a no-op implementation of the Director interface.
+type NopDirector struct{}
+
+func NewNopDirector() *NopDirector {
+	return &NopDirector{}
+}
+
+func (d *NopDirector) SendDirective(ctx context.Context, dir *dax.Directive) error {
+	return nil
+}
+
+func (d *NopDirector) SendSnapshotShardDataRequest(ctx context.Context, req *dax.SnapshotShardDataRequest) error {
+	return nil
+}
+
+func (d *NopDirector) SendSnapshotTableKeysRequest(ctx context.Context, req *dax.SnapshotTableKeysRequest) error {
+	return nil
+}
+
+func (d *NopDirector) SendSnapshotFieldKeysRequest(ctx context.Context, req *dax.SnapshotFieldKeysRequest) error {
+	return nil
+}

dax/mds/controller/errors.go

@@ -0,0 +1,101 @@
+package controller
+
+import (
+	"fmt"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/errors"
+)
+
+const (
+	// ErrCodeCustom can be used to return a custom error message.
+	ErrCodeCustom errors.Code = "CustomError"
+
+	// ErrCodeInternal can be used when the cause of an error can't be
+	// determined. It can be accompanied by a single string message.
+	ErrCodeInternal errors.Code = "InternalError"
+
+	// ErrCodeTODO can be used as a placeholder until a proper error code is
+	// created and assigned.
+	ErrCodeTODO errors.Code = "TODOError"
+
+	ErrCodeNodeExists      errors.Code = "NodeExists"
+	ErrCodeNodeKeyInvalid  errors.Code = "NodeKeyInvalid"
+	ErrCodeNoAvailableNode errors.Code = "NoAvailableNode"
+
+	ErrCodeRoleTypeInvalid errors.Code = "RoleTypeInvalid"
+
+	ErrCodeDirectiveSendFailure errors.Code = "DirectiveSendFailure"
+
+	ErrCodeInvalidRequest errors.Code = "InvalidRequest"
+
+	ErrCodeUnassignedJobs errors.Code = "UnassignedJobs"
+
+	UndefinedErrorMessage string = "undefined message format"
+)
+
+// NewErrCustom can be used to return a custom error message.
+func NewErrCustom() error {
+	return errors.New(
+		ErrCodeCustom,
+		"",
+	)
+}
+
+// NewErrInternal can be used when the cause of an error can't be determined. It
+// can be accompanied by a single string message.
+func NewErrInternal(msg string) error {
+	return errors.New(
+		ErrCodeInternal,
+		fmt.Sprintf("internal error: %s", msg),
+	)
+}
+
+func NewErrNodeExists(addr dax.Address) error {
+	return errors.New(
+		ErrCodeNodeExists,
+		fmt.Sprintf("node '%s' already exists", addr),
+	)
+}
+
+func NewErrNodeKeyInvalid(addr dax.Address) error {
+	return errors.New(
+		ErrCodeNodeKeyInvalid,
+		fmt.Sprintf("node key '%s' is invalid", addr),
+	)
+}
+
+func NewErrNoAvailableNode() error {
+	return errors.New(
+		ErrCodeNoAvailableNode,
+		"no available node",
+	)
+}
+
+func NewErrRoleTypeInvalid(roleType dax.RoleType) error {
+	return errors.New(
+		ErrCodeRoleTypeInvalid,
+		fmt.Sprintf("role type '%s' is invalid", roleType),
+	)
+}
+
+func NewErrDirectiveSendFailure(msg string) error {
+	return errors.New(
+		ErrCodeDirectiveSendFailure,
+		fmt.Sprintf("directive failed to send: %s", msg),
+	)
+}
+
+func NewErrInvalidRequest(msg string) error {
+	return errors.New(
+		ErrCodeInvalidRequest,
+		fmt.Sprintf("invalid request: %s", msg),
+	)
+}
+
+func NewErrUnassignedJobs(jobs []dax.Job) error {
+	return errors.New(
+		ErrCodeUnassignedJobs,
+		fmt.Sprintf("found %d unassigned jobs", len(jobs)),
+	)
+}

dax/mds/controller/http/director.go

@@ -0,0 +1,185 @@
+// Package http provides the http implementation of the Director interface.
+package http
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net"
+	"net/http"
+	"time"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/errors"
+	"github.com/molecula/featurebase/v3/logger"
+)
+
+// Director is an http implementation of the Director interface.
+type Director struct {
+	// directivePath is the path portion of the URI to which directives should
+	// be POSTed.
+	directivePath string
+
+	// snapshotRequestPath is the path portion of the URI to which snapshot
+	// requests should be POSTed.
+	snapshotRequestPath string
+
+	client *http.Client
+
+	logger logger.Logger
+}
+
+func NewDirector(cfg DirectorConfig) *Director {
+	var logr logger.Logger = logger.NopLogger
+	if cfg.Logger != nil {
+		logr = cfg.Logger
+	}
+
+	return &Director{
+		directivePath:       cfg.DirectivePath,
+		snapshotRequestPath: cfg.SnapshotRequestPath,
+		logger:              logr,
+		client: &http.Client{
+			Transport: &http.Transport{
+				Proxy: http.ProxyFromEnvironment,
+				DialContext: (&net.Dialer{
+					Timeout:   2 * time.Second,
+					KeepAlive: 30 * time.Second,
+				}).DialContext,
+				ForceAttemptHTTP2:     true,
+				MaxIdleConns:          100,
+				IdleConnTimeout:       90 * time.Second,
+				TLSHandshakeTimeout:   3 * time.Second,
+				ExpectContinueTimeout: 1 * time.Second,
+			},
+		},
+	}
+}
+
+type DirectorConfig struct {
+	DirectivePath       string
+	SnapshotRequestPath string
+	Logger              logger.Logger
+}
+
+func (d *Director) SendDirective(ctx context.Context, dir *dax.Directive) error {
+	url := fmt.Sprintf("%s/%s", dir.Address.WithScheme("http"), d.directivePath)
+	d.logger.Printf("SEND HTTP directive to: %s\n", url)
+
+	// Encode the request.
+	postBody, err := json.Marshal(dir)
+	if err != nil {
+		return errors.Wrap(err, "marshalling directive to json")
+	}
+	requestBody := bytes.NewBuffer(postBody)
+
+	// Post the request.
+	request, _ := http.NewRequest(http.MethodPost, url, requestBody)
+	request.Header.Add("Content-Type", "application/json")
+	request.Header.Add("Accept", "application/json")
+
+	resp, err := d.client.Do(request)
+	if err != nil {
+		return errors.Wrap(err, "doing send directive")
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return errors.Errorf("status code: %d: %s", resp.StatusCode, b)
+	}
+
+	return nil
+}
+
+func (d *Director) SendSnapshotShardDataRequest(ctx context.Context, req *dax.SnapshotShardDataRequest) error {
+	url := fmt.Sprintf("%s/%s/shard-data", req.Address.WithScheme("http"), d.snapshotRequestPath)
+	d.logger.Printf("SEND HTTP snapshot shard data request to: %s\n", url)
+
+	// Encode the request.
+	postBody, err := json.Marshal(req)
+	if err != nil {
+		return errors.Wrap(err, "marshalling snapshot shard data request to json")
+	}
+	requestBody := bytes.NewBuffer(postBody)
+
+	// Post the request.
+	request, _ := http.NewRequest(http.MethodPost, url, requestBody)
+	request.Header.Add("Content-Type", "application/json")
+	request.Header.Add("Accept", "application/json")
+
+	resp, err := d.client.Do(request)
+	if err != nil {
+		return errors.Wrap(err, "doing snapshot shard data request")
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return errors.Errorf("status code: %d: %s", resp.StatusCode, b)
+	}
+
+	return nil
+}
+
+func (d *Director) SendSnapshotTableKeysRequest(ctx context.Context, req *dax.SnapshotTableKeysRequest) error {
+	url := fmt.Sprintf("%s/%s/table-keys", req.Address.WithScheme("http"), d.snapshotRequestPath)
+	d.logger.Printf("SEND HTTP snapshot table keys request to: %s\n", url)
+
+	// Encode the request.
+	postBody, err := json.Marshal(req)
+	if err != nil {
+		return errors.Wrap(err, "marshalling snapshot table keys request to json")
+	}
+	requestBody := bytes.NewBuffer(postBody)
+
+	// Post the request.
+	request, _ := http.NewRequest(http.MethodPost, url, requestBody)
+	request.Header.Add("Content-Type", "application/json")
+	request.Header.Add("Accept", "application/json")
+
+	resp, err := d.client.Do(request)
+	if err != nil {
+		return errors.Wrap(err, "doing snapshot table keys request")
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return errors.Errorf("status code: %d: %s", resp.StatusCode, b)
+	}
+
+	return nil
+}
+
+func (d *Director) SendSnapshotFieldKeysRequest(ctx context.Context, req *dax.SnapshotFieldKeysRequest) error {
+	url := fmt.Sprintf("%s/%s/field-keys", req.Address.WithScheme("http"), d.snapshotRequestPath)
+	d.logger.Printf("SEND HTTP snapshot field keys request to: %s\n", url)
+
+	// Encode the request.
+	postBody, err := json.Marshal(req)
+	if err != nil {
+		return errors.Wrap(err, "marshalling snapshot field keys request to json")
+	}
+	requestBody := bytes.NewBuffer(postBody)
+
+	// Post the request.
+	request, _ := http.NewRequest(http.MethodPost, url, requestBody)
+	request.Header.Add("Content-Type", "application/json")
+	request.Header.Add("Accept", "application/json")
+
+	resp, err := d.client.Do(request)
+	if err != nil {
+		return errors.Wrap(err, "doing snapshot field keys request")
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return errors.Errorf("status code: %d: %s", resp.StatusCode, b)
+	}
+
+	return nil
+}

dax/mds/controller/naive/balancer.go

@@ -0,0 +1,539 @@
+// Package naive contains a naive implementation of the Balancer interface.
+package naive
+
+import (
+	"context"
+	"fmt"
+	"math"
+	"sort"
+	"strings"
+	"sync"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/errors"
+	"github.com/molecula/featurebase/v3/logger"
+)
+
+// Balancer is a naive implementation of the controller.Balancer interface. It
+// helps manage the relationships between workers and jobs. The logic it uses to
+// balance jobs across workers is very simple; it bases everything off the
+// number of workers and number of jobs. It does not take anything else (such as
+// job size, worker capabilities, etc) into consideration.
+type Balancer struct {
+	mu sync.RWMutex
+
+	// name is used in logging to help identify the balancer responsible for the
+	// log.
+	name string
+
+	// current represents the current state of worker/job assigments.
+	current WorkerJobService
+
+	// freeJobs is the set of jobs which have yet to be assigned to a worker.
+	// This could be because there are no available workers, or because a worker
+	// has been removed and the jobs for which it was responsible have yet to be
+	// reassigned.
+	freeJobs FreeJobService
+
+	logger logger.Logger
+}
+
+type WorkerJobService interface {
+	WorkersJobs(ctx context.Context, balancerName string) ([]dax.WorkerInfo, error)
+
+	WorkerCount(ctx context.Context, balancerName string) (int, error)
+	ListWorkers(ctx context.Context, balancerName string) (dax.Workers, error)
+	WorkerExists(ctx context.Context, balancerName string, worker dax.Worker) (bool, error)
+	CreateWorker(ctx context.Context, balancerName string, worker dax.Worker) error
+	DeleteWorker(ctx context.Context, balancerName string, worker dax.Worker) error
+
+	CreateJob(ctx context.Context, balancerName string, worker dax.Worker, job dax.Job) error
+	DeleteJob(ctx context.Context, balancerName string, worker dax.Worker, job dax.Job) error
+	JobCount(ctx context.Context, balancerName string, worker dax.Worker) (int, error)
+	ListJobs(ctx context.Context, balancerName string, worker dax.Worker) (dax.Jobs, error)
+}
+
+type FreeJobService interface {
+	CreateFreeJob(ctx context.Context, balancerName string, job dax.Job) error
+	DeleteFreeJob(ctx context.Context, balancerName string, job dax.Job) error
+	ListFreeJobs(ctx context.Context, balancerName string) (dax.Jobs, error)
+	MergeFreeJobs(ctx context.Context, balancerName string, jobs dax.Jobs) error
+}
+
+// New returns a new instance of Balancer.
+func New(name string, fjs FreeJobService, wjs WorkerJobService, logger logger.Logger) *Balancer {
+	return &Balancer{
+		name:     name,
+		current:  wjs,
+		freeJobs: fjs,
+		logger:   logger,
+	}
+}
+
+// AddWorker adds a worker to the Balancer's worker pool. This may cause the
+// Balancer to assign existing jobs that are currently in the free list to the
+// worker. Also, the worker will immediately be available for assignments of new
+// jobs.
+func (b *Balancer) AddWorker(ctx context.Context, worker fmt.Stringer) ([]dax.WorkerDiff, error) {
+	b.logger.Debugf("%s: AddWorker(%s)", b.name, worker.String())
+	b.mu.Lock()
+	defer b.mu.Unlock()
+
+	diff, err := b.addWorker(ctx, dax.Worker(worker.String()))
+	if err != nil {
+		return nil, errors.Wrap(err, "adding worker")
+	}
+
+	return diff.output(), nil
+}
+
+func (b *Balancer) addWorker(ctx context.Context, worker dax.Worker) (internalDiffs, error) {
+	// If this worker already exists, don't do anything.
+	if exists, err := b.current.WorkerExists(ctx, b.name, worker); err != nil {
+		return nil, errors.Wrap(err, "checking if worker exists")
+	} else if exists {
+		return internalDiffs{}, nil
+	}
+
+	if err := b.current.CreateWorker(ctx, b.name, worker); err != nil {
+		return nil, errors.Wrap(err, "creating worker")
+	}
+
+	// Process the freeJobs.
+	return b.processFreeJobs(ctx)
+}
+
+// ReplaceWorker is meant to avoid the job re-assignment caused by performing a
+// RemoveWorker followed by an AddWorker. In this case, it does both in one step
+// so that it's more likely that the jobs will just get transferred directly
+// over. NOT IMPLEMENTED YET.
+// func (b *Balancer) ReplaceWorker(fromWorker string, toWorker string) []WorkerDiff {
+// 	b.mu.Lock()
+// 	defer b.mu.Unlock()
+
+// 	return []WorkerDiff{}
+// }
+
+// RemoveWorker removes a worker from the worker pool and moves any of its
+// currently assigned jobs to the free list. If the intention is to remove a
+// worker and reassign its jobs to other workers, then RemoveWorker() should be
+// followed by Balance().
+func (b *Balancer) RemoveWorker(ctx context.Context, worker fmt.Stringer) ([]dax.WorkerDiff, error) {
+	b.mu.Lock()
+	defer b.mu.Unlock()
+
+	diff, err := b.removeWorker(ctx, dax.Worker(worker.String()))
+	if err != nil {
+		return nil, errors.Wrap(err, "removing worker")
+	}
+
+	return diff.output(), nil
+}
+
+func (b *Balancer) removeWorker(ctx context.Context, worker dax.Worker) (internalDiffs, error) {
+	// If this worker doesn't exist, don't do anything else.
+	if exists, err := b.current.WorkerExists(ctx, b.name, worker); err != nil {
+		return nil, errors.Wrap(err, "checking if worker exists")
+	} else if !exists {
+		return internalDiffs{}, nil
+	}
+
+	jobs, err := b.current.ListJobs(ctx, b.name, worker)
+	if err != nil {
+		return nil, errors.Wrap(err, "listing jobs")
+	}
+
+	// Before removing the worker, mark its jobs as free.
+	if err := b.freeJobs.MergeFreeJobs(ctx, b.name, jobs); err != nil {
+		return nil, errors.Wrap(err, "merging free jobs")
+	}
+
+	// Remove the worker.
+	if err := b.current.DeleteWorker(ctx, b.name, worker); err != nil {
+		return nil, errors.Wrap(err, "deleting worker")
+	}
+
+	// Even though this may not be useful to the caller (for example, in the
+	// case where the worker has died and no longer exists), return the diffs
+	// which represent the removal of jobs from the worker.
+	diff := newInternalDiffs()
+	for _, job := range jobs {
+		diff.removed(worker, job)
+	}
+
+	return diff, nil
+}
+
+// AddJob adds a job to an existing worker. If there are no existing workers,
+// the job is placed into the free list and will be assigned to a worker once
+// one becomes available.
+func (b *Balancer) AddJob(ctx context.Context, job fmt.Stringer) ([]dax.WorkerDiff, error) {
+	b.logger.Debugf("%s: AddJob(%s)", b.name, job.String())
+	b.mu.Lock()
+	defer b.mu.Unlock()
+
+	diff, err := b.addJob(ctx, dax.Job(job.String()))
+	if err != nil {
+		return nil, errors.Wrap(err, "adding job")
+	}
+
+	return diff.output(), nil
+}
+
+func (b *Balancer) addJob(ctx context.Context, job dax.Job) (internalDiffs, error) {
+	if cnt, err := b.current.WorkerCount(ctx, b.name); err != nil {
+		return nil, errors.Wrap(err, "getting worker count")
+	} else if cnt == 0 {
+		if err := b.freeJobs.CreateFreeJob(ctx, b.name, job); err != nil {
+			return nil, errors.Wrap(err, "creating free job")
+		}
+		// TODO: we might want to inform the user that a job is in the free list
+		// because there are no workers.
+		return internalDiffs{}, nil
+	}
+
+	// Make sure this job doesn't already exist.
+	if _, ok, err := b.workerForJob(ctx, job); err != nil {
+		return nil, errors.Wrapf(err, "getting worker for job: %s", job)
+	} else if ok {
+		// The job is already being tracked.
+		return internalDiffs{}, nil
+	}
+
+	// Find the worker with the fewest number of jobs and assign it this job.
+	var lowCount int = math.MaxInt
+	var lowWorker dax.Worker
+
+	workerIDs, err := b.current.ListWorkers(ctx, b.name)
+	if err != nil {
+		return nil, errors.Wrap(err, "listing workers")
+	}
+
+	for _, workerID := range workerIDs {
+		if l, err := b.current.JobCount(ctx, b.name, workerID); err != nil {
+			return nil, errors.Wrapf(err, "getting job count for worker: %s", workerID)
+		} else if l < lowCount {
+			lowCount = l
+			lowWorker = workerID
+		}
+	}
+
+	if err := b.current.CreateJob(ctx, b.name, lowWorker, job); err != nil {
+		return nil, errors.Wrap(err, "creating job")
+	}
+
+	diffs := newInternalDiffs()
+	diffs.added(lowWorker, job)
+
+	return diffs, nil
+}
+
+// RemoveJob removes a job from the worker to which is was assigned. If the job
+// is not currently assigned to a worker, but it is in the free list, then it
+// will be removed from the free list.
+func (b *Balancer) RemoveJob(ctx context.Context, job fmt.Stringer) ([]dax.WorkerDiff, error) {
+	b.mu.Lock()
+	defer b.mu.Unlock()
+
+	diff, err := b.removeJob(ctx, dax.Job(job.String()))
+	if err != nil {
+		return nil, errors.Wrapf(err, "removing job: %s", job)
+	}
+
+	return diff.output(), nil
+}
+
+func (b *Balancer) removeJob(ctx context.Context, job dax.Job) (internalDiffs, error) {
+	if worker, ok, err := b.workerForJob(ctx, job); err != nil {
+		return nil, errors.Wrapf(err, "getting worker for job: %s", job)
+	} else if ok {
+		if err := b.current.DeleteJob(ctx, b.name, worker, job); err != nil {
+			return nil, errors.Wrapf(err, "deleting job: %s", job)
+		}
+
+		diffs := newInternalDiffs()
+		diffs.removed(worker, job)
+
+		return diffs, nil
+	}
+
+	// Just in case the job is in the free list (and wasn't assigned to a
+	// worker), remove it; there's no need to provide a diff. There should never
+	// be a case where the same job is both in the free list and assigned to a
+	// worker.
+	if err := b.freeJobs.DeleteFreeJob(ctx, b.name, job); err != nil {
+		return nil, errors.Wrapf(err, "deleting free job: %s", job)
+	}
+
+	return internalDiffs{}, nil
+}
+
+// Balance ensures that all jobs are being handled by a worker by assigning jobs
+// in the free list to workers, and by moving job assignments around in order to
+// balance the load on workers.
+func (b *Balancer) Balance(ctx context.Context) ([]dax.WorkerDiff, error) {
+	b.mu.Lock()
+	defer b.mu.Unlock()
+
+	// If there are no workers, we can't properly balance.
+	if cnt, err := b.current.WorkerCount(ctx, b.name); err != nil {
+		return nil, errors.Wrapf(err, "getting worker count: %s", b.name)
+	} else if cnt == 0 {
+		return []dax.WorkerDiff{}, nil
+	}
+
+	// Process the freeJobs.
+	diffs, err := b.processFreeJobs(ctx)
+	if err != nil {
+		return nil, errors.Wrapf(err, "processing free jobs: %s", b.name)
+	}
+
+	// Balance the jobs among workers.
+	diff, err := b.balance(ctx, diffs)
+	if err != nil {
+		return nil, errors.Wrap(err, "balancing jobs")
+	}
+
+	return diff.output(), nil
+}
+
+// balance moves jobs among workers with the goal of having an equal number of
+// jobs per worker. This method takes an `internalDiffs` as input for cases
+// where some action has preceeded this call which also resulted in
+// `internalDiffs`. Instead of having this method take a value, we could rely on
+// the internalDiffs.merge() method, but we would need to modify that method to
+// be smarter about the order in which it applies the add/remove operations.
+// Until that's in place, we'll pass in a value here.
+func (b *Balancer) balance(ctx context.Context, diffs internalDiffs) (internalDiffs, error) {
+	numWorkers, err := b.current.WorkerCount(ctx, b.name)
+	if err != nil {
+		return nil, errors.Wrapf(err, "getting worker count: %s", b.name)
+	}
+	numJobs := 0
+	if workers, err := b.current.ListWorkers(ctx, b.name); err != nil {
+		return nil, errors.Wrapf(err, "listing workers: %s", b.name)
+	} else {
+		for _, worker := range workers {
+			cnt, err := b.current.JobCount(ctx, b.name, worker)
+			if err != nil {
+				return nil, errors.Wrapf(err, "getting job count: %s", worker)
+			}
+			numJobs += cnt
+		}
+	}
+
+	minJobsPerWorker := numJobs / numWorkers
+	numWorkersAboveMin := numJobs % numWorkers
+
+	// sortedWorkerInfos is used now in order to guarantee a sort order.
+	sortedWorkerInfos, err := b.currentState(ctx, true)
+	if err != nil {
+		return nil, errors.Wrapf(err, "getting current state: %s", b.name)
+	}
+
+	// Loop through each worker, and if the number of jobs for the worker
+	// exceeds the target, then remove the job and add it back (which is
+	// effectively how we rebalance a job).
+	for i, workerInfo := range sortedWorkerInfos {
+		numTargetJobs := minJobsPerWorker
+		if i < numWorkersAboveMin {
+			numTargetJobs += 1
+		}
+
+		numCurrentJobs, err := b.current.JobCount(ctx, b.name, workerInfo.ID)
+		if err != nil {
+			return nil, errors.Wrapf(err, "getting job count: %s", workerInfo.ID)
+		}
+
+		// If we don't need to remove jobs from this worker, then just continue
+		// on to the next worker.
+		if numCurrentJobs <= numTargetJobs {
+			continue
+		}
+
+		sortedJobs, err := b.current.ListJobs(ctx, b.name, workerInfo.ID)
+		if err != nil {
+			return nil, errors.Wrapf(err, "listing jobs: %s", workerInfo.ID)
+		}
+
+		// Remove the extra jobs from the end of the list, and add them back
+		// again (which should place them on a worker with fewer jobs).
+		for i := numCurrentJobs - 1; i >= numTargetJobs; i-- {
+			if rj, err := b.removeJob(ctx, sortedJobs[i]); err != nil {
+				return nil, errors.Wrapf(err, "removing job: %s", sortedJobs[i])
+			} else {
+				diffs.merge(rj)
+			}
+			if aj, err := b.addJob(ctx, sortedJobs[i]); err != nil {
+				return nil, errors.Wrapf(err, "adding job: %s", sortedJobs[i])
+			} else {
+				diffs.merge(aj)
+			}
+		}
+	}
+
+	return diffs, nil
+}
+
+// CurrentState returns the current state of worker and job assignments. Note
+// that there could be unassigned jobs which are not captured in this output.
+// Calling Balance() would force any unassigned jobs to be assigned (assuming
+// there is at least one worker), and the output would then reflect that.
+func (b *Balancer) CurrentState(ctx context.Context) ([]dax.WorkerInfo, error) {
+	b.mu.RLock()
+	defer b.mu.RUnlock()
+
+	return b.currentState(ctx, true)
+}
+
+func (b *Balancer) currentState(ctx context.Context, sorted bool) ([]dax.WorkerInfo, error) {
+	return b.current.WorkersJobs(ctx, b.name)
+}
+
+// WorkerState returns the current state of job assignments for a given worker.
+func (b *Balancer) WorkerState(ctx context.Context, worker dax.Worker) (dax.WorkerInfo, error) {
+	b.mu.RLock()
+	defer b.mu.RUnlock()
+
+	return b.workerState(ctx, worker)
+}
+
+func (b *Balancer) workerState(ctx context.Context, worker dax.Worker) (dax.WorkerInfo, error) {
+	if exists, err := b.current.WorkerExists(ctx, b.name, worker); err != nil {
+		return dax.WorkerInfo{}, errors.Wrapf(err, "checking worker exists: %s", worker)
+	} else if !exists {
+		return dax.WorkerInfo{
+			ID: dax.Worker(worker),
+		}, nil
+	}
+
+	jobs, err := b.current.ListJobs(ctx, b.name, worker)
+	if err != nil {
+		return dax.WorkerInfo{}, errors.Wrapf(err, "listing jobs: %s", worker)
+	}
+
+	return dax.WorkerInfo{
+		ID:   dax.Worker(worker),
+		Jobs: jobs,
+	}, nil
+}
+
+// WorkersForJobs returns the list of workers for the given jobs. If a given job
+// is not currently assigned to a worker, it will be ignored.
+func (b *Balancer) WorkersForJobs(ctx context.Context, jobs []dax.Job) ([]dax.WorkerInfo, error) {
+	b.mu.RLock()
+	defer b.mu.RUnlock()
+
+	return b.workersForJobs(ctx, jobs)
+}
+
+func (b *Balancer) workersForJobs(ctx context.Context, jobs []dax.Job) ([]dax.WorkerInfo, error) {
+	out := make(map[dax.Worker]dax.Set[dax.Job])
+
+	workerJobs, err := b.current.WorkersJobs(ctx, b.name)
+	if err != nil {
+		return nil, errors.Wrapf(err, "getting worker jobs: %s", b.name)
+	}
+	for _, workerInfo := range workerJobs {
+		jset := dax.NewSet(workerInfo.Jobs...)
+
+		matches := dax.NewSet[dax.Job]()
+		for _, job := range jobs {
+			if jset.Contains(job) {
+				matches.Add(job)
+			}
+		}
+
+		if len(matches) > 0 {
+			out[workerInfo.ID] = matches
+		}
+	}
+
+	workers := make([]dax.WorkerInfo, len(out))
+
+	i := 0
+	for w, jset := range out {
+		workers[i] = dax.WorkerInfo{
+			ID:   dax.Worker(w),
+			Jobs: jset.Sorted(),
+		}
+		i++
+	}
+
+	sort.Sort(dax.WorkerInfos(workers))
+
+	return workers, nil
+}
+
+func (b *Balancer) WorkersForJobPrefix(ctx context.Context, prefix string) ([]dax.WorkerInfo, error) {
+	b.mu.RLock()
+	defer b.mu.RUnlock()
+
+	jobs, err := b.freeJobs.ListFreeJobs(ctx, b.name)
+	if err != nil {
+		return nil, errors.Wrap(err, "listing free jobs")
+	}
+	for _, job := range jobs {
+		if strings.HasPrefix(string(job), prefix) {
+			return nil, errors.Errorf("found free job '%s' matching prefix '%s'", job, prefix)
+		}
+	}
+
+	workerJobs, err := b.current.WorkersJobs(ctx, b.name)
+	if err != nil {
+		return nil, errors.Wrapf(err, "getting worker jobs: %s", b.name)
+	}
+
+	result := make([]dax.WorkerInfo, 0)
+	for _, workerInfo := range workerJobs {
+		matchedJobs := make([]dax.Job, 0)
+		for _, job := range workerInfo.Jobs {
+			if strings.HasPrefix(string(job), prefix) {
+				matchedJobs = append(matchedJobs, job)
+			}
+		}
+		if len(matchedJobs) > 0 {
+			result = append(result, dax.WorkerInfo{
+				ID:   workerInfo.ID,
+				Jobs: matchedJobs,
+			})
+		}
+	}
+	return result, nil
+
+}
+
+// processFreeJobs assigns all jobs in the free list to a worker.
+func (b *Balancer) processFreeJobs(ctx context.Context) (internalDiffs, error) {
+	diffs := newInternalDiffs()
+	jobs, err := b.freeJobs.ListFreeJobs(ctx, b.name)
+	if err != nil {
+		return nil, errors.Wrapf(err, "listing free jobs: %s", b.name)
+	}
+	for _, job := range jobs {
+		if aj, err := b.addJob(ctx, job); err != nil {
+			return nil, errors.Wrapf(err, "adding job: %s", job)
+		} else {
+			diffs.merge(aj)
+		}
+		if err := b.freeJobs.DeleteFreeJob(ctx, b.name, job); err != nil {
+			return nil, errors.Wrapf(err, "deleting free job: %s", job)
+		}
+	}
+	return diffs, nil
+}
+
+// workerForJob returns the worker currently assigned to the given job.
+func (b *Balancer) workerForJob(ctx context.Context, job dax.Job) (dax.Worker, bool, error) {
+	workerJobs, err := b.current.WorkersJobs(ctx, b.name)
+	if err != nil {
+		return "", false, errors.Wrapf(err, "getting workers jobs: %s", b.name)
+	}
+	for _, workerInfo := range workerJobs {
+		jset := dax.NewSet(workerInfo.Jobs...)
+		if jset.Contains(job) {
+			return workerInfo.ID, true, nil
+		}
+	}
+	return "", false, nil
+}

dax/mds/controller/naive/balancer_test.go

@@ -0,0 +1,739 @@
+package naive_test
+
+import (
+	"context"
+	"fmt"
+	"os"
+	"testing"
+
+	"github.com/molecula/featurebase/v3/dax"
+	daxbolt "github.com/molecula/featurebase/v3/dax/boltdb"
+	"github.com/molecula/featurebase/v3/dax/mds/controller/naive/boltdb"
+	testbolt "github.com/molecula/featurebase/v3/dax/test/boltdb"
+	"github.com/molecula/featurebase/v3/logger"
+	"github.com/stretchr/testify/assert"
+)
+
+func newBoltBalancer(t *testing.T) (*daxbolt.DB, func()) {
+	db := testbolt.MustOpenDB(t)
+	assert.NoError(t, db.InitializeBuckets(boltdb.NaiveBalancerBuckets...))
+
+	return db, func() {
+		testbolt.MustCloseDB(t, db)
+		testbolt.CleanupDB(t, db.Path())
+	}
+}
+
+func TestBalancer(t *testing.T) {
+	ctx := context.Background()
+	t.Run("SingleWorker", func(t *testing.T) {
+		db, cleanup := newBoltBalancer(t)
+		defer cleanup()
+		bal := boltdb.NewBalancer("test", db, logger.NewStandardLogger(os.Stderr))
+		tests := []struct {
+			fn       func(context.Context, fmt.Stringer) ([]dax.WorkerDiff, error)
+			input    string
+			expDiff  []dax.WorkerDiff
+			expState []dax.WorkerInfo
+		}{
+			{
+				// Add job.
+				fn:       bal.AddJob,
+				input:    "p2",
+				expDiff:  []dax.WorkerDiff{},
+				expState: []dax.WorkerInfo{},
+			},
+			{
+				// Add worker.
+				fn:    bal.AddWorker,
+				input: "n1",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n1",
+						AddedJobs:   []dax.Job{"p2"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p2"},
+					},
+				},
+			},
+			{
+				// Add another job out of order.
+				fn:    bal.AddJob,
+				input: "p1",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n1",
+						AddedJobs:   []dax.Job{"p1"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2"},
+					},
+				},
+			},
+			{
+				// Add another job.
+				fn:    bal.AddJob,
+				input: "p3",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n1",
+						AddedJobs:   []dax.Job{"p3"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2", "p3"},
+					},
+				},
+			},
+			{
+				// Add a duplicate job.
+				fn:      bal.AddJob,
+				input:   "p2",
+				expDiff: []dax.WorkerDiff{},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2", "p3"},
+					},
+				},
+			},
+		}
+		for i, test := range tests {
+			t.Run(fmt.Sprintf("test-%d", i), func(t *testing.T) {
+				diff, err := test.fn(ctx, newStringWrapper(test.input))
+				assert.NoError(t, err)
+				assert.Equal(t, test.expDiff, diff)
+
+				cs, err := bal.CurrentState(ctx)
+				assert.NoError(t, err)
+				assert.Equal(t, test.expState, cs)
+			})
+		}
+	})
+
+	t.Run("MultipleWorkers", func(t *testing.T) {
+		db, cleanup := newBoltBalancer(t)
+		defer cleanup()
+		bal := boltdb.NewBalancer("test", db, logger.NewStandardLogger(os.Stderr))
+		tests := []struct {
+			fn       func(context.Context, fmt.Stringer) ([]dax.WorkerDiff, error)
+			input    string
+			balance  bool
+			expDiff  []dax.WorkerDiff
+			expState []dax.WorkerInfo
+		}{
+			{
+				// Balance when empty.
+				balance:  true,
+				expDiff:  []dax.WorkerDiff{},
+				expState: []dax.WorkerInfo{},
+			},
+			{
+				// Add worker.
+				fn:      bal.AddWorker,
+				input:   "n2",
+				expDiff: []dax.WorkerDiff{},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{},
+					},
+				},
+			},
+			{
+				// Add worker again.
+				fn:      bal.AddWorker,
+				input:   "n2",
+				expDiff: []dax.WorkerDiff{},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{},
+					},
+				},
+			},
+			{
+				// Add a second worker.
+				fn:      bal.AddWorker,
+				input:   "n1",
+				expDiff: []dax.WorkerDiff{},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{},
+					},
+				},
+			},
+			{
+				// Add job.
+				fn:    bal.AddJob,
+				input: "p2",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n1",
+						AddedJobs:   []dax.Job{"p2"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p2"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{},
+					},
+				},
+			},
+			{
+				// Add job.
+				fn:    bal.AddJob,
+				input: "p3",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n2",
+						AddedJobs:   []dax.Job{"p3"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p2"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p3"},
+					},
+				},
+			},
+			{
+				// Add job.
+				fn:    bal.AddJob,
+				input: "p1",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n1",
+						AddedJobs:   []dax.Job{"p1"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p3"},
+					},
+				},
+			},
+			{
+				// Add a third worker.
+				fn:      bal.AddWorker,
+				input:   "n0",
+				expDiff: []dax.WorkerDiff{},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{},
+					},
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p3"},
+					},
+				},
+			},
+			{
+				// Add job.
+				fn:    bal.AddJob,
+				input: "p4",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n0",
+						AddedJobs:   []dax.Job{"p4"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{"p4"},
+					},
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p3"},
+					},
+				},
+			},
+			{
+				// Add job.
+				fn:    bal.AddJob,
+				input: "p5",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n0",
+						AddedJobs:   []dax.Job{"p5"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{"p4", "p5"},
+					},
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p3"},
+					},
+				},
+			},
+			{
+				// Add job.
+				fn:    bal.AddJob,
+				input: "p0",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n2",
+						AddedJobs:   []dax.Job{"p0"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{"p4", "p5"},
+					},
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p0", "p3"},
+					},
+				},
+			},
+			{
+				// Add job.
+				fn:    bal.AddJob,
+				input: "p6",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n0",
+						AddedJobs:   []dax.Job{"p6"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{"p4", "p5", "p6"},
+					},
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p0", "p3"},
+					},
+				},
+			},
+			{
+				// Add job.
+				fn:    bal.AddJob,
+				input: "p7",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n1",
+						AddedJobs:   []dax.Job{"p7"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{"p4", "p5", "p6"},
+					},
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2", "p7"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p0", "p3"},
+					},
+				},
+			},
+
+			//////////////////// Remove /////////////////////////
+
+			{
+				// Remove nonexistent worker.
+				fn:      bal.RemoveWorker,
+				input:   "nonexistent",
+				expDiff: []dax.WorkerDiff{},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{"p4", "p5", "p6"},
+					},
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2", "p7"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p0", "p3"},
+					},
+				},
+			},
+			{
+				// Remove worker.
+				fn:    bal.RemoveWorker,
+				input: "n1",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n1",
+						AddedJobs:   []dax.Job{},
+						RemovedJobs: []dax.Job{"p1", "p2", "p7"},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{"p4", "p5", "p6"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p0", "p3"},
+					},
+				},
+			},
+
+			{
+				// Remove job (from free list).
+				fn:      bal.RemoveJob,
+				input:   "p2",
+				expDiff: []dax.WorkerDiff{},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{"p4", "p5", "p6"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p0", "p3"},
+					},
+				},
+			},
+
+			{
+				// Balance after remove.
+				balance: true,
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n0",
+						AddedJobs:   []dax.Job{"p7"},
+						RemovedJobs: []dax.Job{},
+					},
+					{
+						WorkerID:    "n2",
+						AddedJobs:   []dax.Job{"p1"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{"p4", "p5", "p6", "p7"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p0", "p1", "p3"},
+					},
+				},
+			},
+
+			{
+				// Remove job.
+				fn:    bal.RemoveJob,
+				input: "p1",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n2",
+						AddedJobs:   []dax.Job{},
+						RemovedJobs: []dax.Job{"p1"},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{"p4", "p5", "p6", "p7"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p0", "p3"},
+					},
+				},
+			},
+		}
+		for i, test := range tests {
+			t.Run(fmt.Sprintf("test-%d", i), func(t *testing.T) {
+				var diff []dax.WorkerDiff
+				var err error
+				if test.balance {
+					diff, err = bal.Balance(ctx)
+				} else {
+					diff, err = test.fn(ctx, newStringWrapper(test.input))
+				}
+				assert.NoError(t, err)
+				assert.Equal(t, test.expDiff, diff)
+
+				cs, err := bal.CurrentState(ctx)
+				assert.NoError(t, err)
+				assert.Equal(t, test.expState, cs)
+			})
+		}
+	})
+
+	t.Run("WorkerState", func(t *testing.T) {
+		db, cleanup := newBoltBalancer(t)
+		defer cleanup()
+		bal := boltdb.NewBalancer("test", db, logger.NewStandardLogger(os.Stderr))
+
+		_, err := bal.AddWorker(ctx, newStringWrapper("n1"))
+		assert.NoError(t, err)
+		_, err = bal.AddJob(ctx, newStringWrapper("p1"))
+		assert.NoError(t, err)
+
+		exp := dax.WorkerInfo{
+			ID:   "n1",
+			Jobs: []dax.Job{"p1"},
+		}
+		ws, err := bal.WorkerState(ctx, "n1")
+		assert.NoError(t, err)
+		assert.Equal(t, exp, ws)
+
+		// Worker doesn't exist.
+		exp = dax.WorkerInfo{
+			ID: "x1",
+		}
+		ws, err = bal.WorkerState(ctx, "x1")
+		assert.NoError(t, err)
+		assert.Equal(t, exp, ws)
+	})
+
+	t.Run("WorkersForJobs", func(t *testing.T) {
+		db, cleanup := newBoltBalancer(t)
+		defer cleanup()
+		bal := boltdb.NewBalancer("test", db, logger.NewStandardLogger(os.Stderr))
+
+		_, err := bal.AddWorker(ctx, newStringWrapper("n1"))
+		assert.NoError(t, err)
+		_, err = bal.AddWorker(ctx, newStringWrapper("n2"))
+		assert.NoError(t, err)
+		for i := 0; i < 12; i++ {
+			_, err = bal.AddJob(ctx, newStringWrapper(fmt.Sprintf("p%d", i)))
+			assert.NoError(t, err)
+		}
+
+		exp := dax.WorkerInfo{
+			ID:   "n1",
+			Jobs: []dax.Job{"p0", "p10", "p2", "p4", "p6", "p8"},
+		}
+		ws, err := bal.WorkerState(ctx, "n1")
+		assert.NoError(t, err)
+		assert.Equal(t, exp, ws)
+
+		exp = dax.WorkerInfo{
+			ID:   "n2",
+			Jobs: []dax.Job{"p1", "p11", "p3", "p5", "p7", "p9"},
+		}
+		ws, err = bal.WorkerState(ctx, "n2")
+		assert.NoError(t, err)
+		assert.Equal(t, exp, ws)
+
+		tests := []struct {
+			jobs []dax.Job
+			exp  []dax.WorkerInfo
+		}{
+			{
+				jobs: []dax.Job{"p0"},
+				exp: []dax.WorkerInfo{
+					{ID: "n1", Jobs: []dax.Job{"p0"}},
+				},
+			},
+			{
+				jobs: []dax.Job{"p0", "p4"},
+				exp: []dax.WorkerInfo{
+					{ID: "n1", Jobs: []dax.Job{"p0", "p4"}},
+				},
+			},
+			{
+				jobs: []dax.Job{"p0", "p4", "p999"},
+				exp: []dax.WorkerInfo{
+					{ID: "n1", Jobs: []dax.Job{"p0", "p4"}},
+				},
+			},
+			{
+				jobs: []dax.Job{"p0", "p1"},
+				exp: []dax.WorkerInfo{
+					{ID: "n1", Jobs: []dax.Job{"p0"}},
+					{ID: "n2", Jobs: []dax.Job{"p1"}},
+				},
+			},
+			{
+				jobs: []dax.Job{"p5", "p0", "p1", "p8"},
+				exp: []dax.WorkerInfo{
+					{ID: "n1", Jobs: []dax.Job{"p0", "p8"}},
+					{ID: "n2", Jobs: []dax.Job{"p1", "p5"}},
+				},
+			},
+		}
+		for i, test := range tests {
+			t.Run(fmt.Sprintf("test-%d", i), func(t *testing.T) {
+				workers, err := bal.WorkersForJobs(ctx, test.jobs)
+				assert.NoError(t, err)
+				assert.Equal(t, test.exp, workers)
+			})
+		}
+
+		// Some tests for WorkersForJobPrefix
+		workers, err := bal.WorkersForJobPrefix(ctx, "p1")
+		assert.NoError(t, err)
+		assert.ElementsMatch(t, []dax.WorkerInfo{
+			{ID: "n1", Jobs: []dax.Job{"p10"}},
+			{ID: "n2", Jobs: []dax.Job{"p1", "p11"}},
+		}, workers)
+
+		workers, err = bal.WorkersForJobPrefix(ctx, "p2")
+		assert.NoError(t, err)
+		assert.ElementsMatch(t, []dax.WorkerInfo{
+			{ID: "n1", Jobs: []dax.Job{"p2"}},
+		}, workers)
+
+		workers, err = bal.WorkersForJobPrefix(ctx, "pp")
+		assert.NoError(t, err)
+		assert.ElementsMatch(t, []dax.WorkerInfo{}, workers)
+
+	})
+
+	t.Run("Balance", func(t *testing.T) {
+		db, cleanup := newBoltBalancer(t)
+		defer cleanup()
+		bal := boltdb.NewBalancer("test", db, logger.NewStandardLogger(os.Stderr))
+
+		// Add two workers with some jobs evenly spread across them.
+		_, err := bal.AddWorker(ctx, newStringWrapper("n1"))
+		assert.NoError(t, err)
+		_, err = bal.AddWorker(ctx, newStringWrapper("n2"))
+		assert.NoError(t, err)
+		for i := 0; i < 13; i++ {
+			_, err = bal.AddJob(ctx, newStringWrapper(fmt.Sprintf("p%d", i)))
+			assert.NoError(t, err)
+		}
+
+		exp := dax.WorkerInfo{
+			ID:   "n1",
+			Jobs: []dax.Job{"p0", "p10", "p12", "p2", "p4", "p6", "p8"},
+		}
+		ws, err := bal.WorkerState(ctx, "n1")
+		assert.NoError(t, err)
+		assert.Equal(t, exp, ws)
+
+		exp = dax.WorkerInfo{
+			ID:   "n2",
+			Jobs: []dax.Job{"p1", "p11", "p3", "p5", "p7", "p9"},
+		}
+		ws, err = bal.WorkerState(ctx, "n2")
+		assert.NoError(t, err)
+		assert.Equal(t, exp, ws)
+
+		// Now, add a worker and confirm that it currently has no jobs assigned
+		// to it.
+		_, err = bal.AddWorker(ctx, newStringWrapper("n3"))
+		assert.NoError(t, err)
+		exp = dax.WorkerInfo{
+			ID:   "n3",
+			Jobs: []dax.Job{},
+		}
+		ws, err = bal.WorkerState(ctx, "n3")
+		assert.NoError(t, err)
+		assert.Equal(t, exp, ws)
+
+		// Finally, call Balance() and confirm that the appropriate jobs got
+		// reassigned.
+		_, err = bal.Balance(ctx)
+		assert.NoError(t, err)
+
+		exp = dax.WorkerInfo{
+			ID:   "n1",
+			Jobs: []dax.Job{"p0", "p10", "p12", "p2", "p4"},
+		}
+		ws, err = bal.WorkerState(ctx, "n1")
+		assert.NoError(t, err)
+		assert.Equal(t, exp, ws)
+
+		exp = dax.WorkerInfo{
+			ID:   "n2",
+			Jobs: []dax.Job{"p1", "p11", "p3", "p5"},
+		}
+		ws, err = bal.WorkerState(ctx, "n2")
+		assert.NoError(t, err)
+		assert.Equal(t, exp, ws)
+
+		exp = dax.WorkerInfo{
+			ID:   "n3",
+			Jobs: []dax.Job{"p6", "p7", "p8", "p9"},
+		}
+		ws, err = bal.WorkerState(ctx, "n3")
+		assert.NoError(t, err)
+		assert.Equal(t, exp, ws)
+	})
+}
+
+type stringWrapper struct {
+	s string
+}
+
+func newStringWrapper(s string) *stringWrapper {
+	return &stringWrapper{
+		s: s,
+	}
+}
+
+func (s *stringWrapper) String() string {
+	return s.s
+}

dax/mds/controller/naive/boltdb/balancer.go

@@ -0,0 +1,515 @@
+// Package boltdb contains the boltdb implementation of the Balancer interface.
+package boltdb
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+	"strings"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/boltdb"
+	"github.com/molecula/featurebase/v3/dax/mds/controller"
+	"github.com/molecula/featurebase/v3/dax/mds/controller/naive"
+	"github.com/molecula/featurebase/v3/errors"
+	"github.com/molecula/featurebase/v3/logger"
+)
+
+var (
+	bucketNaiveBalancer = boltdb.Bucket("naiveBalancer")
+)
+
+// NaiveBalancerBuckets defines the buckets used by this package. It can be
+// called during setup to create the buckets ahead of time.
+var NaiveBalancerBuckets []boltdb.Bucket = []boltdb.Bucket{
+	bucketNaiveBalancer,
+}
+
+// NewBalancer returns a new instance of controller.Balancer.
+func NewBalancer(name string, db *boltdb.DB, logger logger.Logger) controller.Balancer {
+	fjs := newFreeJobService(db)
+	wjs := newWorkerJobService(db, logger)
+
+	return naive.New(name, fjs, wjs, logger)
+}
+
+// Ensure type implements interface.
+var _ naive.WorkerJobService = (*workerJobService)(nil)
+
+type workerJobService struct {
+	db     *boltdb.DB
+	logger logger.Logger
+}
+
+func newWorkerJobService(db *boltdb.DB, logger logger.Logger) *workerJobService {
+	return &workerJobService{
+		db:     db,
+		logger: logger,
+	}
+}
+
+func (w *workerJobService) WorkersJobs(ctx context.Context, balancerName string) ([]dax.WorkerInfo, error) {
+	tx, err := w.db.BeginTx(ctx, false)
+	if err != nil {
+		return nil, errors.Wrap(err, "getting tx")
+	}
+	defer tx.Rollback()
+
+	workerInfos, err := getWorkerInfos(ctx, tx, balancerName)
+	if err != nil {
+		return nil, errors.Wrapf(err, "getting worker infos: %s", balancerName)
+	}
+
+	return workerInfos, nil
+}
+
+func (w *workerJobService) WorkerCount(ctx context.Context, balancerName string) (int, error) {
+	tx, err := w.db.BeginTx(ctx, false)
+	if err != nil {
+		return 0, errors.Wrap(err, "getting tx")
+	}
+	defer tx.Rollback()
+
+	workers, err := w.getWorkers(ctx, tx, balancerName)
+	if err != nil {
+		return 0, errors.Wrapf(err, "getting workers: %s", balancerName)
+	}
+
+	return len(workers), nil
+}
+
+func (w *workerJobService) ListWorkers(ctx context.Context, balancerName string) (dax.Workers, error) {
+	tx, err := w.db.BeginTx(ctx, false)
+	if err != nil {
+		return nil, errors.Wrap(err, "beginning tx")
+	}
+	defer tx.Rollback()
+
+	workers, err := w.getWorkers(ctx, tx, balancerName)
+	if err != nil {
+		return nil, errors.Wrapf(err, "getting workers: %s", balancerName)
+	}
+
+	return workers, nil
+}
+
+func (w *workerJobService) getWorkers(ctx context.Context, tx *boltdb.Tx, balancerName string) (dax.Workers, error) {
+	c := tx.Bucket(bucketNaiveBalancer).Cursor()
+
+	// Deserialize rows into Worker objects.
+	workers := make(dax.Workers, 0)
+
+	prefix := []byte(fmt.Sprintf(prefixFmtWorkers, balancerName))
+	for k, v := c.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, v = c.Next() {
+		if v == nil {
+			w.logger.Printf("nil value for key: %s", k)
+			continue
+		}
+
+		worker, err := keyWorker(k)
+		if err != nil {
+			return nil, errors.Wrapf(err, "getting worker from key: %v", k)
+		}
+
+		workers = append(workers, worker)
+	}
+
+	return workers, nil
+}
+
+func getWorkerInfos(ctx context.Context, tx *boltdb.Tx, balancerName string) (dax.WorkerInfos, error) {
+	c := tx.Bucket(bucketNaiveBalancer).Cursor()
+
+	// Deserialize rows into WorkerInfo objects.
+	workerInfos := make(dax.WorkerInfos, 0)
+
+	prefix := []byte(fmt.Sprintf(prefixFmtWorkers, balancerName))
+	for k, v := c.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, v = c.Next() {
+		worker, err := keyWorker(k)
+		if err != nil {
+			return nil, errors.Wrapf(err, "getting worker from key: %v", k)
+		}
+
+		jobs := dax.NewSet[dax.Job]()
+		if v != nil {
+			jobs, err = decodeJobSet(v)
+			if err != nil {
+				return nil, errors.Wrap(err, "decoding job set")
+			}
+		}
+
+		workerInfo := dax.WorkerInfo{
+			ID:   worker,
+			Jobs: jobs.Sorted(),
+		}
+
+		workerInfos = append(workerInfos, workerInfo)
+	}
+
+	return workerInfos, nil
+}
+
+func (w *workerJobService) WorkerExists(ctx context.Context, balancerName string, worker dax.Worker) (bool, error) {
+	tx, err := w.db.BeginTx(ctx, false)
+	if err != nil {
+		return false, errors.Wrapf(err, "getting tx: %s", balancerName)
+	}
+	defer tx.Rollback()
+
+	bkt := tx.Bucket(bucketNaiveBalancer)
+	if bkt == nil {
+		return false, errors.Errorf(boltdb.ErrFmtBucketNotFound, bucketNaiveBalancer)
+	}
+
+	wrkr := bkt.Get(workerKey(balancerName, worker))
+
+	return wrkr != nil, nil
+}
+
+func (w *workerJobService) CreateWorker(ctx context.Context, balancerName string, worker dax.Worker) error {
+	tx, err := w.db.BeginTx(ctx, true)
+	if err != nil {
+		return errors.Wrap(err, "getting transaction")
+	}
+	defer tx.Rollback()
+
+	bkt := tx.Bucket(bucketNaiveBalancer)
+	if bkt == nil {
+		return errors.Errorf(boltdb.ErrFmtBucketNotFound, bucketNaiveBalancer)
+	}
+
+	// If this worker already exists, don't do anything.
+	wrkr := bkt.Get(workerKey(balancerName, worker))
+	if wrkr != nil {
+		return nil
+	}
+
+	val := []byte("[]")
+	if err := bkt.Put(workerKey(balancerName, worker), val); err != nil {
+		return errors.Wrap(err, "putting worker")
+	}
+
+	return tx.Commit()
+}
+
+func (w *workerJobService) DeleteWorker(ctx context.Context, balancerName string, worker dax.Worker) error {
+	tx, err := w.db.BeginTx(ctx, true)
+	if err != nil {
+		return errors.Wrap(err, "beginning tx")
+	}
+	defer tx.Rollback()
+
+	bkt := tx.Bucket(bucketNaiveBalancer)
+	if bkt == nil {
+		return errors.Errorf(boltdb.ErrFmtBucketNotFound, bucketNaiveBalancer)
+	}
+
+	if err := bkt.Delete(workerKey(balancerName, worker)); err != nil {
+		return errors.Wrapf(err, "deleting node key: %s", workerKey(balancerName, worker))
+	}
+
+	return tx.Commit()
+}
+
+func (w *workerJobService) CreateJob(ctx context.Context, balancerName string, worker dax.Worker, job dax.Job) error {
+	tx, err := w.db.BeginTx(ctx, true)
+	if err != nil {
+		return errors.Wrap(err, "beginning tx")
+	}
+	defer tx.Rollback()
+
+	bkt := tx.Bucket(bucketNaiveBalancer)
+	if bkt == nil {
+		return errors.Errorf(boltdb.ErrFmtBucketNotFound, bucketNaiveBalancer)
+	}
+
+	jobset := dax.NewSet[dax.Job]()
+
+	// get worker
+	wrkr := bkt.Get(workerKey(balancerName, worker))
+	if wrkr != nil {
+		jobset, err = decodeJobSet(wrkr)
+		if err != nil {
+			return errors.Wrap(err, "decoding job set")
+		}
+	}
+
+	jobset.Add(job)
+	val, err := encodeJobSet(jobset)
+	if err != nil {
+		return errors.Wrap(err, "encoding job set")
+	}
+
+	if err := bkt.Put(workerKey(balancerName, worker), val); err != nil {
+		return errors.Wrap(err, "putting worker")
+	}
+
+	return tx.Commit()
+}
+
+func (w *workerJobService) DeleteJob(ctx context.Context, balancerName string, worker dax.Worker, job dax.Job) error {
+	tx, err := w.db.BeginTx(ctx, true)
+	if err != nil {
+		return errors.Wrap(err, "beginning tx")
+	}
+	defer tx.Rollback()
+
+	bkt := tx.Bucket(bucketNaiveBalancer)
+	if bkt == nil {
+		return errors.Errorf(boltdb.ErrFmtBucketNotFound, bucketNaiveBalancer)
+	}
+
+	// get worker
+	wrkr := bkt.Get(workerKey(balancerName, worker))
+	if wrkr == nil {
+		return nil
+	}
+
+	jobset, err := decodeJobSet(wrkr)
+	if err != nil {
+		return errors.Wrap(err, "decoding job set")
+	}
+	if !jobset.Contains(job) {
+		return nil
+	}
+
+	jobset.Remove(job)
+	val, err := encodeJobSet(jobset)
+	if err != nil {
+		return errors.Wrap(err, "encoding job set")
+	}
+
+	if err := bkt.Put(workerKey(balancerName, worker), val); err != nil {
+		return errors.Wrap(err, "putting worker")
+	}
+
+	return tx.Commit()
+}
+
+func (w *workerJobService) ListJobs(ctx context.Context, balancerName string, worker dax.Worker) (dax.Jobs, error) {
+	tx, err := w.db.BeginTx(ctx, false)
+	if err != nil {
+		return nil, errors.Wrap(err, "beginning tx")
+	}
+	defer tx.Rollback()
+
+	bkt := tx.Bucket(bucketNaiveBalancer)
+	if bkt == nil {
+		return nil, errors.Errorf(boltdb.ErrFmtBucketNotFound, bucketNaiveBalancer)
+	}
+
+	jobset := dax.NewSet[dax.Job]()
+
+	// get worker
+	wrkr := bkt.Get(workerKey(balancerName, worker))
+	if wrkr != nil {
+		jobset, err = decodeJobSet(wrkr)
+		if err != nil {
+			return nil, errors.Wrap(err, "decoding job set")
+		}
+	}
+
+	return jobset.Sorted(), nil
+}
+
+func (w *workerJobService) JobCount(ctx context.Context, balancerName string, worker dax.Worker) (int, error) {
+	tx, err := w.db.BeginTx(ctx, false)
+	if err != nil {
+		return 0, errors.Wrapf(err, "getting tx: %s", balancerName)
+	}
+	defer tx.Rollback()
+
+	bkt := tx.Bucket(bucketNaiveBalancer)
+	if bkt == nil {
+		return 0, errors.Errorf(boltdb.ErrFmtBucketNotFound, bucketNaiveBalancer)
+	}
+
+	jobset := dax.NewSet[dax.Job]()
+
+	// get worker
+	wrkr := bkt.Get(workerKey(balancerName, worker))
+	if wrkr != nil {
+		jobset, err = decodeJobSet(wrkr)
+		if err != nil {
+			return 0, errors.Wrap(err, "decoding job set")
+		}
+	}
+
+	return len(jobset), nil
+}
+
+// encodeJobSet encode the jobSet into a JSON array of strings.
+func encodeJobSet(jobSet dax.Set[dax.Job]) ([]byte, error) {
+	arr := jobSet.Sorted()
+	b, err := json.Marshal(arr)
+	if err != nil {
+		return nil, errors.Wrap(err, "marshalling json")
+	}
+	return b, nil
+}
+
+// decodeJobSet decode the string (a JSON array of strings) into jobSet.
+func decodeJobSet(v []byte) (dax.Set[dax.Job], error) {
+	var arr []string
+	err := json.Unmarshal(v, &arr)
+	if err != nil {
+		return nil, errors.Wrap(err, "unmarshalling json")
+	}
+
+	js := dax.NewSet[dax.Job]()
+	for _, s := range arr {
+		js.Add(dax.Job(s))
+	}
+
+	return js, nil
+}
+
+// Ensure type implements interface.
+var _ naive.FreeJobService = (*freeJobService)(nil)
+
+type freeJobService struct {
+	db *boltdb.DB
+}
+
+func newFreeJobService(db *boltdb.DB) *freeJobService {
+	return &freeJobService{
+		db: db,
+	}
+}
+
+func (f *freeJobService) CreateFreeJob(ctx context.Context, balancerName string, job dax.Job) error {
+	return f.MergeFreeJobs(ctx, balancerName, dax.Jobs{job})
+}
+
+func (f *freeJobService) DeleteFreeJob(ctx context.Context, balancerName string, job dax.Job) error {
+	tx, err := f.db.BeginTx(ctx, true)
+	if err != nil {
+		return errors.Wrap(err, "beginning tx")
+	}
+	defer tx.Rollback()
+
+	bkt := tx.Bucket(bucketNaiveBalancer)
+	if bkt == nil {
+		return errors.Errorf(boltdb.ErrFmtBucketNotFound, bucketNaiveBalancer)
+	}
+
+	// get free jobs
+	fjs := bkt.Get(freeJobKey(balancerName))
+	if fjs == nil {
+		return nil
+	}
+
+	jobset, err := decodeJobSet(fjs)
+	if err != nil {
+		return errors.Wrap(err, "decoding job set")
+	}
+	if !jobset.Contains(job) {
+		return nil
+	}
+
+	jobset.Remove(job)
+	val, err := encodeJobSet(jobset)
+	if err != nil {
+		return errors.Wrap(err, "encoding job set")
+	}
+
+	if err := bkt.Put(freeJobKey(balancerName), val); err != nil {
+		return errors.Wrap(err, "putting free job")
+	}
+
+	return tx.Commit()
+}
+
+func (f *freeJobService) ListFreeJobs(ctx context.Context, balancerName string) (dax.Jobs, error) {
+	tx, err := f.db.BeginTx(ctx, false)
+	if err != nil {
+		return nil, errors.Wrap(err, "beginning tx")
+	}
+	defer tx.Rollback()
+
+	bkt := tx.Bucket(bucketNaiveBalancer)
+	if bkt == nil {
+		return nil, errors.Errorf(boltdb.ErrFmtBucketNotFound, bucketNaiveBalancer)
+	}
+
+	jobset := dax.NewSet[dax.Job]()
+
+	// get free jobs
+	fjs := bkt.Get(freeJobKey(balancerName))
+	if fjs != nil {
+		jobset, err = decodeJobSet(fjs)
+		if err != nil {
+			return nil, errors.Wrap(err, "decoding job set")
+		}
+	}
+
+	return jobset.Sorted(), nil
+}
+
+func (f *freeJobService) MergeFreeJobs(ctx context.Context, balancerName string, jobs dax.Jobs) error {
+	tx, err := f.db.BeginTx(ctx, true)
+	if err != nil {
+		return err
+	}
+	defer tx.Rollback()
+
+	bkt := tx.Bucket(bucketNaiveBalancer)
+	if bkt == nil {
+		return errors.Errorf(boltdb.ErrFmtBucketNotFound, bucketNaiveBalancer)
+	}
+
+	jobset := dax.NewSet[dax.Job]()
+
+	// get free jobs
+	fjs := bkt.Get(freeJobKey(balancerName))
+	if fjs != nil {
+		jobset, err = decodeJobSet(fjs)
+		if err != nil {
+			return errors.Wrap(err, "decoding job set")
+		}
+	}
+
+	for _, j := range jobs {
+		jobset.Add(j)
+	}
+	val, err := encodeJobSet(jobset)
+	if err != nil {
+		return errors.Wrap(err, "encoding job set")
+	}
+
+	if err := bkt.Put(freeJobKey(balancerName), val); err != nil {
+		return errors.Wrap(err, "putting free job")
+	}
+
+	return tx.Commit()
+}
+
+//////////////////////////////////////////////////////
+
+const (
+	prefixFmtWorkers  = "workers/%s/" // %s - balancerName
+	prefixFmtFreeJobs = "freejobs/%s" // %s - balancerName
+)
+
+// workerKey returns a key based on worker.
+func workerKey(bal string, worker dax.Worker) []byte {
+	key := fmt.Sprintf(prefixFmtWorkers+"%s", bal, worker)
+	return []byte(key)
+}
+
+// keyWorker gets the worker out of the key.
+func keyWorker(key []byte) (dax.Worker, error) {
+	parts := strings.Split(string(key), "/")
+	if len(parts) != 3 {
+		return "", errors.New(errors.ErrUncoded, "worker key format expected: `workers/balancer/worker`")
+	}
+
+	return dax.Worker(parts[2]), nil
+}
+
+// freeJobKey returns a key for all freeJobs.
+func freeJobKey(bal string) []byte {
+	key := fmt.Sprintf(prefixFmtFreeJobs, bal)
+	return []byte(key)
+}

dax/mds/controller/naive/boltdb/balancer_test.go

@@ -0,0 +1,709 @@
+package boltdb_test
+
+import (
+	"context"
+	"fmt"
+	"testing"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/mds/controller/naive/boltdb"
+	testbolt "github.com/molecula/featurebase/v3/dax/test/boltdb"
+	"github.com/molecula/featurebase/v3/logger"
+	"github.com/stretchr/testify/assert"
+)
+
+func TestBalancer(t *testing.T) {
+	db := testbolt.MustOpenDB(t)
+	defer testbolt.MustCloseDB(t, db)
+
+	t.Cleanup(func() {
+		testbolt.CleanupDB(t, db.Path())
+	})
+
+	ctx := context.Background()
+
+	// Initialize the buckets.
+	assert.NoError(t, db.InitializeBuckets(boltdb.NaiveBalancerBuckets...))
+
+	t.Run("SingleWorker", func(t *testing.T) {
+		bal := boltdb.NewBalancer("test-single-worker", db, logger.NopLogger)
+		tests := []struct {
+			fn       func(context.Context, fmt.Stringer) ([]dax.WorkerDiff, error)
+			input    string
+			expDiff  []dax.WorkerDiff
+			expState []dax.WorkerInfo
+		}{
+			{
+				// Add job.
+				fn:       bal.AddJob,
+				input:    "p2",
+				expDiff:  []dax.WorkerDiff{},
+				expState: []dax.WorkerInfo{},
+			},
+			{
+				// Add worker.
+				fn:    bal.AddWorker,
+				input: "n1",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n1",
+						AddedJobs:   []dax.Job{"p2"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p2"},
+					},
+				},
+			},
+			{
+				// Add another job out of order.
+				fn:    bal.AddJob,
+				input: "p1",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n1",
+						AddedJobs:   []dax.Job{"p1"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2"},
+					},
+				},
+			},
+			{
+				// Add another job.
+				fn:    bal.AddJob,
+				input: "p3",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n1",
+						AddedJobs:   []dax.Job{"p3"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2", "p3"},
+					},
+				},
+			},
+			{
+				// Add a duplicate job.
+				fn:      bal.AddJob,
+				input:   "p2",
+				expDiff: []dax.WorkerDiff{},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2", "p3"},
+					},
+				},
+			},
+		}
+		for i, test := range tests {
+			t.Run(fmt.Sprintf("test-%d", i), func(t *testing.T) {
+				diff, err := test.fn(ctx, newStringWrapper(test.input))
+				assert.NoError(t, err)
+				assert.Equal(t, test.expDiff, diff)
+
+				cs, err := bal.CurrentState(ctx)
+				assert.NoError(t, err)
+				assert.Equal(t, test.expState, cs)
+			})
+		}
+	})
+
+	t.Run("MultipleWorkers", func(t *testing.T) {
+		bal := boltdb.NewBalancer("test-multiple-workers", db, logger.NopLogger)
+		tests := []struct {
+			fn       func(context.Context, fmt.Stringer) ([]dax.WorkerDiff, error)
+			input    string
+			balance  bool
+			expDiff  []dax.WorkerDiff
+			expState []dax.WorkerInfo
+		}{
+			{
+				// Balance when empty.
+				balance:  true,
+				expDiff:  []dax.WorkerDiff{},
+				expState: []dax.WorkerInfo{},
+			},
+			{
+				// Add worker.
+				fn:      bal.AddWorker,
+				input:   "n2",
+				expDiff: []dax.WorkerDiff{},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{},
+					},
+				},
+			},
+			{
+				// Add worker again.
+				fn:      bal.AddWorker,
+				input:   "n2",
+				expDiff: []dax.WorkerDiff{},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{},
+					},
+				},
+			},
+			{
+				// Add a second worker.
+				fn:      bal.AddWorker,
+				input:   "n1",
+				expDiff: []dax.WorkerDiff{},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{},
+					},
+				},
+			},
+			{
+				// Add job.
+				fn:    bal.AddJob,
+				input: "p2",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n1",
+						AddedJobs:   []dax.Job{"p2"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p2"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{},
+					},
+				},
+			},
+			{
+				// Add job.
+				fn:    bal.AddJob,
+				input: "p3",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n2",
+						AddedJobs:   []dax.Job{"p3"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p2"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p3"},
+					},
+				},
+			},
+			{
+				// Add job.
+				fn:    bal.AddJob,
+				input: "p1",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n1",
+						AddedJobs:   []dax.Job{"p1"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p3"},
+					},
+				},
+			},
+			{
+				// Add a third worker.
+				fn:      bal.AddWorker,
+				input:   "n0",
+				expDiff: []dax.WorkerDiff{},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{},
+					},
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p3"},
+					},
+				},
+			},
+			{
+				// Add job.
+				fn:    bal.AddJob,
+				input: "p4",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n0",
+						AddedJobs:   []dax.Job{"p4"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{"p4"},
+					},
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p3"},
+					},
+				},
+			},
+			{
+				// Add job.
+				fn:    bal.AddJob,
+				input: "p5",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n0",
+						AddedJobs:   []dax.Job{"p5"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{"p4", "p5"},
+					},
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p3"},
+					},
+				},
+			},
+			{
+				// Add job.
+				fn:    bal.AddJob,
+				input: "p0",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n2",
+						AddedJobs:   []dax.Job{"p0"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{"p4", "p5"},
+					},
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p0", "p3"},
+					},
+				},
+			},
+			{
+				// Add job.
+				fn:    bal.AddJob,
+				input: "p6",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n0",
+						AddedJobs:   []dax.Job{"p6"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{"p4", "p5", "p6"},
+					},
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p0", "p3"},
+					},
+				},
+			},
+			{
+				// Add job.
+				fn:    bal.AddJob,
+				input: "p7",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n1",
+						AddedJobs:   []dax.Job{"p7"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{"p4", "p5", "p6"},
+					},
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2", "p7"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p0", "p3"},
+					},
+				},
+			},
+
+			//////////////////// Remove /////////////////////////
+
+			{
+				// Remove nonexistent worker.
+				fn:      bal.RemoveWorker,
+				input:   "nonexistent",
+				expDiff: []dax.WorkerDiff{},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{"p4", "p5", "p6"},
+					},
+					{
+						ID:   "n1",
+						Jobs: []dax.Job{"p1", "p2", "p7"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p0", "p3"},
+					},
+				},
+			},
+			{
+				// Remove worker.
+				fn:    bal.RemoveWorker,
+				input: "n1",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n1",
+						AddedJobs:   []dax.Job{},
+						RemovedJobs: []dax.Job{"p1", "p2", "p7"},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{"p4", "p5", "p6"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p0", "p3"},
+					},
+				},
+			},
+
+			{
+				// Remove job (from free list).
+				fn:      bal.RemoveJob,
+				input:   "p2",
+				expDiff: []dax.WorkerDiff{},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{"p4", "p5", "p6"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p0", "p3"},
+					},
+				},
+			},
+
+			{
+				// Balance after remove.
+				balance: true,
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n0",
+						AddedJobs:   []dax.Job{"p7"},
+						RemovedJobs: []dax.Job{},
+					},
+					{
+						WorkerID:    "n2",
+						AddedJobs:   []dax.Job{"p1"},
+						RemovedJobs: []dax.Job{},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{"p4", "p5", "p6", "p7"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p0", "p1", "p3"},
+					},
+				},
+			},
+
+			{
+				// Remove job.
+				fn:    bal.RemoveJob,
+				input: "p1",
+				expDiff: []dax.WorkerDiff{
+					{
+						WorkerID:    "n2",
+						AddedJobs:   []dax.Job{},
+						RemovedJobs: []dax.Job{"p1"},
+					},
+				},
+				expState: []dax.WorkerInfo{
+					{
+						ID:   "n0",
+						Jobs: []dax.Job{"p4", "p5", "p6", "p7"},
+					},
+					{
+						ID:   "n2",
+						Jobs: []dax.Job{"p0", "p3"},
+					},
+				},
+			},
+		}
+		for i, test := range tests {
+			t.Run(fmt.Sprintf("test-%d", i), func(t *testing.T) {
+				var diff []dax.WorkerDiff
+				var err error
+				if test.balance {
+					diff, err = bal.Balance(ctx)
+				} else {
+					diff, err = test.fn(ctx, newStringWrapper(test.input))
+				}
+				assert.NoError(t, err)
+				assert.Equal(t, test.expDiff, diff)
+
+				cs, err := bal.CurrentState(ctx)
+				assert.NoError(t, err)
+				assert.Equal(t, test.expState, cs)
+			})
+		}
+	})
+
+	t.Run("WorkerState", func(t *testing.T) {
+		bal := boltdb.NewBalancer("test-worker-state", db, logger.NopLogger)
+
+		_, err := bal.AddWorker(ctx, newStringWrapper("n1"))
+		assert.NoError(t, err)
+		_, err = bal.AddJob(ctx, newStringWrapper("p1"))
+		assert.NoError(t, err)
+
+		exp := dax.WorkerInfo{
+			ID:   "n1",
+			Jobs: []dax.Job{"p1"},
+		}
+		ws, err := bal.WorkerState(ctx, "n1")
+		assert.NoError(t, err)
+		assert.Equal(t, exp, ws)
+
+		// Worker doesn't exist.
+		exp = dax.WorkerInfo{
+			ID: "x1",
+		}
+		ws, err = bal.WorkerState(ctx, "x1")
+		assert.NoError(t, err)
+		assert.Equal(t, exp, ws)
+	})
+
+	t.Run("WorkersForJobs", func(t *testing.T) {
+		bal := boltdb.NewBalancer("test-workers-for-jobs", db, logger.NopLogger)
+
+		_, err := bal.AddWorker(ctx, newStringWrapper("n1"))
+		assert.NoError(t, err)
+		_, err = bal.AddWorker(ctx, newStringWrapper("n2"))
+		assert.NoError(t, err)
+		for i := 0; i < 12; i++ {
+			_, err = bal.AddJob(ctx, newStringWrapper(fmt.Sprintf("p%d", i)))
+			assert.NoError(t, err)
+		}
+
+		exp := dax.WorkerInfo{
+			ID:   "n1",
+			Jobs: []dax.Job{"p0", "p10", "p2", "p4", "p6", "p8"},
+		}
+		ws, err := bal.WorkerState(ctx, "n1")
+		assert.NoError(t, err)
+		assert.Equal(t, exp, ws)
+
+		exp = dax.WorkerInfo{
+			ID:   "n2",
+			Jobs: []dax.Job{"p1", "p11", "p3", "p5", "p7", "p9"},
+		}
+		ws, err = bal.WorkerState(ctx, "n2")
+		assert.NoError(t, err)
+		assert.Equal(t, exp, ws)
+
+		tests := []struct {
+			jobs []dax.Job
+			exp  []dax.WorkerInfo
+		}{
+			{
+				jobs: []dax.Job{"p0"},
+				exp: []dax.WorkerInfo{
+					{ID: "n1", Jobs: []dax.Job{"p0"}},
+				},
+			},
+			{
+				jobs: []dax.Job{"p0", "p4"},
+				exp: []dax.WorkerInfo{
+					{ID: "n1", Jobs: []dax.Job{"p0", "p4"}},
+				},
+			},
+			{
+				jobs: []dax.Job{"p0", "p4", "p999"},
+				exp: []dax.WorkerInfo{
+					{ID: "n1", Jobs: []dax.Job{"p0", "p4"}},
+				},
+			},
+			{
+				jobs: []dax.Job{"p0", "p1"},
+				exp: []dax.WorkerInfo{
+					{ID: "n1", Jobs: []dax.Job{"p0"}},
+					{ID: "n2", Jobs: []dax.Job{"p1"}},
+				},
+			},
+			{
+				jobs: []dax.Job{"p5", "p0", "p1", "p8"},
+				exp: []dax.WorkerInfo{
+					{ID: "n1", Jobs: []dax.Job{"p0", "p8"}},
+					{ID: "n2", Jobs: []dax.Job{"p1", "p5"}},
+				},
+			},
+		}
+		for i, test := range tests {
+			t.Run(fmt.Sprintf("test-%d", i), func(t *testing.T) {
+				workers, err := bal.WorkersForJobs(ctx, test.jobs)
+				assert.NoError(t, err)
+				assert.Equal(t, test.exp, workers)
+			})
+		}
+	})
+
+	t.Run("Balance", func(t *testing.T) {
+		bal := boltdb.NewBalancer("test-balance", db, logger.NopLogger)
+
+		// Add two workers with some jobs evenly spread across them.
+		_, err := bal.AddWorker(ctx, newStringWrapper("n1"))
+		assert.NoError(t, err)
+		_, err = bal.AddWorker(ctx, newStringWrapper("n2"))
+		assert.NoError(t, err)
+		for i := 0; i < 13; i++ {
+			_, err = bal.AddJob(ctx, newStringWrapper(fmt.Sprintf("p%d", i)))
+			assert.NoError(t, err)
+		}
+
+		exp := dax.WorkerInfo{
+			ID:   "n1",
+			Jobs: []dax.Job{"p0", "p10", "p12", "p2", "p4", "p6", "p8"},
+		}
+		ws, err := bal.WorkerState(ctx, "n1")
+		assert.NoError(t, err)
+		assert.Equal(t, exp, ws)
+
+		exp = dax.WorkerInfo{
+			ID:   "n2",
+			Jobs: []dax.Job{"p1", "p11", "p3", "p5", "p7", "p9"},
+		}
+		ws, err = bal.WorkerState(ctx, "n2")
+		assert.NoError(t, err)
+		assert.Equal(t, exp, ws)
+
+		// Now, add a worker and confirm that it currently has no jobs assigned
+		// to it.
+		_, err = bal.AddWorker(ctx, newStringWrapper("n3"))
+		assert.NoError(t, err)
+		exp = dax.WorkerInfo{
+			ID:   "n3",
+			Jobs: []dax.Job{},
+		}
+		ws, err = bal.WorkerState(ctx, "n3")
+		assert.NoError(t, err)
+		assert.Equal(t, exp, ws)
+
+		// Finally, call Balance() and confirm that the appropriate jobs got
+		// reassigned.
+		_, err = bal.Balance(ctx)
+		assert.NoError(t, err)
+
+		exp = dax.WorkerInfo{
+			ID:   "n1",
+			Jobs: []dax.Job{"p0", "p10", "p12", "p2", "p4"},
+		}
+		ws, err = bal.WorkerState(ctx, "n1")
+		assert.NoError(t, err)
+		assert.Equal(t, exp, ws)
+
+		exp = dax.WorkerInfo{
+			ID:   "n2",
+			Jobs: []dax.Job{"p1", "p11", "p3", "p5"},
+		}
+		ws, err = bal.WorkerState(ctx, "n2")
+		assert.NoError(t, err)
+		assert.Equal(t, exp, ws)
+
+		exp = dax.WorkerInfo{
+			ID:   "n3",
+			Jobs: []dax.Job{"p6", "p7", "p8", "p9"},
+		}
+		ws, err = bal.WorkerState(ctx, "n3")
+		assert.NoError(t, err)
+		assert.Equal(t, exp, ws)
+	})
+}
+
+type stringWrapper struct {
+	s string
+}
+
+func newStringWrapper(s string) *stringWrapper {
+	return &stringWrapper{
+		s: s,
+	}
+}
+
+func (s *stringWrapper) String() string {
+	return s.s
+}

dax/mds/controller/naive/types.go

@@ -0,0 +1,81 @@
+package naive
+
+import (
+	"sort"
+
+	"github.com/molecula/featurebase/v3/dax"
+)
+
+// jobSetDiffs is used internally to capture the diffs as they're happening. We
+// call output() to generate the final result.
+type jobSetDiffs struct {
+	added   dax.Set[dax.Job]
+	removed dax.Set[dax.Job]
+}
+
+func newJobSetDiffs() jobSetDiffs {
+	return jobSetDiffs{
+		added:   dax.NewSet[dax.Job](),
+		removed: dax.NewSet[dax.Job](),
+	}
+}
+
+type internalDiffs map[dax.Worker]jobSetDiffs
+
+func newInternalDiffs() internalDiffs {
+	return make(internalDiffs)
+}
+
+func (d internalDiffs) added(worker dax.Worker, job dax.Job) {
+	if _, ok := d[worker]; !ok {
+		d[worker] = newJobSetDiffs()
+	}
+
+	// Before adding the job, make sure we haven't indicated that it has been
+	// removed prior to this. If it has, we need to invalidate that "remove"
+	// instruction.
+	d[worker].removed.Remove(job)
+
+	d[worker].added.Add(job)
+}
+
+func (d internalDiffs) removed(worker dax.Worker, job dax.Job) {
+	if _, ok := d[worker]; !ok {
+		d[worker] = newJobSetDiffs()
+	}
+
+	// Before removing the job, make sure we haven't indicated that it has been
+	// added prior to this. If it has, we need to invalidate that "add"
+	// instruction.
+	d[worker].added.Remove(job)
+
+	d[worker].removed.Add(job)
+}
+
+func (d internalDiffs) merge(d2 internalDiffs) {
+	for k, v := range d2 {
+		if _, ok := d[k]; !ok {
+			d[k] = newJobSetDiffs()
+		}
+		d[k].added.Merge(v.added)
+		d[k].removed.Merge(v.removed)
+	}
+}
+
+// output converts internalDiff to []controller.WorkerDiff for external
+// consumption.
+func (d internalDiffs) output() []dax.WorkerDiff {
+	out := make([]dax.WorkerDiff, len(d))
+
+	i := 0
+	for k, v := range d {
+		out[i].WorkerID = k
+		out[i].AddedJobs = v.added.Sorted()
+		out[i].RemovedJobs = v.removed.Sorted()
+		i++
+	}
+
+	sort.Sort(dax.WorkerDiffs(out))
+
+	return out
+}

dax/mds/controller/node_registerer.go

@@ -0,0 +1,60 @@
+package controller
+
+import (
+	"context"
+	"time"
+
+	"github.com/molecula/featurebase/v3/dax"
+)
+
+// nodeRegistrationRoutine is a long-running goroutine that reads
+// newly registered nodes from a channel and sends out
+// new directives to rebalance among all the nodes.
+//
+// If the provided timeout is 0, this routine will register each node placed on
+// the channel immediately.
+//
+// If the provided timeout is >0, this routine will batch the nodes until the
+// timeout time has passed. This prevents (for the case when scaling up by >1
+// node at a time) multiple directives being sent out serially as each node
+// joins, and instead tries to handle all new nodes simultaneously.
+func (c *Controller) nodeRegistrationRoutine(nodes chan *dax.Node, timeout time.Duration) error {
+	if timeout > 0 {
+		return c.nodeRegistrationDelayed(nodes, timeout)
+	}
+	return c.nodeRegistrationInstant(nodes)
+}
+
+func (c *Controller) nodeRegistrationInstant(nodes chan *dax.Node) error {
+	for node := range nodes {
+		err := c.RegisterNodes(context.Background(), node)
+		if err != nil {
+			c.logger.Errorf("Registering node: %v, encountered error: %v", node, err)
+		}
+	}
+	return nil
+}
+
+func (c *Controller) nodeRegistrationDelayed(nodes chan *dax.Node, timeout time.Duration) error {
+	batch := []*dax.Node{}
+	c.logger.Printf("Running with batch registration timeout: %v", timeout)
+	for {
+		select {
+		case <-c.stopping:
+			return nil
+		case node := <-nodes:
+			c.logger.Debugf("adding node: %+v", node)
+			batch = append(batch, node)
+		case <-time.After(timeout):
+			c.logger.Debugf("no new nodes in last %s, batch: %d", timeout, len(batch))
+			if len(batch) > 0 {
+				err := c.RegisterNodes(context.Background(), batch...)
+				if err != nil {
+					c.logger.Errorf("Registering nodes: %v, encountered error: %v", batch, err)
+				}
+
+				batch = batch[:0] // reset batch
+			}
+		}
+	}
+}

dax/mds/controller/partitioner/partitioner.go

@@ -0,0 +1,55 @@
+// Package partitioner provides the Partitioner type, which provides helper
+// methods for determining partitions based on string keys.
+package partitioner
+
+import (
+	"encoding/binary"
+	"hash/fnv"
+
+	"github.com/molecula/featurebase/v3/dax"
+)
+
+// Partitioner encapsulates helper methods for determining partitions
+type Partitioner struct{}
+
+// NewPartitioner returns a new instance of Partitioner with default values.
+func NewPartitioner() *Partitioner {
+	return &Partitioner{}
+}
+
+// PartitionsForKeys returns a map of partitions to the list of strings which
+// fall into that partition.
+func (p *Partitioner) PartitionsForKeys(tkey dax.TableKey, partitionN int, keys ...string) map[dax.PartitionNum][]string {
+	out := make(map[dax.PartitionNum][]string)
+
+	for _, key := range keys {
+		p := keyToPartition(tkey, partitionN, key)
+		if _, found := out[p]; !found {
+			out[p] = []string{}
+		}
+		out[p] = append(out[p], key)
+	}
+
+	return out
+}
+
+// keyToPartition returns the partition to which the given key belongs.
+func keyToPartition(tkey dax.TableKey, partitionN int, key string) dax.PartitionNum {
+	// Hash the bytes and mod by partition count.
+	h := fnv.New64a()
+	_, _ = h.Write([]byte(tkey))
+	_, _ = h.Write([]byte(key))
+	return dax.PartitionNum(h.Sum64() % uint64(partitionN))
+}
+
+// ShardToPartition returns the PartitionNum for the given shard.
+func (p *Partitioner) ShardToPartition(tkey dax.TableKey, shard dax.ShardNum, partitionN int) dax.PartitionNum {
+	var buf [8]byte
+	binary.BigEndian.PutUint64(buf[:], uint64(shard))
+
+	// Hash the bytes and mod by partition count.
+	h := fnv.New64a()
+	_, _ = h.Write([]byte(tkey))
+	_, _ = h.Write(buf[:])
+	return dax.PartitionNum(h.Sum64() % uint64(partitionN))
+}

dax/mds/controller/partitioner/partitioner_test.go

@@ -0,0 +1,86 @@
+package partitioner_test
+
+import (
+	"fmt"
+	"testing"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/mds/controller/partitioner"
+	"github.com/stretchr/testify/assert"
+)
+
+func TestPartitioner(t *testing.T) {
+	tableKey := dax.TableKey("foo")
+	partitionN := 8
+
+	t.Run("PartitionForKeys", func(t *testing.T) {
+		p := partitioner.NewPartitioner()
+
+		tests := []struct {
+			tkey       dax.TableKey
+			partitionN int
+			keys       []string
+			exp        map[dax.PartitionNum][]string
+		}{
+			{
+				tkey:       tableKey,
+				partitionN: partitionN,
+				keys:       []string{"a"},
+				exp: map[dax.PartitionNum][]string{
+					2: {"a"},
+				},
+			},
+			{
+				tkey:       tableKey,
+				partitionN: partitionN,
+				keys:       []string{"a", "a"},
+				exp: map[dax.PartitionNum][]string{
+					2: {"a", "a"},
+				},
+			},
+			{
+				tkey:       "differentTableName",
+				partitionN: partitionN,
+				keys:       []string{"a"},
+				exp: map[dax.PartitionNum][]string{
+					4: {"a"},
+				},
+			},
+			{
+				tkey:       tableKey,
+				partitionN: partitionN,
+				keys:       []string{"a", "b", "c", "d", "e", "f", "g", "h", "i"},
+				exp: map[dax.PartitionNum][]string{
+					0: {"g"},
+					1: {"d"},
+					2: {"a", "i"},
+					3: {"f"},
+					4: {"c"},
+					5: {"h"},
+					6: {"e"},
+					7: {"b"},
+				},
+			},
+		}
+		for i, test := range tests {
+			t.Run(fmt.Sprintf("test-%d", i), func(t *testing.T) {
+				out := p.PartitionsForKeys(test.tkey, test.partitionN, test.keys...)
+
+				assert.ElementsMatch(t, keys(test.exp), keys(out))
+
+				for k := range out {
+					assert.ElementsMatch(t, test.exp[k], out[k])
+				}
+			})
+		}
+
+	})
+}
+
+func keys(m map[dax.PartitionNum][]string) dax.PartitionNums {
+	keys := make(dax.PartitionNums, 0, len(m))
+	for k := range m {
+		keys = append(keys, k)
+	}
+	return keys
+}

dax/mds/controller/sets.go

@@ -0,0 +1,169 @@
+package controller
+
+import (
+	"sort"
+
+	"github.com/molecula/featurebase/v3/dax"
+)
+
+// StringSet is a set of strings.
+type StringSet map[string]struct{}
+
+func NewStringSet() StringSet {
+	return make(StringSet)
+}
+
+func (s StringSet) Add(p string) {
+	s[p] = struct{}{}
+}
+
+func (s StringSet) Remove(p string) {
+	delete(s, p)
+}
+
+func (s StringSet) Contains(p string) bool {
+	_, ok := s[p]
+	return ok
+}
+
+func (s StringSet) SortedSlice() []string {
+	ps := make([]string, 0, len(s))
+	for p := range s {
+		ps = append(ps, p)
+	}
+	sort.Strings(ps)
+
+	return ps
+}
+
+func (s StringSet) Minus(m StringSet) []string {
+	diff := []string{}
+
+	for sk := range s {
+		var found bool
+		for mk := range m {
+			if mk == sk {
+				found = true
+				break
+			}
+		}
+		if !found {
+			diff = append(diff, sk)
+		}
+	}
+
+	return diff
+}
+
+// TableSet is a set of strings.
+type TableSet map[dax.TableKey]struct{}
+
+func NewTableSet() TableSet {
+	return make(TableSet)
+}
+
+func (s TableSet) Add(t dax.TableKey) {
+	s[t] = struct{}{}
+}
+
+func (s TableSet) Remove(t dax.TableKey) {
+	delete(s, t)
+}
+
+func (s TableSet) Contains(t dax.TableKey) bool {
+	_, ok := s[t]
+	return ok
+}
+
+func (s TableSet) SortedSlice() dax.TableKeys {
+	ps := make(dax.TableKeys, 0, len(s))
+	for p := range s {
+		ps = append(ps, p)
+	}
+	sort.Sort(ps)
+
+	return ps
+}
+
+func (s TableSet) QualifiedSortedSlice() map[dax.TableQualifier]dax.TableIDs {
+	m := make(map[dax.TableQualifier]dax.TableIDs)
+	for p := range s {
+		qtid := p.QualifiedTableID()
+		m[qtid.TableQualifier] = append(m[qtid.TableQualifier], qtid.ID)
+	}
+
+	// Sort the slices in the map.
+	for _, v := range m {
+		sort.Sort(v)
+	}
+
+	return m
+}
+
+func (s TableSet) Minus(m TableSet) dax.TableKeys {
+	diff := dax.TableKeys{}
+
+	for sk := range s {
+		var found bool
+		for mk := range m {
+			if mk == sk {
+				found = true
+				break
+			}
+		}
+		if !found {
+			diff = append(diff, sk)
+		}
+	}
+
+	return diff
+}
+
+// AddressSet is a set of strings.
+type AddressSet map[dax.Address]struct{}
+
+func NewAddressSet() AddressSet {
+	return make(AddressSet)
+}
+
+func (s AddressSet) Add(p dax.Address) {
+	s[p] = struct{}{}
+}
+
+func (s AddressSet) Remove(p dax.Address) {
+	delete(s, p)
+}
+
+func (s AddressSet) Contains(p dax.Address) bool {
+	_, ok := s[p]
+	return ok
+}
+
+func (s AddressSet) SortedSlice() []dax.Address {
+	ps := make([]dax.Address, 0, len(s))
+	for p := range s {
+		ps = append(ps, p)
+	}
+	sort.Slice(ps, func(i, j int) bool { return ps[i] < ps[j] })
+
+	return ps
+}
+
+func (s AddressSet) Minus(m AddressSet) []dax.Address {
+	diff := []dax.Address{}
+
+	for sk := range s {
+		var found bool
+		for mk := range m {
+			if mk == sk {
+				found = true
+				break
+			}
+		}
+		if !found {
+			diff = append(diff, sk)
+		}
+	}
+
+	return diff
+}

dax/mds/controller/stringers.go

@@ -0,0 +1,104 @@
+package controller
+
+import (
+	"fmt"
+	"strconv"
+	"strings"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/errors"
+)
+
+// pUnit represents a table/partition combination. As a Stringer, it can be
+// used as a job in the Balancer.
+type pUnit struct {
+	t dax.TableKey
+	p dax.Partition
+}
+
+func (p pUnit) String() string {
+	return fmt.Sprintf("%s|part_%d", p.t, p.p.Num)
+}
+
+func (p pUnit) table() dax.TableKey {
+	return p.t
+}
+
+func (p pUnit) partitionNum() dax.PartitionNum {
+	return p.p.Num
+}
+
+func partition(t dax.TableKey, p dax.Partition) pUnit {
+	return pUnit{t, p}
+}
+
+func decodePartition(j dax.Job) (pUnit, error) {
+	s := string(j)
+	parts := strings.Split(s, "|")
+	if len(parts) != 2 {
+		return pUnit{}, errors.Errorf("cannot decode string to partition: %s", s)
+	}
+	pparts := strings.Split(parts[1], "_")
+	if len(pparts) != 2 {
+		return pUnit{}, errors.Errorf("cannot decode partition part of string: %s", pparts[1])
+	}
+	intVar, err := strconv.Atoi(pparts[1])
+	if err != nil {
+		return pUnit{}, errors.Wrap(err, "converting string to int")
+	}
+
+	return pUnit{
+		t: dax.TableKey(parts[0]),
+		p: dax.Partition{
+			Num:     dax.PartitionNum(intVar),
+			Version: -1,
+		},
+	}, nil
+}
+
+// sUnit represents a table/shard combination. As a Stringer, it can be used as
+// a job in the Balancer.
+type sUnit struct {
+	t dax.TableKey
+	s dax.Shard
+}
+
+func (s sUnit) String() string {
+	return fmt.Sprintf("%s|shard_%s", s.t, s.s.Num)
+}
+
+func (s sUnit) table() dax.TableKey {
+	return s.t
+}
+
+func (s sUnit) shardNum() dax.ShardNum {
+	return s.s.Num
+}
+
+func shard(t dax.TableKey, s dax.Shard) sUnit {
+	return sUnit{t, s}
+}
+
+func decodeShard(j dax.Job) (sUnit, error) {
+	s := string(j)
+	parts := strings.Split(s, "|")
+	if len(parts) != 2 {
+		return sUnit{}, errors.Errorf("cannot decode string to shardV: %s", s)
+	}
+	pparts := strings.Split(parts[1], "_")
+	if len(pparts) != 2 {
+		return sUnit{}, errors.Errorf("cannot decode shard part of string: %s", pparts[1])
+	}
+	uint64Var, err := strconv.ParseUint(pparts[1], 10, 64)
+	if err != nil {
+		return sUnit{}, errors.Wrap(err, "converting string to int")
+	}
+
+	return sUnit{
+		t: dax.TableKey(parts[0]),
+		s: dax.Shard{
+			Num:     dax.ShardNum(uint64Var),
+			Version: -1,
+		},
+	}, nil
+}

dax/mds/controller/types.go

@@ -0,0 +1,19 @@
+package controller
+
+import "github.com/molecula/featurebase/v3/dax"
+
+// ComputeNode represents a compute node and the table/shards for which it is
+// responsible.
+type ComputeNode struct {
+	Address dax.Address   `json:"address"`
+	Table   dax.TableKey  `json:"table"`
+	Shards  dax.ShardNums `json:"shards"`
+}
+
+// TranslateNode represents a translate node and the table/partitions for which
+// it is responsible.
+type TranslateNode struct {
+	Address    dax.Address       `json:"address"`
+	Table      dax.TableKey      `json:"table"`
+	Partitions dax.PartitionNums `json:"partitions"`
+}

dax/mds/http/addressmanager.go

@@ -0,0 +1,76 @@
+package http
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"log"
+	"net/http"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/errors"
+)
+
+var ErrNotImplemented = errors.New(errors.ErrUncoded, "not implemented")
+
+// Ensure type implements interface.
+var _ dax.AddressManager = &AddressManager{}
+
+// AddressManager is an http implementation of the AddressManager interface.
+type AddressManager struct {
+	mdsAddress dax.Address
+}
+
+func NewAddressManager(mdsAddress dax.Address) *AddressManager {
+	return &AddressManager{
+		mdsAddress: mdsAddress,
+	}
+}
+
+func (m *AddressManager) AddAddresses(ctx context.Context, addr ...dax.Address) error {
+	// Not implemented because it's currently not used
+	return ErrNotImplemented
+}
+
+func (m *AddressManager) RemoveAddresses(ctx context.Context, addrs ...dax.Address) error {
+	if len(addrs) == 0 {
+		return nil
+	}
+
+	if m.mdsAddress == "" {
+		return errors.Errorf("mdsAddress is empty; could not deregister: %s", addrs)
+	}
+	url := fmt.Sprintf("%s/deregister-nodes", m.mdsAddress.WithScheme("http"))
+	log.Printf("SEND deregister-nodes to: %s\n", url)
+
+	req := DeregisterNodesRequest{
+		Addresses: addrs,
+	}
+
+	// Encode the request.
+	postBody, err := json.Marshal(req)
+	if err != nil {
+		return errors.Wrap(err, "marshalling deregister node request to json")
+	}
+	requestBody := bytes.NewBuffer(postBody)
+
+	// Post the request.
+	request, _ := http.NewRequest(http.MethodPost, url, requestBody)
+	request.Header.Add("Content-Type", "application/json")
+	request.Header.Add("Accept", "application/json")
+
+	resp, err := http.DefaultClient.Do(request)
+	if err != nil {
+		return errors.Wrap(err, "doing deregister node request")
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return errors.Errorf("status code: %d: %s", resp.StatusCode, b)
+	}
+
+	return nil
+}

dax/mds/http/handler.go

@@ -0,0 +1,664 @@
+package http
+
+import (
+	"encoding/json"
+	"net/http"
+
+	"github.com/gorilla/mux"
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/mds"
+	"github.com/molecula/featurebase/v3/dax/mds/controller"
+)
+
+func Handler(mds *mds.MDS) http.Handler {
+	server := &server{
+		mds: mds,
+	}
+
+	router := mux.NewRouter()
+	router.HandleFunc("/health", server.getHealth).Methods("GET").Name("GetHealth")
+
+	// mds endpoints.
+	router.HandleFunc("/create-table", server.postCreateTable).Methods("POST").Name("PostCreateTable")
+	router.HandleFunc("/drop-table", server.postDropTable).Methods("POST").Name("PostDropTable")
+	router.HandleFunc("/create-field", server.postCreateField).Methods("POST").Name("PostCreateField")
+	router.HandleFunc("/drop-field", server.postDropField).Methods("POST").Name("PostDropField")
+	router.HandleFunc("/table", server.postTable).Methods("POST").Name("PostTable")
+	router.HandleFunc("/table-id", server.postTableID).Methods("POST").Name("PostTable")
+	router.HandleFunc("/tables", server.postTables).Methods("POST").Name("PostTables")
+
+	router.HandleFunc("/ingest-partition", server.postIngestPartition).Methods("POST").Name("PostIngestPartition")
+	router.HandleFunc("/ingest-shard", server.postIngestShard).Methods("POST").Name("PostIngestShard")
+
+	router.HandleFunc("/snapshot", server.postSnapshot).Methods("POST").Name("PostSnapshot")
+	router.HandleFunc("/snapshot/shard-data", server.postSnapshotShardData).Methods("POST").Name("PostShapshotShardData")
+	router.HandleFunc("/snapshot/table-keys", server.postSnapshotTableKeys).Methods("POST").Name("PostShapshotTableKeys")
+	router.HandleFunc("/snapshot/field-keys", server.postSnapshotFieldKeys).Methods("POST").Name("PostShapshotFieldKeys")
+
+	// controller endpoints.
+	router.HandleFunc("/register-node", server.postRegisterNode).Methods("POST").Name("PostRegisterNode")
+	router.HandleFunc("/register-nodes", server.postRegisterNodes).Methods("POST").Name("PostRegisterNodes")
+	router.HandleFunc("/deregister-nodes", server.postDeregisterNodes).Methods("POST").Name("PostDeregisterNodes")
+	router.HandleFunc("/check-in-node", server.postCheckInNode).Methods("POST").Name("PostCheckInNode")
+	router.HandleFunc("/compute-nodes", server.postComputeNodes).Methods("POST").Name("PostComputeNodes")
+	router.HandleFunc("/translate-nodes", server.postTranslateNodes).Methods("POST").Name("PostTranslateNodes")
+
+	// debug endpoints
+	router.HandleFunc("/debug/nodes", server.getDebugNodes).Methods("GET").Name("GetDebugNodes")
+
+	return router
+}
+
+type server struct {
+	mds *mds.MDS
+}
+
+// GET /health
+func (s *server) getHealth(w http.ResponseWriter, r *http.Request) {
+	w.WriteHeader(http.StatusOK)
+}
+
+// POST /create-table
+func (s *server) postCreateTable(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := &dax.QualifiedTable{}
+	if err := json.NewDecoder(body).Decode(req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	err := s.mds.CreateTable(ctx, req)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	resp := CreateTableResponse(*req)
+	if err := json.NewEncoder(w).Encode(resp); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+}
+
+type CreateTableResponse dax.QualifiedTable
+
+// POST /table
+func (s *server) postTable(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	qtid := dax.QualifiedTableID{}
+	if err := json.NewDecoder(body).Decode(&qtid); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+	resp, err := s.mds.Table(ctx, qtid)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	if err := json.NewEncoder(w).Encode(resp); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+}
+
+// POST /table-id
+func (s *server) postTableID(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := dax.QualifiedTableID{}
+	if err := json.NewDecoder(body).Decode(&req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	qtid, err := s.mds.TableID(ctx, req.TableQualifier, req.Name)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	if err := json.NewEncoder(w).Encode(qtid); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+}
+
+// POST /drop-table
+func (s *server) postDropTable(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := dax.QualifiedTableID{}
+	if err := json.NewDecoder(body).Decode(&req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	err := s.mds.DropTable(ctx, req)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+}
+
+// POST /create-field
+func (s *server) postCreateField(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := CreateFieldRequest{}
+	if err := json.NewDecoder(body).Decode(&req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	qtid := req.TableKey.QualifiedTableID()
+
+	err := s.mds.CreateField(ctx, qtid, req.Field)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+}
+
+type CreateFieldRequest struct {
+	TableKey dax.TableKey `json:"table-key"`
+	Field    *dax.Field   `json:"field"`
+}
+
+// POST /drop-field
+func (s *server) postDropField(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := DropFieldRequest{}
+	if err := json.NewDecoder(body).Decode(&req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	qtid := req.Table
+
+	err := s.mds.DropField(ctx, qtid, req.Field)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+}
+
+type DropFieldRequest struct {
+	Table dax.QualifiedTableID `json:"table"`
+	Field dax.FieldName        `json:"fields"`
+}
+
+// POST /tables
+func (s *server) postTables(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := TablesRequest{}
+	if err := json.NewDecoder(body).Decode(&req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	qual := dax.NewTableQualifier(req.OrganizationID, req.DatabaseID)
+	ids := req.TableIDs
+
+	resp, err := s.mds.Tables(ctx, qual, ids...)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	if err := json.NewEncoder(w).Encode(resp); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+}
+
+type TablesRequest struct {
+	OrganizationID dax.OrganizationID `json:"org-id"`
+	DatabaseID     dax.DatabaseID     `json:"db-id"`
+	TableIDs       dax.TableIDs       `json:"table-ids"`
+	TableNames     dax.TableNames     `json:"table-names"`
+}
+
+// POST /ingest-partition
+func (s *server) postIngestPartition(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := IngestPartitionRequest{}
+	if err := json.NewDecoder(body).Decode(&req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	qtid := req.Table
+
+	addr, err := s.mds.IngestPartition(ctx, qtid, req.Partition)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	resp := &IngestPartitionResponse{
+		Address: addr,
+	}
+
+	if err := json.NewEncoder(w).Encode(resp); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+}
+
+type IngestPartitionRequest struct {
+	Table     dax.QualifiedTableID `json:"table"`
+	Partition dax.PartitionNum     `json:"partition"`
+}
+
+type IngestPartitionResponse struct {
+	Address dax.Address `json:"address"`
+}
+
+// POST /ingest-shard
+func (s *server) postIngestShard(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := IngestShardRequest{}
+	if err := json.NewDecoder(body).Decode(&req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	qtid := req.Table
+
+	addr, err := s.mds.IngestShard(ctx, qtid, req.Shard)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	resp := &IngestShardResponse{
+		Address: addr,
+	}
+
+	if err := json.NewEncoder(w).Encode(resp); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+}
+
+type IngestShardRequest struct {
+	Table dax.QualifiedTableID `json:"table"`
+	Shard dax.ShardNum         `json:"shard"`
+}
+
+type IngestShardResponse struct {
+	Address dax.Address `json:"address"`
+}
+
+// POST /snapshot
+// High level snapshot endpoint to snapshot everything in a table.
+func (s *server) postSnapshot(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := dax.QualifiedTableID{}
+	if err := json.NewDecoder(body).Decode(&req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	if err := s.mds.SnapshotTable(ctx, req); err != nil {
+		http.Error(w, err.Error(), http.StatusInternalServerError)
+		return
+	}
+	w.WriteHeader(http.StatusOK)
+}
+
+// POST /snapshot/shard-data
+func (s *server) postSnapshotShardData(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := SnapshotShardRequest{}
+	if err := json.NewDecoder(body).Decode(&req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	qtid := req.Table
+
+	if err := s.mds.SnapshotShardData(ctx, qtid, req.Shard); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	w.WriteHeader(http.StatusOK)
+}
+
+// SnapshotShardRequest is used to specify the table/shard to snapshot.
+type SnapshotShardRequest struct {
+	Table dax.QualifiedTableID `json:"table"`
+	Shard dax.ShardNum         `json:"shard"`
+}
+
+// POST /snapshot/table-keys
+func (s *server) postSnapshotTableKeys(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := SnapshotTableKeysRequest{}
+	if err := json.NewDecoder(body).Decode(&req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	qtid := req.Table
+
+	if err := s.mds.SnapshotTableKeys(ctx, qtid, req.Partition); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	w.WriteHeader(http.StatusOK)
+}
+
+// SnapshotTableKeysRequest is used to specify the table/partition/keys to
+// snapshot.
+type SnapshotTableKeysRequest struct {
+	Table     dax.QualifiedTableID `json:"table"`
+	Partition dax.PartitionNum     `json:"partition"`
+}
+
+// POST /snapshot/field-keys
+func (s *server) postSnapshotFieldKeys(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := SnapshotFieldKeysRequest{}
+	if err := json.NewDecoder(body).Decode(&req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	qtid := req.Table
+
+	if err := s.mds.SnapshotFieldKeys(ctx, qtid, req.Field); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	w.WriteHeader(http.StatusOK)
+}
+
+// SnapshotFieldKeysRequest is used to specify the field/keys to snapshot.
+type SnapshotFieldKeysRequest struct {
+	Table dax.QualifiedTableID `json:"table"`
+	Field dax.FieldName        `json:"field"`
+}
+
+// POST /register-node
+func (s *server) postRegisterNode(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := RegisterNodeRequest{}
+	if err := json.NewDecoder(body).Decode(&req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	node := &dax.Node{
+		Address:   req.Address,
+		RoleTypes: req.RoleTypes,
+	}
+
+	if err := s.mds.RegisterNode(ctx, node); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	w.WriteHeader(http.StatusOK)
+}
+
+type RegisterNodeRequest struct {
+	Address dax.Address `json:"address"`
+
+	// RoleTypes allows a registering node to specify which role type(s) it is
+	// capable of filling. The controller will not assign a role to this node
+	// with a type not included in RoleTypes.
+	RoleTypes []dax.RoleType `json:"role-types"`
+}
+
+// POST /register-nodes
+func (s *server) postRegisterNodes(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := RegisterNodesRequest{}
+	if err := json.NewDecoder(body).Decode(&req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	if err := s.mds.RegisterNodes(ctx, req.Nodes...); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	w.WriteHeader(http.StatusOK)
+}
+
+type RegisterNodesRequest struct {
+	Nodes []*dax.Node `json:"nodes"`
+}
+
+// POST /deregister-nodes
+func (s *server) postDeregisterNodes(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := DeregisterNodesRequest{}
+	if err := json.NewDecoder(body).Decode(&req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	if err := s.mds.DeregisterNodes(ctx, req.Addresses...); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	w.WriteHeader(http.StatusOK)
+}
+
+type DeregisterNodesRequest struct {
+	Addresses []dax.Address `json:"addresses"`
+}
+
+// POST /check-in-node
+func (s *server) postCheckInNode(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := CheckInNodeRequest{}
+	if err := json.NewDecoder(body).Decode(&req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	node := &dax.Node{
+		Address:   req.Address,
+		RoleTypes: req.RoleTypes,
+	}
+
+	if err := s.mds.CheckInNode(ctx, node); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	w.WriteHeader(http.StatusOK)
+}
+
+type CheckInNodeRequest struct {
+	Address dax.Address `json:"address"`
+
+	// RoleTypes allows a registering node to specify which role type(s) it is
+	// capable of filling. The controller will not assign a role to this node
+	// with a type not included in RoleTypes.
+	RoleTypes []dax.RoleType `json:"role-types"`
+}
+
+// POST /compute-nodes
+func (s *server) postComputeNodes(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := ComputeNodesRequest{}
+	if err := json.NewDecoder(body).Decode(&req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	qtid := req.Table
+
+	nodes, err := s.mds.ComputeNodes(ctx, qtid, req.Shards...)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	resp := ComputeNodesResponse{
+		ComputeNodes: nodes,
+	}
+
+	if err := json.NewEncoder(w).Encode(resp); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+}
+
+func (s *server) getDebugNodes(w http.ResponseWriter, r *http.Request) {
+	nodes, err := s.mds.DebugNodes(r.Context())
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusInternalServerError)
+		return
+	}
+	if err := json.NewEncoder(w).Encode(nodes); err != nil {
+		http.Error(w, err.Error(), http.StatusInternalServerError)
+		return
+	}
+
+}
+
+// ComputeNodesRequest is used to specify the table/shards to consider in the
+// ComputeNodes method call. If IsWrite is true, shards which are not currently
+// being managed by the underlying Controller will be added to (registered with)
+// the Controller and, if adequate compute is available, will be associated with
+// a compute node.
+type ComputeNodesRequest struct {
+	Table   dax.QualifiedTableID `json:"table"`
+	Shards  dax.ShardNums        `json:"shards"`
+	IsWrite bool                 `json:"is-write"`
+}
+
+// ComputeNodesResponse contains the list of compute nodes returned based on the
+// table/shards specified in the ComputeNodeRequest. It's possible that shards
+// provided are not included in this response. That might happen if there are
+// currently no active compute nodes.
+type ComputeNodesResponse struct {
+	ComputeNodes []controller.ComputeNode `json:"compute-nodes"`
+}
+
+// POST /translate-nodes
+func (s *server) postTranslateNodes(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := TranslateNodesRequest{}
+	if err := json.NewDecoder(body).Decode(&req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	qtid := req.Table
+
+	nodes, err := s.mds.TranslateNodes(ctx, qtid, req.Partitions...)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	resp := TranslateNodesResponse{
+		TranslateNodes: nodes,
+	}
+
+	if err := json.NewEncoder(w).Encode(resp); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+}
+
+// TranslateNodesRequest is used to specify the table/partitions to consider in
+// the TranslateNodes method call.
+type TranslateNodesRequest struct {
+	Table      dax.QualifiedTableID `json:"table"`
+	Partitions dax.PartitionNums    `json:"partitions"`
+	IsWrite    bool                 `json:"is-write"`
+}
+
+// TranslateNodesResponse contains the list of translate nodes returned based on
+// the table/partitions specified in the TranslateNodeRequest. It's possible
+// that partitions provided are not included in this response. That might happen
+// if there are currently no active translate nodes.
+type TranslateNodesResponse struct {
+	TranslateNodes []controller.TranslateNode `json:"translate-nodes"`
+}

dax/mds/mds.go

@@ -0,0 +1,463 @@
+// Package mds provides the overall interface to Metadata Services.
+package mds
+
+import (
+	"context"
+	"fmt"
+	"log"
+	"os"
+	"sync"
+	"time"
+
+	fb "github.com/molecula/featurebase/v3"
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/boltdb"
+	"github.com/molecula/featurebase/v3/dax/mds/controller"
+	naiveboltdb "github.com/molecula/featurebase/v3/dax/mds/controller/naive/boltdb"
+	"github.com/molecula/featurebase/v3/dax/mds/poller"
+	"github.com/molecula/featurebase/v3/dax/mds/schemar"
+	schemarboltdb "github.com/molecula/featurebase/v3/dax/mds/schemar/boltdb"
+	"github.com/molecula/featurebase/v3/errors"
+	"github.com/molecula/featurebase/v3/logger"
+)
+
+type Config struct {
+	// Controller
+	Director controller.Director
+	// RegistrationBatchTimeout is the time that the controller will
+	// wait after a node registers itself to see if any more nodes
+	// will register before sending out directives to all nodes which
+	// have been registered.
+	RegistrationBatchTimeout time.Duration
+
+	// Poller
+	PollInterval time.Duration
+
+	// Storage
+	StorageMethod string
+	StorageDSN    string
+
+	// Logger
+	Logger logger.Logger
+}
+
+// Ensure type implements interface.
+var _ fb.MDS = (*MDS)(nil)
+
+// MDS provides public MDS methods for an MDS service.
+type MDS struct {
+	mu sync.RWMutex
+
+	controller *controller.Controller
+	poller     *poller.Poller
+	schemar    schemar.Schemar
+
+	logger logger.Logger
+}
+
+// New returns a new instance of MDS.
+func New(cfg Config) *MDS {
+	// Set up logger.
+	var logr = logger.NopLogger
+	if cfg.Logger != nil {
+		logr = cfg.Logger
+	}
+
+	// Storage methods.
+	if cfg.StorageMethod != "boltdb" && cfg.StorageMethod != "" {
+		log.Printf("storagemethod %s not supported, try 'boltdb'", cfg.StorageMethod)
+	}
+	if cfg.StorageDSN == "" {
+		dir, err := os.MkdirTemp("", "mds_*")
+		if err != nil {
+			logr.Printf("Making temp dir for MDS storage: %v", err)
+			os.Exit(1)
+		}
+		cfg.StorageDSN = fmt.Sprintf("file:%s", dir)
+		logr.Warnf("no StorageDSN given (like 'file:/path/to/directory') using temp dir at '%s'", cfg.StorageDSN)
+	}
+	schemarDB, err := boltdb.NewSvcBolt(cfg.StorageDSN, "schemar", schemarboltdb.SchemarBuckets...)
+	if err != nil {
+		logr.Printf("Error creating schemar db: %v", err)
+		os.Exit(1)
+	}
+	schemar := schemarboltdb.NewSchemar(schemarDB, logr)
+
+	boltDB, err := boltdb.NewSvcBolt(cfg.StorageDSN, "balancer", naiveboltdb.NaiveBalancerBuckets...)
+	if err != nil {
+		log.Println(errors.Wrap(err, "creating balancer bolt"))
+		os.Exit(1)
+	}
+
+	controllerCfg := controller.Config{
+		Director:          cfg.Director,
+		Schemar:           schemar,
+		ComputeBalancer:   naiveboltdb.NewBalancer("compute", boltDB, logr),
+		TranslateBalancer: naiveboltdb.NewBalancer("translate", boltDB, logr),
+
+		RegistrationBatchTimeout: cfg.RegistrationBatchTimeout,
+
+		StorageMethod: cfg.StorageMethod,
+		// just reusing this bolt for internal controller svcs
+		// rn... ultimately controller shouldn't know what bolt is at
+		// all
+		BoltDB: boltDB,
+
+		Logger: logr,
+	}
+	controller := controller.New(controllerCfg)
+
+	pollerCfg := poller.Config{
+		AddressManager: controller,
+		NodePoller:     poller.NewHTTPNodePoller(logr),
+		PollInterval:   cfg.PollInterval,
+		Logger:         logr,
+	}
+	poller := poller.New(pollerCfg)
+
+	// The controller needs to tell the poller about nodes which have been
+	// added/removed.
+	// TODO: this feels hacky. We need an elegant way to register interface
+	// implementations across services without an explicit Set method like this.
+	controller.SetPoller(poller)
+
+	return &MDS{
+		controller: controller,
+		poller:     poller,
+		schemar:    schemar,
+
+		logger: logr,
+	}
+}
+
+////////////////////////////////////////////////////
+// mds specific endpoints
+////////////////////////////////////////////////////
+
+// Run starts MDS services, such as the Poller.
+func (m *MDS) Run() error {
+	// Initialize the poller (in the case where this MDS instance has restarted
+	// or is a replacement). Then start the poller.
+	if err := m.controller.InitializePoller(context.Background()); err != nil {
+		return errors.Wrap(err, "initializing the poller")
+	}
+	m.poller.Run()
+
+	return m.controller.Run()
+}
+
+// Stop stops MDS services, such as the Poller and the controller's node
+// registration routine.
+func (m *MDS) Stop() error {
+	m.poller.Stop()
+	m.controller.Stop()
+	return nil
+}
+
+// sanitizeQTID populates Table.ID (by looking up the table, by name, in
+// schemar) for a given table having only a Name value, but no ID.
+func (m *MDS) sanitizeQTID(ctx context.Context, qtid *dax.QualifiedTableID) error {
+	if qtid.ID == "" {
+		nqtid, err := m.schemar.TableID(ctx, qtid.TableQualifier, qtid.Name)
+		if err != nil {
+			return errors.Wrap(err, "getting table ID")
+		}
+		qtid.ID = nqtid.ID
+	}
+	return nil
+}
+
+// CreateTable handles a create table request.
+func (m *MDS) CreateTable(ctx context.Context, qtbl *dax.QualifiedTable) error {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+
+	// Create Table ID.
+	if _, err := qtbl.CreateID(); err != nil {
+		return errors.Wrap(err, "creating table ID")
+	}
+
+	// Create the table in schemar.
+	if err := m.schemar.CreateTable(ctx, qtbl); err != nil {
+		return errors.Wrapf(err, "creating table: %s", qtbl)
+	}
+
+	// TODO: if error here, we should probably roll-back the
+	// schemar.CreateTable() request.
+
+	// Add the table to the controller.
+	return m.controller.CreateTable(ctx, qtbl)
+}
+
+// DropTable handles a drop table request. // TODO(jaffee) how do we
+// reason about consistency here? What if controller DropTable
+// succeeds, but schemar fails?
+func (m *MDS) DropTable(ctx context.Context, qtid dax.QualifiedTableID) error {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+
+	if err := m.sanitizeQTID(ctx, &qtid); err != nil {
+		return errors.Wrap(err, "sanitizing")
+	}
+
+	if err := m.controller.DropTable(ctx, qtid); err != nil {
+		return errors.Wrapf(err, "dropping table: %s", qtid)
+	}
+
+	return m.schemar.DropTable(ctx, qtid)
+}
+
+type CreateFieldRequest struct {
+	Table dax.TableName
+	Field *dax.Field
+}
+
+// CreateField handles a create Field request.
+func (m *MDS) CreateField(ctx context.Context, qtid dax.QualifiedTableID, fld *dax.Field) error {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+
+	if err := m.sanitizeQTID(ctx, &qtid); err != nil {
+		return errors.Wrap(err, "sanitizing")
+	}
+
+	// Create the field in schemar.
+	if err := m.schemar.CreateField(ctx, qtid, fld); err != nil {
+		return errors.Wrapf(err, "creating field: %s, %s", qtid, fld)
+	}
+
+	// Add the table to the controller.
+	return m.controller.CreateField(ctx, qtid, fld)
+}
+
+// DropField handles a drop Field request.
+func (m *MDS) DropField(ctx context.Context, qtid dax.QualifiedTableID, fldName dax.FieldName) error {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+
+	if err := m.sanitizeQTID(ctx, &qtid); err != nil {
+		return errors.Wrap(err, "sanitizing")
+	}
+
+	// Drop the field from schemar.
+	if err := m.schemar.DropField(ctx, qtid, fldName); err != nil {
+		return errors.Wrapf(err, "dropping field: %s, %s", qtid, fldName)
+	}
+
+	// Drop the field from the controller.
+	return m.controller.DropField(ctx, qtid, fldName)
+}
+
+type DropFieldResponse struct{}
+
+// Table handles a table request.
+func (m *MDS) Table(ctx context.Context, qtid dax.QualifiedTableID) (*dax.QualifiedTable, error) {
+	m.mu.RLock()
+	defer m.mu.RUnlock()
+
+	if err := m.sanitizeQTID(ctx, &qtid); err != nil {
+		return nil, errors.Wrap(err, "sanitizing")
+	}
+
+	return m.schemar.Table(ctx, qtid)
+}
+
+// Tables handles a tables request.
+func (m *MDS) Tables(ctx context.Context, qual dax.TableQualifier, ids ...dax.TableID) ([]*dax.QualifiedTable, error) {
+	m.mu.RLock()
+	defer m.mu.RUnlock()
+
+	return m.schemar.Tables(ctx, qual, ids...)
+}
+
+// TableID handles a table id (i.e. by name) request.
+func (m *MDS) TableID(ctx context.Context, qual dax.TableQualifier, name dax.TableName) (dax.QualifiedTableID, error) {
+	m.mu.RLock()
+	defer m.mu.RUnlock()
+
+	return m.schemar.TableID(ctx, qual, name)
+}
+
+// IngestPartition handles an ingest partition request.
+func (m *MDS) IngestPartition(ctx context.Context, qtid dax.QualifiedTableID, partnNum dax.PartitionNum) (dax.Address, error) {
+	m.mu.RLock()
+	defer m.mu.RUnlock()
+
+	if err := m.sanitizeQTID(ctx, &qtid); err != nil {
+		return "", errors.Wrap(err, "sanitizing")
+	}
+
+	// Verify that the table exists.
+	if _, err := m.schemar.Table(ctx, qtid); err != nil {
+		return "", err
+	}
+
+	partitions := dax.PartitionNums{partnNum}
+
+	nodes, err := m.controller.TranslateNodes(ctx, qtid, partitions, true)
+	if err != nil {
+		return "", err
+	}
+
+	if l := len(nodes); l == 0 {
+		return "", controller.NewErrNoAvailableNode()
+	} else if l > 1 {
+		return "", controller.NewErrInternal(
+			fmt.Sprintf("unexpected number of nodes: %d", l))
+	}
+
+	node := nodes[0]
+
+	// Verify that the node returned is actually responsible for the partition
+	// requested.
+	if node.Table != qtid.Key() {
+		return "", controller.NewErrInternal(
+			fmt.Sprintf("table returned (%s) does not match requested (%s)", node.Table, qtid))
+	} else if l := len(node.Partitions); l != 1 {
+		return "", controller.NewErrInternal(
+			fmt.Sprintf("unexpected number of partitions returned: %d", l))
+	} else if p := node.Partitions[0]; p != partnNum {
+		return "", controller.NewErrInternal(
+			fmt.Sprintf("partition returned (%d) does not match requested (%d)", p, partnNum))
+	}
+
+	return node.Address, nil
+}
+
+// IngestShard handles an ingest shard request.
+func (m *MDS) IngestShard(ctx context.Context, qtid dax.QualifiedTableID, shrdNum dax.ShardNum) (dax.Address, error) {
+	m.mu.RLock()
+	defer m.mu.RUnlock()
+
+	if err := m.sanitizeQTID(ctx, &qtid); err != nil {
+		return "", errors.Wrap(err, "sanitizing")
+	}
+
+	// Verify that the table exists.
+	if _, err := m.schemar.Table(ctx, qtid); err != nil {
+		return "", err
+	}
+
+	shards := dax.ShardNums{shrdNum}
+
+	nodes, err := m.controller.ComputeNodes(ctx, qtid, shards, true)
+	if err != nil {
+		return "", err
+	}
+
+	if l := len(nodes); l == 0 {
+		return "", controller.NewErrNoAvailableNode()
+	} else if l > 1 {
+		return "", controller.NewErrInternal(
+			fmt.Sprintf("unexpected number of nodes: %d", l))
+	}
+
+	node := nodes[0]
+
+	// Verify that the node returned is actually responsible for the shard
+	// requested.
+	if node.Table != qtid.Key() {
+		return "", controller.NewErrInternal(
+			fmt.Sprintf("table returned (%s) does not match requested (%s)", node.Table, qtid))
+	} else if l := len(node.Shards); l != 1 {
+		return "", controller.NewErrInternal(
+			fmt.Sprintf("unexpected number of shards returned: %d", l))
+	} else if s := node.Shards[0]; s != shrdNum {
+		return "", controller.NewErrInternal(
+			fmt.Sprintf("shard returned (%d) does not match requested (%d)", s, shrdNum))
+	}
+
+	return node.Address, nil
+}
+
+// SnapshotTable handles a snapshot table request.
+func (m *MDS) SnapshotTable(ctx context.Context, qtid dax.QualifiedTableID) error {
+	if err := m.sanitizeQTID(ctx, &qtid); err != nil {
+		return errors.Wrap(err, "sanitizing")
+	}
+
+	return m.controller.SnapshotTable(ctx, qtid)
+}
+
+// SnapshotShardData handles a snapshot shard request.
+func (m *MDS) SnapshotShardData(ctx context.Context, qtid dax.QualifiedTableID, shardNum dax.ShardNum) error {
+	if err := m.sanitizeQTID(ctx, &qtid); err != nil {
+		return errors.Wrap(err, "sanitizing")
+	}
+
+	return m.controller.SnapshotShardData(ctx, qtid, shardNum)
+}
+
+// SnapshotTableKeys handles a snapshot table/keys request.
+func (m *MDS) SnapshotTableKeys(ctx context.Context, qtid dax.QualifiedTableID, partitionNum dax.PartitionNum) error {
+	if err := m.sanitizeQTID(ctx, &qtid); err != nil {
+		return errors.Wrap(err, "sanitizing")
+	}
+
+	return m.controller.SnapshotTableKeys(ctx, qtid, partitionNum)
+}
+
+// SnapshotFieldKeys handles a snapshot field/keys request.
+func (m *MDS) SnapshotFieldKeys(ctx context.Context, qtid dax.QualifiedTableID, fldName dax.FieldName) error {
+	if err := m.sanitizeQTID(ctx, &qtid); err != nil {
+		return errors.Wrap(err, "sanitizing")
+	}
+
+	return m.controller.SnapshotFieldKeys(ctx, qtid, fldName)
+}
+
+////////////////////////////////////////////////////
+// controller specific endpoints
+// These are just pass-throughs for now.
+////////////////////////////////////////////////////
+
+// RegisterNode handles a node registration request. It does not
+// synchronously do much of anything, but the node will eventually
+// probably get a directive... unless the MDS crashes or something in
+// which case the fact that this endpoint was ever called will be lost
+// to time.
+func (m *MDS) RegisterNode(ctx context.Context, node *dax.Node) error {
+	return m.controller.RegisterNode(ctx, node)
+}
+
+// CheckInNode handles a node check-in request. If MDS is not aware of the node,
+// it will be sent through the RegisterNode process.
+func (m *MDS) CheckInNode(ctx context.Context, node *dax.Node) error {
+	return m.controller.CheckInNode(ctx, node)
+}
+
+// RegisterNodes immediately registers the given nodes and sends out
+// new directives synchronously, bypassing the wait time of the
+// RegisterNode endpoint.
+func (m *MDS) RegisterNodes(ctx context.Context, nodes ...*dax.Node) error {
+	return m.controller.RegisterNodes(ctx, nodes...)
+}
+
+// DeregisterNodes handles a request to deregister multiple nodes at once.
+func (m *MDS) DeregisterNodes(ctx context.Context, addrs ...dax.Address) error {
+	return m.controller.DeregisterNodes(ctx, addrs...)
+}
+
+// ComputeNodes gets the compute nodes responsible for the table/shards
+// specified in the ComputeNodeRequest.
+func (m *MDS) ComputeNodes(ctx context.Context, qtid dax.QualifiedTableID, shardNums ...dax.ShardNum) ([]controller.ComputeNode, error) {
+	if err := m.sanitizeQTID(ctx, &qtid); err != nil {
+		return nil, errors.Wrap(err, "sanitizing")
+	}
+
+	return m.controller.ComputeNodes(ctx, qtid, shardNums, false)
+}
+
+func (m *MDS) DebugNodes(ctx context.Context) ([]*dax.Node, error) {
+	return m.controller.DebugNodes(ctx)
+}
+
+// TranslateNodes gets the translate nodes responsible for the table/partitions
+// specified in the TranslateNodeRequest.
+func (m *MDS) TranslateNodes(ctx context.Context, qtid dax.QualifiedTableID, partitionNums ...dax.PartitionNum) ([]controller.TranslateNode, error) {
+	if err := m.sanitizeQTID(ctx, &qtid); err != nil {
+		return nil, errors.Wrap(err, "sanitizing")
+	}
+
+	return m.controller.TranslateNodes(ctx, qtid, partitionNums, false)
+}

dax/mds/poller/config.go

@@ -0,0 +1,15 @@
+package poller
+
+import (
+	"time"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/logger"
+)
+
+type Config struct {
+	AddressManager dax.AddressManager
+	NodePoller     NodePoller
+	PollInterval   time.Duration
+	Logger         logger.Logger
+}

dax/mds/poller/interfaces.go

@@ -0,0 +1,59 @@
+package poller
+
+import (
+	"fmt"
+	"net/http"
+	"time"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/logger"
+)
+
+// NodePoller is an interface to anything which has the ability to poll a
+// node.
+type NodePoller interface {
+	Poll(dax.Address) bool
+}
+
+// Ensure type implements interface.
+var _ NodePoller = (*NopNodePoller)(nil)
+var _ NodePoller = (*HTTPNodePoller)(nil)
+
+// NopNodePoller is a no-op implementation of the NodePoller interface.
+type NopNodePoller struct{}
+
+func NewNopNodePoller() *NopNodePoller {
+	return &NopNodePoller{}
+}
+
+func (p *NopNodePoller) Poll(addr dax.Address) bool {
+	return true
+}
+
+// HTTPNodePoller is an http implementation of the NodePoller interface.
+type HTTPNodePoller struct {
+	logger logger.Logger
+	client *http.Client
+}
+
+func NewHTTPNodePoller(logger logger.Logger) *HTTPNodePoller {
+	return &HTTPNodePoller{
+		logger: logger,
+		client: &http.Client{
+			Timeout: time.Second, // short timeout for polling to detect issues quickly. /health endpoints should always respond fast.
+		},
+	}
+}
+
+func (p *HTTPNodePoller) Poll(addr dax.Address) bool {
+	url := fmt.Sprintf("%s/health", addr.WithScheme("http"))
+
+	if resp, err := p.client.Get(url); err != nil {
+		p.logger.Printf("poll error: %s\n", err)
+		return false
+	} else if resp.StatusCode != http.StatusOK {
+		return false
+	}
+
+	return true
+}

dax/mds/poller/poller.go

@@ -0,0 +1,156 @@
+// Package poller provides the core Poller struct.
+package poller
+
+import (
+	"context"
+	"sync"
+	"time"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/logger"
+)
+
+// Poller maintains a list of nodes to poll. It also polls them.
+type Poller struct {
+	mu sync.RWMutex
+
+	addresses map[dax.Address]struct{}
+
+	addressManager dax.AddressManager
+
+	nodePoller   NodePoller
+	pollInterval time.Duration
+
+	running  bool
+	stopping chan struct{}
+
+	logger logger.Logger
+}
+
+// New returns a new instance of Poller with default values.
+func New(cfg Config) *Poller {
+	p := &Poller{
+		addresses:      make(map[dax.Address]struct{}),
+		addressManager: dax.NewNopAddressManager(),
+		nodePoller:     NewNopNodePoller(),
+		pollInterval:   time.Second,
+		stopping:       make(chan struct{}),
+		logger:         logger.NopLogger,
+	}
+
+	// Set config options.
+	if cfg.AddressManager != nil {
+		p.addressManager = cfg.AddressManager
+	}
+	if cfg.NodePoller != nil {
+		p.nodePoller = cfg.NodePoller
+	}
+	if cfg.PollInterval != 0 {
+		p.pollInterval = cfg.PollInterval
+	}
+	if cfg.Logger != nil {
+		p.logger = cfg.Logger
+	}
+
+	return p
+}
+
+func (p *Poller) AddAddresses(ctx context.Context, addrs ...dax.Address) error {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+
+	for _, addr := range addrs {
+		p.addresses[addr] = struct{}{}
+	}
+
+	return nil
+}
+
+func (p *Poller) RemoveAddresses(ctx context.Context, addrs ...dax.Address) error {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+
+	for _, addr := range addrs {
+		delete(p.addresses, addr)
+	}
+
+	return nil
+}
+
+func (p *Poller) Addresses() []dax.Address {
+	p.mu.RLock()
+	defer p.mu.RUnlock()
+
+	addrs := make([]dax.Address, 0, len(p.addresses))
+	for addr := range p.addresses {
+		addrs = append(addrs, addr)
+	}
+
+	return addrs
+}
+
+// Run starts the polling goroutine.
+func (p *Poller) Run() {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+
+	if p.running {
+		p.logger.Printf("poller is already running")
+		return
+	}
+	p.running = true
+
+	go func() { p.run() }()
+}
+
+func (p *Poller) run() {
+	ticker := time.NewTicker(p.pollInterval)
+	defer ticker.Stop()
+
+	for {
+		// Wait for tick or a close.
+		select {
+		case <-p.stopping:
+			return
+		case <-ticker.C:
+		}
+
+		p.pollAll()
+	}
+
+}
+
+// Stop stops the polling routine.
+func (p *Poller) Stop() {
+	close(p.stopping)
+}
+
+func (p *Poller) pollAll() {
+	addrs := p.Addresses()
+
+	ctx := context.Background()
+
+	toRemove := []dax.Address{}
+
+	for _, addr := range addrs {
+		p.logger.Debugf("polling:   %s", addr)
+		start := time.Now()
+		up := p.nodePoller.Poll(addr)
+		if !up {
+			p.logger.Printf("poller removing %s", addr)
+			toRemove = append(toRemove, addr)
+		}
+		p.logger.Debugf("done poll: %s, %s", addr, time.Since(start))
+	}
+
+	if len(toRemove) > 0 {
+		p.logger.Debugf("removing addresses: %v", toRemove)
+		start := time.Now()
+		err := p.addressManager.RemoveAddresses(ctx, toRemove...)
+		if err != nil {
+			p.logger.Printf("removing %s: %v", toRemove, err)
+		}
+		p.logger.Debugf("remove complete: %s", time.Since(start))
+	}
+
+}

dax/mds/poller/poller_test.go

@@ -0,0 +1,182 @@
+package poller_test
+
+import (
+	"context"
+	"encoding/json"
+	"log"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+	"time"
+
+	"github.com/molecula/featurebase/v3/dax"
+	mds_http "github.com/molecula/featurebase/v3/dax/mds/http"
+	"github.com/molecula/featurebase/v3/dax/mds/poller"
+	"github.com/molecula/featurebase/v3/logger"
+	"github.com/stretchr/testify/assert"
+)
+
+// TestPoller runs for a total of 5 seconds. It begins by polling two healthy
+// nodes. After 3 seconds, one of the nodes dies. At that point, the poller
+// de-registers the dead node from the node manager, after which the node
+// manager tells the poller to stop polling the dead node.
+func TestPoller(t *testing.T) {
+	ctx := context.Background()
+
+	// node 1
+	node1 := newMockNode(t, "health", 0)
+	defer node1.Close()
+	addr1 := dax.Address(node1.URL())
+
+	// node 1
+	node2 := newMockNode(t, "health", 3*time.Second)
+	defer node2.Close()
+	addr2 := dax.Address(node2.URL())
+
+	// manager
+	manager := newMockManager(t, ctx, "deregister-nodes", []dax.Address{addr1, addr2})
+	defer manager.Close()
+	managerAddr := dax.Address(manager.URL())
+
+	t.Run("Poller", func(t *testing.T) {
+		cfg := poller.Config{
+			AddressManager: mds_http.NewAddressManager(managerAddr),
+			NodePoller:     poller.NewHTTPNodePoller(logger.NopLogger),
+		}
+		p := poller.New(cfg)
+
+		// This is a little strange, but basically we need the manager to be
+		// able to call poller.RemoveAddresses, and since this test poller isn't
+		// running as an http server (unlike everything else in this test:
+		// manager, nodes), we give the manager a pointer to the Poller here so
+		// it can call the RemoveAddresses method directly.
+		manager.setPoller(p)
+
+		done := make(chan struct{})
+		go func() {
+			time.Sleep(5 * time.Second)
+			close(done)
+		}()
+
+		p.Run()
+		defer p.Stop()
+
+		p.AddAddresses(ctx, addr1, addr2)
+
+		// wait for a done
+		<-done
+
+		assert.Contains(t, p.Addresses(), addr1)
+		assert.NotContains(t, p.Addresses(), addr2)
+	})
+}
+
+///////////////////////////////////////////////////////////////
+
+type mockManager struct {
+	t      *testing.T
+	server *httptest.Server
+
+	poller    *poller.Poller
+	addresses map[dax.Address]struct{}
+}
+
+func newMockManager(t *testing.T, ctx context.Context, deregisterPath string, addrs []dax.Address) *mockManager {
+	addresses := make(map[dax.Address]struct{})
+	for _, addr := range addrs {
+		addresses[addr] = struct{}{}
+	}
+
+	mm := &mockManager{
+		t:         t,
+		addresses: addresses,
+	}
+
+	// deregister is a function used in this mock to remove the address from the
+	// addresses cache in the mock manager, as well as call RemoveAddresses on
+	// the Poller.
+	deregister := func(addrs ...dax.Address) {
+		for _, addr := range addrs {
+			delete(mm.addresses, addr)
+		}
+		mm.poller.RemoveAddresses(ctx, addrs...)
+	}
+
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		assert.Equal(t, "/"+deregisterPath, r.URL.Path)
+		assert.Equal(t, "application/json", r.Header.Get("Content-Type"))
+		assert.Equal(t, "application/json", r.Header.Get("Accept"))
+
+		////// handle payload
+		body := r.Body
+		defer body.Close()
+
+		req := mds_http.DeregisterNodesRequest{}
+		if err := json.NewDecoder(body).Decode(&req); err != nil {
+			http.Error(w, err.Error(), http.StatusBadRequest)
+			return
+		}
+
+		log.Printf("deregister addresses: %s", req.Addresses)
+		deregister(req.Addresses...)
+		//////
+
+		w.WriteHeader(http.StatusOK)
+	}))
+
+	mm.server = server
+	return mm
+}
+
+func (m *mockManager) setPoller(p *poller.Poller) {
+	m.poller = p
+}
+
+func (m *mockManager) URL() string {
+	if m.server != nil {
+		return m.server.URL
+	}
+	return ""
+}
+
+func (m *mockManager) Close() {
+	if m.server != nil {
+		m.server.Close()
+	}
+}
+
+type mockNode struct {
+	t      *testing.T
+	server *httptest.Server
+}
+
+func newMockNode(t *testing.T, healthPath string, dieAfter time.Duration) *mockNode {
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		assert.Equal(t, "/"+healthPath, r.URL.Path)
+		w.WriteHeader(http.StatusOK)
+	}))
+	if dieAfter > 0 {
+		go func() {
+			time.Sleep(dieAfter)
+			log.Printf("stopping node: %s", server.URL)
+			server.Close()
+		}()
+	}
+	return &mockNode{
+		t:      t,
+		server: server,
+	}
+}
+
+func (m *mockNode) URL() string {
+	if m.server != nil {
+		return m.server.URL
+	}
+	return ""
+}
+
+func (m *mockNode) Close() {
+	if m.server != nil {
+		m.server.Close()
+	}
+}

dax/mds/schemar/boltdb/schemar.go

@@ -0,0 +1,386 @@
+// Package boltdb contains the boltdb implementation of the Schemar
+// interfaces.
+package boltdb
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+	"strings"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/boltdb"
+	"github.com/molecula/featurebase/v3/dax/mds/schemar"
+	"github.com/molecula/featurebase/v3/errors"
+	"github.com/molecula/featurebase/v3/logger"
+)
+
+var (
+	bucketSchemar = boltdb.Bucket("schemar")
+)
+
+// SchemarBuckets defines the buckets used by this package. It can be called
+// during setup to create the buckets ahead of time.
+var SchemarBuckets []boltdb.Bucket = []boltdb.Bucket{
+	bucketSchemar,
+}
+
+// Ensure type implements interface.
+var _ schemar.Schemar = (*Schemar)(nil)
+
+type Schemar struct {
+	db *boltdb.DB
+
+	logger logger.Logger
+}
+
+// NewSchemar returns a new instance of Schemar with default values.
+func NewSchemar(db *boltdb.DB, logger logger.Logger) *Schemar {
+	return &Schemar{
+		db:     db,
+		logger: logger,
+	}
+}
+
+// CreateTable creates the table provided. If a table with the same name already
+// exists then an error is returned.
+func (s *Schemar) CreateTable(ctx context.Context, qtbl *dax.QualifiedTable) error {
+	// Ensure the table id is not blank.
+	if qtbl.ID == "" {
+		return schemar.NewErrTableIDInvalid(qtbl.ID)
+	}
+
+	// Ensure the table name is not blank.
+	if qtbl.Name == "" {
+		return schemar.NewErrTableNameInvalid(qtbl.Name)
+	}
+
+	// Ensure that a primary key field is present and valid.
+	if !qtbl.HasValidPrimaryKey() {
+		return schemar.NewErrInvalidPrimaryKey()
+	}
+
+	//////////// end validation
+
+	tx, err := s.db.BeginTx(ctx, true)
+	if err != nil {
+		return errors.Wrap(err, "getting transaction")
+	}
+	defer tx.Rollback()
+
+	// Ensure a table with that ID doesn't already exist.
+	if t, _ := s.tableByID(tx, qtbl.TableQualifier, qtbl.ID); t != nil {
+		return dax.NewErrTableIDExists(qtbl.QualifiedID())
+	}
+
+	if err := s.putTable(tx, qtbl); err != nil {
+		return errors.Wrap(err, "putting table")
+	}
+
+	// In addition to storing the table in tableKey, we want to store a reverse-lookup
+	// (i.e. index) on table name to the tableKey.
+	if err := s.putTableName(tx, qtbl); err != nil {
+		return errors.Wrap(err, "putting table name")
+	}
+
+	return tx.Commit()
+}
+
+// CreateField creates the field provided in the given table. If a field with
+// the same name already exists then an error is returned.
+func (s *Schemar) CreateField(ctx context.Context, qtid dax.QualifiedTableID, fld *dax.Field) error {
+	// Ensure the field name is not blank.
+	if fld.Name == "" {
+		return schemar.NewErrFieldNameInvalid(fld.Name)
+	}
+
+	//////////// end validation
+
+	tx, err := s.db.BeginTx(ctx, true)
+	if err != nil {
+		return errors.Wrap(err, "getting transaction")
+	}
+	defer tx.Rollback()
+
+	// Get the table.
+	qtbl, err := s.tableByQTID(tx, qtid)
+	if err != nil {
+		return errors.Wrap(err, "getting table by id")
+	}
+
+	// Ensure a field with that name doesn't already exist.
+	if _, ok := qtbl.Field(fld.Name); ok {
+		return dax.NewErrFieldExists(fld.Name)
+	}
+
+	qtbl.Fields = append(qtbl.Fields, fld)
+
+	// Write table back to database.
+	if err := s.putTable(tx, qtbl); err != nil {
+		return errors.Wrap(err, "putting table")
+	}
+
+	return tx.Commit()
+}
+
+// DropField removes the field from the table.
+func (s *Schemar) DropField(ctx context.Context, qtid dax.QualifiedTableID, fldName dax.FieldName) error {
+	tx, err := s.db.BeginTx(ctx, true)
+	if err != nil {
+		return errors.Wrap(err, "getting transaction")
+	}
+	defer tx.Rollback()
+
+	// Get the table.
+	qtbl, err := s.tableByQTID(tx, qtid)
+	if err != nil {
+		return errors.Wrap(err, "getting table by id")
+	}
+
+	// Ensure a field with that name exists.
+	if _, ok := qtbl.Field(fldName); !ok {
+		return dax.NewErrFieldDoesNotExist(fldName)
+	}
+
+	_ = qtbl.RemoveField(fldName)
+
+	// Write table back to database.
+	if err := s.putTable(tx, qtbl); err != nil {
+		return errors.Wrap(err, "putting table")
+	}
+
+	return tx.Commit()
+}
+
+func (s *Schemar) putTable(tx *boltdb.Tx, qtbl *dax.QualifiedTable) error {
+	bkt := tx.Bucket(bucketSchemar)
+	if bkt == nil {
+		return errors.Errorf(boltdb.ErrFmtBucketNotFound, bucketSchemar)
+	}
+
+	val, err := json.Marshal(qtbl)
+	if err != nil {
+		return errors.Wrap(err, "marshalling table to json")
+	}
+
+	return bkt.Put(tableKey(qtbl.OrganizationID, qtbl.DatabaseID, qtbl.Table.ID), val)
+}
+
+func (s *Schemar) putTableName(tx *boltdb.Tx, qtbl *dax.QualifiedTable) error {
+	bkt := tx.Bucket(bucketSchemar)
+	if bkt == nil {
+		return errors.Errorf(boltdb.ErrFmtBucketNotFound, bucketSchemar)
+	}
+
+	return bkt.Put(tableNameKey(qtbl.OrganizationID, qtbl.DatabaseID, qtbl.Name), tableKey(qtbl.OrganizationID, qtbl.DatabaseID, qtbl.ID))
+}
+
+// Table returns the TableInfo for the given table. An error is returned if the
+// table does not exist.
+func (s *Schemar) Table(ctx context.Context, qtid dax.QualifiedTableID) (*dax.QualifiedTable, error) {
+	tx, err := s.db.BeginTx(ctx, false)
+	if err != nil {
+		return nil, errors.Wrap(err, "beginning tx")
+	}
+	defer tx.Rollback()
+
+	return s.tableByQTID(tx, qtid)
+}
+
+// tableByQTID gets the full qualified table by the QualifiedTableID whether it has Name or ID set.
+func (s *Schemar) tableByQTID(tx *boltdb.Tx, qtid dax.QualifiedTableID) (*dax.QualifiedTable, error) {
+	if qtid.ID == "" {
+		return s.tableByName(tx, qtid.TableQualifier, qtid.Name)
+	}
+
+	return s.tableByID(tx, qtid.TableQualifier, qtid.ID)
+}
+
+func (s *Schemar) tableByName(tx *boltdb.Tx, qual dax.TableQualifier, name dax.TableName) (*dax.QualifiedTable, error) {
+	qtid, err := s.tableIDByName(tx, qual, name)
+	if err != nil {
+		return nil, errors.Wrap(err, "getting table ID")
+	}
+
+	return s.tableByID(tx, qtid.TableQualifier, qtid.ID) // TODO remove?
+}
+
+func (s *Schemar) tableByID(tx *boltdb.Tx, qual dax.TableQualifier, id dax.TableID) (*dax.QualifiedTable, error) {
+	bkt := tx.Bucket(bucketSchemar)
+	if bkt == nil {
+		return nil, errors.Errorf(boltdb.ErrFmtBucketNotFound, bucketSchemar)
+	}
+
+	b := bkt.Get(tableKey(qual.OrganizationID, qual.DatabaseID, id))
+	if b == nil {
+		return nil, dax.NewErrTableIDDoesNotExist(dax.QualifiedTableID{TableQualifier: qual, ID: id})
+	}
+
+	table := &dax.QualifiedTable{}
+	if err := json.Unmarshal(b, table); err != nil {
+		return nil, errors.Wrap(err, "unmarshalling table json")
+	}
+
+	return table, nil
+}
+
+func (s *Schemar) tableIDByName(tx *boltdb.Tx, qual dax.TableQualifier, name dax.TableName) (dax.QualifiedTableID, error) {
+	bkt := tx.Bucket(bucketSchemar)
+	if bkt == nil {
+		return dax.QualifiedTableID{}, errors.Errorf(boltdb.ErrFmtBucketNotFound, bucketSchemar)
+	}
+
+	b := bkt.Get(tableNameKey(qual.OrganizationID, qual.DatabaseID, name))
+	if b == nil {
+		return dax.QualifiedTableID{}, dax.NewErrTableNameDoesNotExist(name)
+	}
+
+	return keyQualifiedTableID(b)
+}
+
+// Tables returns a list of Table for all existing tables. If one or more table
+// names is provided, then only those will be included in the output.
+func (s *Schemar) Tables(ctx context.Context, qual dax.TableQualifier, ids ...dax.TableID) ([]*dax.QualifiedTable, error) {
+	tx, err := s.db.BeginTx(ctx, false)
+	if err != nil {
+		return nil, errors.Wrap(err, "beginning tx")
+	}
+	defer tx.Rollback()
+
+	return s.getTables(ctx, tx, qual, ids...)
+}
+
+func (s *Schemar) getTables(ctx context.Context, tx *boltdb.Tx, qual dax.TableQualifier, ids ...dax.TableID) (dax.QualifiedTables, error) {
+	c := tx.Bucket(bucketSchemar).Cursor()
+
+	// Deserialize rows into Table objects.
+	tables := make(dax.QualifiedTables, 0)
+
+	var filterByID bool
+	if len(ids) > 0 {
+		filterByID = true
+	}
+
+	prefix := []byte(fmt.Sprintf(prefixFmtTables, qual.OrganizationID, qual.DatabaseID))
+	for k, v := c.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, v = c.Next() {
+		if v == nil {
+			s.logger.Printf("nil value for key: %s", k)
+			continue
+		}
+
+		tblID, err := keyTableID(k)
+		if err != nil {
+			return nil, errors.Wrap(err, "getting table from key")
+		}
+
+		// Only include tables provided in the ids filter.
+		if filterByID && !containsTableID(ids, tblID) {
+			continue
+		}
+
+		table := &dax.QualifiedTable{}
+		if err := json.Unmarshal(v, table); err != nil {
+			return nil, errors.Wrap(err, "unmarshalling table json")
+		}
+
+		tables = append(tables, table)
+	}
+
+	return tables, nil
+}
+
+func containsTableID(s []dax.TableID, e dax.TableID) bool {
+	for _, a := range s {
+		if a == e {
+			return true
+		}
+	}
+	return false
+}
+
+// DropTable drops the given table. If the named/IDed table does not exist
+// then an error is returned.
+func (s *Schemar) DropTable(ctx context.Context, qtid dax.QualifiedTableID) error {
+	tx, err := s.db.BeginTx(ctx, true)
+	if err != nil {
+		return errors.Wrap(err, "getting transaction")
+	}
+	defer tx.Rollback()
+
+	// Ensure the table exists.
+	qtbl, err := s.tableByQTID(tx, qtid)
+	if err != nil {
+		return errors.Wrap(err, "getting table by id")
+	}
+
+	bkt := tx.Bucket(bucketSchemar)
+	if bkt == nil {
+		return errors.Errorf(boltdb.ErrFmtBucketNotFound, bucketSchemar)
+	}
+
+	// Delete the table by ID.
+	if err := bkt.Delete(tableKey(qtbl.OrganizationID, qtbl.DatabaseID, qtbl.ID)); err != nil {
+		return errors.Wrap(err, "deleting table by id")
+	}
+
+	// Delete the reverse-lookup table by Name.
+	if err := bkt.Delete(tableNameKey(qtbl.OrganizationID, qtbl.DatabaseID, qtbl.Name)); err != nil {
+		return errors.Wrap(err, "deleting table by name")
+	}
+
+	return tx.Commit()
+}
+
+const (
+	prefixFmtTables     = "tables/%s/%s/"
+	prefixFmtTableNames = "tablenames/%s/%s/"
+)
+
+// tableKey returns a key based on a qualified table ID.
+func tableKey(orgID dax.OrganizationID, dbID dax.DatabaseID, tblID dax.TableID) []byte {
+	key := fmt.Sprintf(prefixFmtTables+"%s", orgID, dbID, tblID)
+	return []byte(key)
+}
+
+// tableNameKey returns a key based on a qualified table name.
+func tableNameKey(orgID dax.OrganizationID, dbID dax.DatabaseID, name dax.TableName) []byte {
+	key := fmt.Sprintf(prefixFmtTableNames+"%s", orgID, dbID, name)
+	return []byte(key)
+}
+
+// keyTableID gets the TableID out of the key.
+func keyTableID(key []byte) (dax.TableID, error) {
+	parts := strings.Split(string(key), "/")
+	if len(parts) != 4 {
+		return "", errors.New(errors.ErrUncoded, "table key format expected: `tables/orgID/dbID/tblID`")
+	}
+
+	return dax.TableID(parts[3]), nil
+}
+
+// keyQualifedTableID gets the QualifiedTableID out of the key.
+func keyQualifiedTableID(key []byte) (dax.QualifiedTableID, error) {
+	parts := strings.Split(string(key), "/")
+	if len(parts) != 4 {
+		return dax.QualifiedTableID{}, errors.New(errors.ErrUncoded, "table key format expected: `tables/orgID/dbID/tblID`")
+	}
+
+	return dax.NewQualifiedTableID(
+		dax.NewTableQualifier(
+			dax.OrganizationID(parts[1]),
+			dax.DatabaseID(parts[2]),
+		),
+		dax.TableID(parts[3]),
+	), nil
+}
+
+func (s *Schemar) TableID(ctx context.Context, qual dax.TableQualifier, name dax.TableName) (dax.QualifiedTableID, error) {
+	tx, err := s.db.BeginTx(ctx, false)
+	if err != nil {
+		return dax.QualifiedTableID{}, err
+	}
+	defer tx.Rollback()
+
+	return s.tableIDByName(tx, qual, name)
+}

dax/mds/schemar/boltdb/schemar_test.go

@@ -0,0 +1,146 @@
+package boltdb_test
+
+import (
+	"context"
+	"testing"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/mds/schemar/boltdb"
+	daxtest "github.com/molecula/featurebase/v3/dax/test"
+	testbolt "github.com/molecula/featurebase/v3/dax/test/boltdb"
+	"github.com/molecula/featurebase/v3/errors"
+	"github.com/molecula/featurebase/v3/logger"
+	"github.com/stretchr/testify/assert"
+)
+
+func TestSchemar(t *testing.T) {
+	orgID := dax.OrganizationID("acme")
+	dbID := dax.DatabaseID("db1")
+	invalidTableID := dax.TableID("invalidID")
+	tableName := dax.TableName("foo")
+	tableName0 := dax.TableName("foo")
+	tableName1 := dax.TableName("bar")
+	tableID0 := "2"
+	tableID1 := "1"
+	partitionN := 12
+
+	ctx := context.Background()
+	qual := dax.NewTableQualifier(orgID, dbID)
+
+	db := testbolt.MustOpenDB(t)
+	defer testbolt.MustCloseDB(t, db)
+
+	t.Cleanup(func() {
+		testbolt.CleanupDB(t, db.Path())
+	})
+
+	// Initialize the buckets.
+	assert.NoError(t, db.InitializeBuckets(boltdb.SchemarBuckets...))
+
+	t.Run("NewSchemar", func(t *testing.T) {
+		s := boltdb.NewSchemar(db, logger.NopLogger)
+
+		// Add new table.
+		tbl := dax.NewTable(tableName)
+		tbl.CreateID()
+		tbl.Fields = []*dax.Field{
+			{
+				Name: dax.PrimaryKeyFieldName,
+				Type: dax.FieldTypeString,
+			},
+			{
+				Name: "intField",
+				Type: dax.FieldTypeInt,
+			},
+		}
+		qtbl := dax.NewQualifiedTable(qual, tbl)
+		assert.NoError(t, s.CreateTable(ctx, qtbl))
+
+		// Try adding the table again.
+		err := s.CreateTable(ctx, qtbl)
+		if assert.Error(t, err) {
+			assert.True(t, errors.Is(err, dax.ErrTableIDExists))
+		}
+
+		qtid := qtbl.QualifiedID()
+
+		// Get the table.
+		{
+			tbl, err := s.Table(ctx, qtid)
+			assert.NoError(t, err)
+			assert.Equal(t, tableName, tbl.Name)
+		}
+
+		// Drop the table.
+		assert.NoError(t, s.DropTable(ctx, qtid))
+
+		// Make sure the reverse-lookup (table by name) was dropped as well.
+		{
+			_, err := s.TableID(ctx, qual, tableName)
+			if assert.Error(t, err) {
+				assert.True(t, errors.Is(err, dax.ErrTableNameDoesNotExist))
+			}
+		}
+
+		// Try adding the table (i.e. the same table name) again.
+		assert.NoError(t, s.CreateTable(ctx, qtbl))
+
+		// Drop the table again.
+		assert.NoError(t, s.DropTable(ctx, qtid))
+
+		// Drop invalid table.
+		{
+			iqtid := dax.NewQualifiedTableID(qual, invalidTableID)
+			err := s.DropTable(ctx, iqtid)
+			if assert.Error(t, err) {
+				assert.True(t, errors.Is(err, dax.ErrTableIDDoesNotExist))
+			}
+		}
+	})
+
+	t.Run("GetTables", func(t *testing.T) {
+		s := boltdb.NewSchemar(db, logger.NopLogger)
+
+		exp := []*dax.QualifiedTable{}
+		tables, err := s.Tables(ctx, qual)
+		assert.NoError(t, err)
+		assert.Equal(t, exp, tables)
+
+		qtbl0 := daxtest.TestQualifiedTableWithID(t, qual, tableID0, tableName0, partitionN, false)
+		qtbl1 := daxtest.TestQualifiedTableWithID(t, qual, tableID1, tableName1, partitionN, false)
+
+		// Add a couple of tables.
+		assert.NoError(t, s.CreateTable(ctx, qtbl0))
+		assert.NoError(t, s.CreateTable(ctx, qtbl1))
+
+		exp = []*dax.QualifiedTable{
+			qtbl1,
+			qtbl0,
+		}
+
+		// All tables.
+		tables, err = s.Tables(ctx, qual)
+		assert.NoError(t, err)
+		assert.Equal(t, exp, tables)
+
+		// With a valid filter.
+		tables, err = s.Tables(ctx, qual, qtbl0.ID)
+		assert.NoError(t, err)
+		assert.Equal(t, exp[1:], tables)
+
+		// With an invalid filter.
+		tables, err = s.Tables(ctx, qual, invalidTableID)
+		assert.NoError(t, err)
+		assert.Equal(t, exp[0:0], tables)
+
+		// With both valid and invalid filters.
+		tables, err = s.Tables(ctx, qual, qtbl0.ID, invalidTableID)
+		assert.NoError(t, err)
+		assert.Equal(t, exp[1:], tables)
+
+		// With all valid filters.
+		tables, err = s.Tables(ctx, qual, qtbl0.ID, qtbl1.ID)
+		assert.NoError(t, err)
+		assert.Equal(t, exp, tables)
+	})
+}

dax/mds/schemar/errors.go

@@ -0,0 +1,44 @@
+package schemar
+
+import (
+	"fmt"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/errors"
+)
+
+const (
+	ErrCodeTableIDInvalid    errors.Code = "TableIDInvalid"
+	ErrCodeTableNameInvalid  errors.Code = "TableNameInvalid"
+	ErrCodeInvalidPrimaryKey errors.Code = "InvalidPrimaryKey"
+
+	ErrCodeFieldNameInvalid errors.Code = "FieldNameInvalid"
+)
+
+func NewErrTableIDInvalid(tableID dax.TableID) error {
+	return errors.New(
+		ErrCodeTableIDInvalid,
+		fmt.Sprintf("table ID '%s' is invalid", tableID),
+	)
+}
+
+func NewErrTableNameInvalid(tableName dax.TableName) error {
+	return errors.New(
+		ErrCodeTableNameInvalid,
+		fmt.Sprintf("table name '%s' is invalid", tableName),
+	)
+}
+
+func NewErrInvalidPrimaryKey() error {
+	return errors.New(
+		ErrCodeInvalidPrimaryKey,
+		"invalid primary key",
+	)
+}
+
+func NewErrFieldNameInvalid(fieldName dax.FieldName) error {
+	return errors.New(
+		ErrCodeFieldNameInvalid,
+		fmt.Sprintf("field name '%s' is invalid", fieldName),
+	)
+}

dax/mds/schemar/http/handler.go

@@ -0,0 +1,199 @@
+package http
+
+import (
+	"encoding/json"
+	"net/http"
+
+	"github.com/gorilla/mux"
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/mds/schemar"
+)
+
+func Handler(s schemar.Schemar) http.Handler {
+	svr := &server{
+		schemar: s,
+	}
+
+	router := mux.NewRouter()
+	router.HandleFunc("/health", svr.getHealth).Methods("GET").Name("GetHealth")
+	router.HandleFunc("/create-table", svr.postCreateTable).Methods("POST").Name("PostCreateTable")
+	router.HandleFunc("/drop-table", svr.postDropTable).Methods("POST").Name("PostDropTable")
+	router.HandleFunc("/table", svr.postTable).Methods("POST").Name("PostTable")
+	router.HandleFunc("/tables", svr.postTables).Methods("POST").Name("PostTables")
+	return router
+}
+
+type server struct {
+	schemar schemar.Schemar
+}
+
+// GET /health
+func (s *server) getHealth(w http.ResponseWriter, r *http.Request) {
+	w.WriteHeader(http.StatusOK)
+}
+
+// POST /create-table
+func (s *server) postCreateTable(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := &dax.QualifiedTable{}
+	if err := json.NewDecoder(body).Decode(req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	err := s.schemar.CreateTable(ctx, req)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	resp := struct{}{}
+	if err := json.NewEncoder(w).Encode(resp); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+}
+
+// POST /drop-table
+func (s *server) postDropTable(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := DropTableRequest{}
+	if err := json.NewDecoder(body).Decode(&req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	qtid := req.TableKey.QualifiedTableID()
+
+	err := s.schemar.DropTable(ctx, qtid)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	resp := struct{}{}
+	if err := json.NewEncoder(w).Encode(resp); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+}
+
+// // POST /create-field
+// func (s *server) postCreateField(w http.ResponseWriter, r *http.Request) {
+// 	body := r.Body
+// 	defer body.Close()
+
+// 	req := mds.CreateFieldRequest{}
+// 	if err := json.NewDecoder(body).Decode(&req); err != nil {
+// 		http.Error(w, err.Error(), http.StatusBadRequest)
+// 		return
+// 	}
+
+// 	resp, err := s.mds.CreateField(req)
+// 	if err != nil {
+// 		http.Error(w, err.Error(), http.StatusBadRequest)
+// 		return
+// 	}
+
+// 	if err := json.NewEncoder(w).Encode(resp); err != nil {
+// 		http.Error(w, err.Error(), http.StatusBadRequest)
+// 		return
+// 	}
+// }
+
+// // POST /drop-field
+// func (s *server) postDropField(w http.ResponseWriter, r *http.Request) {
+// 	body := r.Body
+// 	defer body.Close()
+
+// 	req := mds.DropFieldRequest{}
+// 	if err := json.NewDecoder(body).Decode(&req); err != nil {
+// 		http.Error(w, err.Error(), http.StatusBadRequest)
+// 		return
+// 	}
+
+// 	resp, err := s.mds.DropField(req)
+// 	if err != nil {
+// 		http.Error(w, err.Error(), http.StatusBadRequest)
+// 		return
+// 	}
+
+// 	if err := json.NewEncoder(w).Encode(resp); err != nil {
+// 		http.Error(w, err.Error(), http.StatusBadRequest)
+// 		return
+// 	}
+// }
+
+// POST /table
+func (s *server) postTable(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := TableRequest{}
+	if err := json.NewDecoder(body).Decode(&req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	qtid := req.TableKey.QualifiedTableID()
+
+	resp, err := s.schemar.Table(ctx, qtid)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	if err := json.NewEncoder(w).Encode(resp); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+}
+
+// POST /tables
+func (s *server) postTables(w http.ResponseWriter, r *http.Request) {
+	body := r.Body
+	defer body.Close()
+
+	ctx := r.Context()
+
+	req := TablesRequest{}
+	if err := json.NewDecoder(body).Decode(&req); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+	qual := dax.NewTableQualifier(req.OrganizationID, req.DatabaseID)
+	resp, err := s.schemar.Tables(ctx, qual, req.TableIDs...)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	if err := json.NewEncoder(w).Encode(resp); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+}
+
+type DropTableRequest struct {
+	TableKey dax.TableKey `json:"table-key"`
+}
+
+type TableRequest struct {
+	TableKey dax.TableKey `json:"table-key"`
+}
+
+type TablesRequest struct {
+	OrganizationID dax.OrganizationID `json:"org-id"`
+	DatabaseID     dax.DatabaseID     `json:"db-id"`
+	TableIDs       dax.TableIDs       `json:"table-ids"`
+}

dax/mds/schemar/schemar.go

@@ -0,0 +1,54 @@
+// Package schemar provides the core Schemar interface.
+package schemar
+
+import (
+	"context"
+
+	"github.com/molecula/featurebase/v3/dax"
+)
+
+type Schemar interface {
+	CreateTable(context.Context, *dax.QualifiedTable) error
+	DropTable(context.Context, dax.QualifiedTableID) error
+	CreateField(context.Context, dax.QualifiedTableID, *dax.Field) error
+	DropField(context.Context, dax.QualifiedTableID, dax.FieldName) error
+	Table(context.Context, dax.QualifiedTableID) (*dax.QualifiedTable, error)
+	Tables(context.Context, dax.TableQualifier, ...dax.TableID) ([]*dax.QualifiedTable, error)
+
+	// TableID is a reverse-lookup method to get the TableID for a given
+	// qualified TableName.
+	TableID(context.Context, dax.TableQualifier, dax.TableName) (dax.QualifiedTableID, error)
+}
+
+//////////////////////////////////////////////
+
+// Ensure type implements interface.
+var _ Schemar = &NopSchemar{}
+
+// NopSchemar is a no-op implementation of the Schemar interface.
+type NopSchemar struct{}
+
+func NewNopSchemar() *NopSchemar {
+	return &NopSchemar{}
+}
+
+func (s *NopSchemar) CreateTable(ctx context.Context, qtbl *dax.QualifiedTable) error { return nil }
+func (s *NopSchemar) DropTable(ctx context.Context, qtid dax.QualifiedTableID) error {
+	return nil
+}
+func (s *NopSchemar) CreateField(ctx context.Context, qtid dax.QualifiedTableID, fld *dax.Field) error {
+	return nil
+}
+func (s *NopSchemar) DropField(ctx context.Context, qtid dax.QualifiedTableID, fld dax.FieldName) error {
+	return nil
+}
+func (s *NopSchemar) Table(ctx context.Context, qtid dax.QualifiedTableID) (*dax.QualifiedTable, error) {
+	return nil, nil
+}
+func (s *NopSchemar) Tables(ctx context.Context, qual dax.TableQualifier, ids ...dax.TableID) ([]*dax.QualifiedTable, error) {
+	return []*dax.QualifiedTable{}, nil
+}
+
+func (s *NopSchemar) TableID(context.Context, dax.TableQualifier, dax.TableName) (dax.QualifiedTableID, error) {
+	return dax.QualifiedTableID{}, nil
+}

dax/mds/schemar/schemar_test.go

@@ -0,0 +1,124 @@
+package schemar_test
+
+import (
+	"context"
+	"testing"
+
+	"github.com/molecula/featurebase/v3/dax"
+	daxtest "github.com/molecula/featurebase/v3/dax/test"
+	"github.com/molecula/featurebase/v3/errors"
+	"github.com/stretchr/testify/assert"
+)
+
+func TestSchemar(t *testing.T) {
+	orgID := dax.OrganizationID("acme")
+	dbID := dax.DatabaseID("db1")
+	invalidTableID := dax.TableID("invalidID")
+	tableName := dax.TableName("foo")
+	tableName0 := dax.TableName("foo")
+	tableName1 := dax.TableName("bar")
+	tableID0 := "2"
+	tableID1 := "1"
+	partitionN := 12
+
+	ctx := context.Background()
+	qual := dax.NewTableQualifier(orgID, dbID)
+
+	t.Run("NewSchemar", func(t *testing.T) {
+		s, cleanup := daxtest.NewSchemar(t)
+		defer cleanup()
+
+		// Add new table.
+		tbl := dax.NewTable(tableName)
+		tbl.Fields = []*dax.Field{
+			{
+				Name: dax.PrimaryKeyFieldName,
+				Type: dax.FieldTypeString,
+			},
+			{
+				Name: "intField",
+				Type: dax.FieldTypeInt,
+			},
+		}
+		qtbl := dax.NewQualifiedTable(qual, tbl)
+		qtbl.CreateID()
+		assert.NoError(t, s.CreateTable(ctx, qtbl))
+
+		// Try adding the table again.
+		err := s.CreateTable(ctx, qtbl)
+		if assert.Error(t, err) {
+			assert.True(t, errors.Is(err, dax.ErrTableIDExists))
+		}
+
+		qtid := qtbl.QualifiedID()
+
+		// Get the table.
+		{
+			tbl, err := s.Table(ctx, qtid)
+			assert.NoError(t, err)
+			assert.Equal(t, tableName, tbl.Name)
+		}
+
+		// Drop the table.
+		{
+			err := s.DropTable(ctx, qtid)
+			assert.NoError(t, err)
+		}
+
+		// Drop invalid table.
+		{
+			iqtid := dax.NewQualifiedTableID(qual, invalidTableID)
+			err := s.DropTable(ctx, iqtid)
+			if assert.Error(t, err) {
+				assert.True(t, errors.Is(err, dax.ErrTableIDDoesNotExist))
+			}
+		}
+	})
+
+	t.Run("GetTables", func(t *testing.T) {
+		s, cleanup := daxtest.NewSchemar(t)
+		defer cleanup()
+
+		exp := []*dax.QualifiedTable{}
+		tables, err := s.Tables(ctx, qual)
+		assert.NoError(t, err)
+		assert.Equal(t, exp, tables)
+
+		qtbl0 := daxtest.TestQualifiedTableWithID(t, qual, tableID0, tableName0, partitionN, false)
+		qtbl1 := daxtest.TestQualifiedTableWithID(t, qual, tableID1, tableName1, partitionN, false)
+
+		// Add a couple of tables.
+		assert.NoError(t, s.CreateTable(ctx, qtbl0))
+		assert.NoError(t, s.CreateTable(ctx, qtbl1))
+
+		exp = []*dax.QualifiedTable{
+			qtbl1,
+			qtbl0,
+		}
+
+		// All tables.
+		tables, err = s.Tables(ctx, qual)
+		assert.NoError(t, err)
+		assert.Equal(t, exp, tables)
+
+		// With a valid filter.
+		tables, err = s.Tables(ctx, qual, qtbl0.ID)
+		assert.NoError(t, err)
+		assert.Equal(t, exp[1:], tables)
+
+		// With an invalid filter.
+		tables, err = s.Tables(ctx, qual, invalidTableID)
+		assert.NoError(t, err)
+		assert.Equal(t, exp[0:0], tables)
+
+		// With both valid and invalid filters.
+		tables, err = s.Tables(ctx, qual, qtbl0.ID, invalidTableID)
+		assert.NoError(t, err)
+		assert.Equal(t, exp[1:], tables)
+
+		// With all valid filters.
+		tables, err = s.Tables(ctx, qual, qtbl0.ID, qtbl1.ID)
+		assert.NoError(t, err)
+		assert.Equal(t, exp, tables)
+	})
+}

dax/node.go

@@ -0,0 +1,45 @@
+package dax
+
+import (
+	"context"
+	"fmt"
+
+	"github.com/molecula/featurebase/v3/errors"
+)
+
+// Node is used in API requests, like RegisterNode (before being assigned
+// roles).
+type Node struct {
+	Address Address `json:"address"`
+
+	RoleTypes []RoleType `json:"role-types"`
+}
+
+// AssignedNode is used in API responses.
+type AssignedNode struct {
+	Address Address `json:"address"`
+	Role    Role    `json:"role"`
+}
+
+// NodeService represents a service for managing Nodes.
+type NodeService interface {
+	CreateNode(context.Context, Address, *Node) error
+	ReadNode(context.Context, Address) (*Node, error)
+	DeleteNode(context.Context, Address) error
+	Nodes(context.Context) ([]*Node, error)
+}
+
+////////////////////////////////////////////////////
+// Errors
+////////////////////////////////////////////////////
+
+const (
+	ErrNodeDoesNotExist errors.Code = "NodeDoesNotExist"
+)
+
+func NewErrNodeDoesNotExist(addr Address) error {
+	return errors.New(
+		ErrNodeDoesNotExist,
+		fmt.Sprintf("node '%s' does not exist", addr),
+	)
+}

dax/partition.go

@@ -0,0 +1,65 @@
+package dax
+
+import "fmt"
+
+// PartitionNum is the numerical (int) partition value.
+type PartitionNum int
+
+// PartitionNums is a slice of PartitionNum.
+type PartitionNums []PartitionNum
+
+// String returns the PartitionNum as a string.
+func (p PartitionNum) String() string {
+	return fmt.Sprintf("%d", p)
+}
+
+// Partition is a versioned partition.
+type Partition struct {
+	Num     PartitionNum `json:"num"`
+	Version int          `json:"version"`
+}
+
+// NewPartition returns a Partition with the provided num and version.
+func NewPartition(num PartitionNum, version int) Partition {
+	return Partition{
+		Num:     num,
+		Version: version,
+	}
+}
+
+// String returns the Partition (i.e. its Num and Version) as a string.
+func (p Partition) String() string {
+	return fmt.Sprintf("%d.%d", p.Num, p.Version)
+}
+
+// Partitions is a sortable slice of Partition.
+type Partitions []Partition
+
+func (p Partitions) Len() int           { return len(p) }
+func (p Partitions) Less(i, j int) bool { return p[i].Num < p[j].Num }
+func (p Partitions) Swap(i, j int)      { p[i], p[j] = p[j], p[i] }
+
+// NewPartitions returns the provided list of partition nums as a list of
+// Partition with an invalid version (-1). This is to use for cases where the
+// request should not be aware of a partition versioning.
+func NewPartitions(partitionNums ...PartitionNum) Partitions {
+	pvs := make(Partitions, len(partitionNums))
+
+	for i := range partitionNums {
+		pvs[i] = Partition{
+			Num:     partitionNums[i],
+			Version: -1,
+		}
+	}
+
+	return pvs
+}
+
+// Nums returns a slice of all the partition numbers in Partitions.
+func (p Partitions) Nums() []PartitionNum {
+	pp := make([]PartitionNum, len(p))
+	for i := range p {
+		pp[i] = p[i].Num
+	}
+	return pp
+}

dax/queryer/alpha/router.go

@@ -0,0 +1,36 @@
+package alpha
+
+import (
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/queryer"
+	"github.com/molecula/featurebase/v3/errors"
+	featurebaseserver "github.com/molecula/featurebase/v3/server"
+)
+
+// Ensure type implements interface.
+var _ queryer.Router = (*Router)(nil)
+
+type Router struct {
+	computers map[dax.Address]*featurebaseserver.Command
+}
+
+func NewRouter() *Router {
+	return &Router{
+		computers: make(map[dax.Address]*featurebaseserver.Command),
+	}
+}
+
+func (r *Router) AddCmd(addr dax.Address, cmd *featurebaseserver.Command) error {
+	if cmd == nil {
+		return errors.New(errors.ErrUncoded, "cannot add nil cmd to director")
+	}
+	r.computers[addr] = cmd
+	return nil
+}
+
+func (r *Router) Importer(addr dax.Address) queryer.Importer {
+	if cmd, found := r.computers[addr]; found {
+		return queryer.NewFeatureBaseImporter(cmd.API)
+	}
+	return nil
+}

dax/queryer/api/openapi.yaml

@@ -0,0 +1,99 @@
+openapi: 3.0.3
+
+info:
+  title: Queryer
+  description: The query layer of the DAX architecture.
+  version: 0.0.0
+
+paths:
+  /queryer/health:
+    get:
+      summary: Health check endpoint.
+      description: Provides an endpoint to check the overall health of the Queryer service.
+      operationId: GetHealth
+      responses:
+        200:
+          description: Service is healthy.
+
+  /queryer/query:
+    post:
+      summary: Execute either a PQL or SQL command.
+      description: Executes the given PQL or SQL command based on input, and returns the results in a standard format.
+      operationId: PostQuery
+      requestBody:
+        content:
+          application/json:
+            examples:
+              pql:
+                summary: Query via PQL
+                value:
+                  table: tbl
+                  pql: Row(fld=1)
+              sql:
+                summary: Query via SQL
+                value:
+                  sql: SELECT * from tbl
+            schema:
+              type: object
+              properties:
+                table:
+                  type: string
+                pql:
+                  type: string
+                sql:
+                  type: string
+      responses:
+        200:
+          $ref: '#/components/responses/QueryResponse'
+
+  /queryer/sql:
+    post:
+      summary: Execute a SQL command.
+      description: Executes the given SQL command, and returns the results in a standard format.
+      operationId: PostSQL
+      requestBody:
+        content:
+          text/plain:
+            example: SELECT * FROM tbl
+            schema:
+              type: string
+      responses:
+        200:
+          $ref: '#/components/responses/QueryResponse'
+
+components:
+  responses:
+    QueryResponse:
+      description: Standard tabular response with optional error and warnings.
+      content:
+        application/json:
+          schema:
+            $ref: '#/components/schemas/QueryResult'
+  schemas:
+    QueryResult:
+      type: object
+      properties:
+        schema:
+          type: array # fields
+          items:
+            type: object  # field
+            properties:
+              name:
+                type: string # column name
+              type:
+                type: string # column type
+        data:
+          type: array
+          items:
+            type: array
+            items:
+              type: string # this could really be any type; interface{}
+        error:
+          type: string
+        warnings:
+          type: array
+          items:
+            type: string
+        exec_time:
+          type: integer
+          format: int64