Introduce ServiceManager and Refactor DAX Integration tests (#2320)

* Introduce ServiceManager and Refactor DAX Integration tests

The ServiceManager provides an interface with which to manage
featurebase (dax) services (mds, queryer, computer). It replaces the
confusing interface implementations in /dax/server/server.go (which
optionally used pointers to in-process objects to satisfy an interface)
with (for now) http implementations. The thought is that even if we're
running all services in-process, we should communicate between services
over http in order to mirror what we would do in a production
environment where the services are running on different nodes.

This batch of commits does quit a lot, most of which is captured here:

- Added `path` support to `dax.Address`. Address is now a string of the form [scheme]://[host]:[port]/[path].
- Added `Holder.directiveApplied` to determine (in tests) if the computer has completed applying the latest directive. This is somewhat temporary until we improve the mds-to-computer logic.
- Removed the "service prefix" code which was prepending client URL paths with the prefix. Instead, the serviceType (mds, queryer, computer[n] is now part of `dax.Address`).
- Removed, from the dax config, the top level `StorageMethod` and `StorageDSN` and now just have `MDS.Config.DataDir`.
- Added `Computer.Config.N` to specify the number of computers to run in-process.
- Moved the `pilosa.MDS` interface to `computer.Registrar`. This is an example of getting the interfaces defined in the right packages.
- Added `SnapshotTable()` method to the mds client (to align with its API).
- Changed `Balancer.AddJob()` to `Balancer.AddJobs()` to support, for example, adding 256 partitions in a single call. Refactored some of the naive Balancer to account for this.
- Added a `Seed` to the top-level config. It's not really useful because of package `crypto/rand`.
- Added an in-memory implementation of the DisCo interface and disabled etcd in a computer service.
- Create sepearte data-dirs for each in-process computer.
- Disabled grpc in dax.
- Modified the sql3 test definition format to support multiple insert steps and separate query results (to align with those steps).

* Changes necessary to get multiple computer instance running in-process

For now the config looks like this:

```
[computer]
run = true
n = 4
```

but we can probably just change that to be something like:

```
[computer]
run = 4
```

*Issues found running multiple "computers" in-process*
- grpc was trying to bind on the same port
  - changed GRPCListener from `*net.TCPListener` to `net.Listener`
  - created a nopListener and set to that for now (i.e. disabled grpc)
- etcd was starting more than once
  - changed dax to use in-memory implementations of the disco interfaces (i.e. stop using etcd)
- IDAllocator (which uses boltdb) was trying to open the `idalloc.db` file more than once
  - realized we have to set separate data-dirs for each holder. that fixed it.

* Port dax integration tests to ManagedCommand

* Modify Balancer-related methods like AddJob to AddJobs

There were (and still are) a lot of places where we were adding on job
at a time, even when we had a long list of jobs to add. This resulted in
every job add (for example adding 1 of 256 shards) taking ~40ms, or over
10s to create a keyed table. One reason was because each job add was
making multiple boltdb transactions.

* Port over more dax integration test stuff

* Add DirectiveApplied to signify that snapshot/writes have loaded.

We use this in tests to avoid using sleeps.
This should be considered temporary; we're going to need a more robust
solution for determining when a computer node is ready to serve complete
data.

* Finish porting dax integration tests

* Improve godocs

* Remove docker-based DAX integration tests.

* go mod tidy

* Move test/managed.go to avoid package conflicts

* Modify IDK integration tests to work with ServiceManager changes

This is really just computer -> computer0
And the MDS DataDir config change.

* cleanup found during review

* echo $CI_COMMIT_REF_SLUG in CI

* remove docker image arg, use build instead

(cherry picked from commit 2843f218bc)

api.go

@@ -3091,10 +3091,19 @@ func (api *API) RBFDebugInfo() map[string]*rbf.DebugInfo {
 	return infos
 }
 
+// Directive applies the provided Directive to the local computer.
 func (api *API) Directive(ctx context.Context, d *dax.Directive) error {
 	return api.ApplyDirective(ctx, d)
 }
 
+// DirectiveApplied returns true if the computer's current Directive has been
+// applied and is ready to be queried. This it temporary (primarily for tests)
+// and needs to be refactored as we improve the logic around mds-to-computer
+// communication.
+func (api *API) DirectiveApplied(ctx context.Context) (bool, error) {
+	return api.holder.DirectiveApplied(), nil
+}
+
 // SnapshotShardData triggers the node to perform a shard snapshot based on the
 // provided SnapshotShardDataRequest.
 func (api *API) SnapshotShardData(ctx context.Context, req *dax.SnapshotShardDataRequest) error {
@@ -3135,6 +3144,7 @@ func (api *API) SnapshotShardData(ctx context.Context, req *dax.SnapshotShardDat
 
 	// Update the cached directive on the holder.
 	api.holder.SetDirective(&req.Directive)
+	api.holder.SetDirectiveApplied(true)
 
 	// Finally, delete the log file for the previous version.
 	return api.writeLogWriter.DeleteShard(ctx, qtid, partitionNum, req.ShardNum, req.FromVersion)
@@ -3183,6 +3193,7 @@ func (api *API) SnapshotTableKeys(ctx context.Context, req *dax.SnapshotTableKey
 
 	// Update the cached directive on the holder.
 	api.holder.SetDirective(&req.Directive)
+	api.holder.SetDirectiveApplied(true)
 
 	// Finally, delete the log file for the previous version.
 	return api.writeLogWriter.DeleteTableKeys(ctx, qtid, req.PartitionNum, req.FromVersion)
@@ -3224,6 +3235,7 @@ func (api *API) SnapshotFieldKeys(ctx context.Context, req *dax.SnapshotFieldKey
 
 	// Update the cached directive on the holder.
 	api.holder.SetDirective(&req.Directive)
+	api.holder.SetDirectiveApplied(true)
 
 	// Finally, delete the log file for the previous version.
 	return api.writeLogWriter.DeleteFieldKeys(ctx, qtid, req.Field, req.FromVersion)

api_directive.go

@@ -66,6 +66,7 @@ func (api *API) ApplyDirective(ctx context.Context, d *dax.Directive) error {
 	// TODO(tlt): despite what this comment says, this logic is not sound; we
 	// shouldn't be setting the directive until enactiveDirective() succeeds.
 	api.holder.SetDirective(d)
+	defer api.holder.SetDirectiveApplied(true)
 
 	return api.enactDirective(ctx, &previousDirective, d)
 }

client/client.go

@@ -311,7 +311,7 @@ func (c *Client) Query(query PQLQuery, options ...interface{}) (*QueryResponse,
 	if err != nil {
 		return nil, errors.Wrap(err, "making request data")
 	}
-	path := fmt.Sprintf("%s/index/%s/query", c.prefix(), query.Index().name)
+	path := fmt.Sprintf("/index/%s/query", query.Index().name)
 	_, respData, err := c.HTTPRequest("POST", path, reqData, c.augmentHeaders(defaultProtobufHeaders()))
 	if err != nil {
 		return nil, err
@@ -334,7 +334,7 @@ func (c *Client) CreateIndex(index *Index) error {
 	defer span.Finish()
 
 	data := []byte(index.options.String())
-	path := fmt.Sprintf("%s/index/%s", c.prefix(), index.name)
+	path := fmt.Sprintf("/index/%s", index.name)
 	status, body, err := c.HTTPRequest("POST", path, data, c.augmentHeaders(nil))
 	if err != nil {
 		return errors.Wrapf(err, "creating index: %s", index.name)
@@ -358,7 +358,7 @@ func (c *Client) CreateField(field *Field) error {
 	defer span.Finish()
 
 	data := []byte(field.options.String())
-	path := fmt.Sprintf("%s/index/%s/field/%s", c.prefix(), field.index.name, field.name)
+	path := fmt.Sprintf("/index/%s/field/%s", field.index.name, field.name)
 	status, body, err := c.HTTPRequest("POST", path, data, c.augmentHeaders(nil))
 	if err != nil {
 		return errors.Wrapf(err, "creating field: %s in index: %s", field.name, field.index.name)
@@ -426,7 +426,7 @@ func (c *Client) DeleteIndexByName(index string) error {
 	span := c.tracer.StartSpan("Client.DeleteIndex")
 	defer span.Finish()
 
-	path := fmt.Sprintf("%s/index/%s", c.prefix(), index)
+	path := fmt.Sprintf("/index/%s", index)
 	_, _, err := c.HTTPRequest("DELETE", path, nil, c.augmentHeaders(nil))
 	return err
 }
@@ -436,7 +436,7 @@ func (c *Client) DeleteField(field *Field) error {
 	span := c.tracer.StartSpan("Client.DeleteField")
 	defer span.Finish()
 
-	path := fmt.Sprintf("%s/index/%s/field/%s", c.prefix(), field.index.name, field.name)
+	path := fmt.Sprintf("/index/%s/field/%s", field.index.name, field.name)
 	_, _, err := c.HTTPRequest("DELETE", path, nil, c.augmentHeaders(nil))
 	return err
 }
@@ -547,7 +547,7 @@ func (c *Client) EncodeImport(field *Field, shard uint64, vals, ids []uint64, cl
 	if err != nil {
 		return "", nil, errors.Wrap(err, "marshaling Import to protobuf")
 	}
-	path = fmt.Sprintf("%s/index/%s/field/%s/import?clear=%s&ignoreKeyCheck=true", c.prefix(), field.index.Name(), field.Name(), strconv.FormatBool(clear))
+	path = fmt.Sprintf("/index/%s/field/%s/import?clear=%s&ignoreKeyCheck=true", field.index.Name(), field.Name(), strconv.FormatBool(clear))
 	return path, data, nil
 }
 
@@ -594,7 +594,7 @@ func (c *Client) EncodeImportValues(field *Field, shard uint64, vals []int64, id
 	if err != nil {
 		return "", nil, errors.Wrap(err, "marshaling ImportValue to protobuf")
 	}
-	path = fmt.Sprintf("%s/index/%s/field/%s/import?clear=%s&ignoreKeyCheck=true", c.prefix(), field.index.Name(), field.Name(), strconv.FormatBool(clear))
+	path = fmt.Sprintf("/index/%s/field/%s/import?clear=%s&ignoreKeyCheck=true", field.index.Name(), field.Name(), strconv.FormatBool(clear))
 	return path, data, nil
 }
 
@@ -625,7 +625,7 @@ func (c *Client) fetchFragmentNodes(indexName string, shard uint64) ([]fragmentN
 	if c.manualFragmentNode != nil {
 		return []fragmentNode{*c.manualFragmentNode}, nil
 	}
-	path := fmt.Sprintf("%s/internal/fragment/nodes?shard=%d&index=%s", c.prefix(), shard, indexName)
+	path := fmt.Sprintf("/internal/fragment/nodes?shard=%d&index=%s", shard, indexName)
 	_, body, err := c.HTTPRequest("GET", path, []byte{}, c.augmentHeaders(nil))
 	if err != nil {
 		return nil, err
@@ -688,7 +688,7 @@ func (c *Client) ImportRoaringShard(index string, shard uint64, request *pilosa.
 	for _, uri := range uris {
 		uri := uri
 		eg.Go(func() error {
-			return c.importData(uri, fmt.Sprintf("%s/index/%s/shard/%d/import-roaring", c.prefix(), index, shard), data)
+			return c.importData(uri, fmt.Sprintf("/index/%s/shard/%d/import-roaring", index, shard), data)
 		})
 	}
 	err = eg.Wait()
@@ -722,7 +722,7 @@ func (c *Client) importRoaringBitmap(uri *pnet.URI, field *Field, shard uint64,
 	}
 	params := url.Values{}
 	params.Add("clear", strconv.FormatBool(options.clear))
-	path := makeRoaringImportPath(field, shard, params, c.prefix())
+	path := makeRoaringImportPath(field, shard, params)
 	req := &pb.ImportRoaringRequest{
 		Clear:          options.clear,
 		Views:          protoViews,
@@ -776,7 +776,7 @@ func (c *Client) Info() (Info, error) {
 	span := c.tracer.StartSpan("Client.Info")
 	defer span.Finish()
 
-	path := fmt.Sprintf("%s/info", c.prefix())
+	path := "/info"
 	_, data, err := c.HTTPRequest("GET", path, nil, c.augmentHeaders(nil))
 	if err != nil {
 		return Info{}, errors.Wrap(err, "requesting /info")
@@ -794,7 +794,7 @@ func (c *Client) Status() (Status, error) {
 	span := c.tracer.StartSpan("Client.Status")
 	defer span.Finish()
 
-	path := fmt.Sprintf("%s/status", c.prefix())
+	path := "/status"
 	_, data, err := c.HTTPRequest("GET", path, nil, nil)
 	if err != nil {
 		return Status{}, errors.Wrap(err, "requesting /status")
@@ -808,7 +808,7 @@ func (c *Client) Status() (Status, error) {
 }
 
 func (c *Client) readSchema() ([]SchemaIndex, error) {
-	path := fmt.Sprintf("%s/schema", c.prefix())
+	path := "/schema"
 	_, data, err := c.HTTPRequest("GET", path, nil, c.augmentHeaders(nil))
 	if err != nil {
 		return nil, errors.Wrap(err, "requesting /schema")
@@ -822,7 +822,7 @@ func (c *Client) readSchema() ([]SchemaIndex, error) {
 }
 
 func (c *Client) shardsMax() (map[string]uint64, error) {
-	path := fmt.Sprintf("%s/internal/shards/max", c.prefix())
+	path := "/internal/shards/max"
 	_, data, err := c.HTTPRequest("GET", path, nil, nil)
 	if err != nil {
 		return nil, errors.Wrap(err, "requesting /internal/shards/max")
@@ -866,7 +866,7 @@ func (c *Client) httpRequest(method string, path string, data []byte, headers ma
 		// doRequest implements expotential backoff
 		status, body, err = c.doRequest(host, method, path, c.augmentHeaders(headers), data)
 		// conditions when primary should not be tried
-		pathCheck := fmt.Sprintf("%s/status", c.prefix())
+		pathCheck := "/status"
 		if err == nil || usePrimary || path == pathCheck {
 			break
 		}
@@ -902,8 +902,9 @@ func (c *Client) host(usePrimary bool) (*pnet.URI, error) {
 					c.primaryLock.Unlock()
 					return nil, errors.Wrap(err, "fetching primary node")
 				}
-				if host, err = pnet.NewURIFromAddress(fmt.Sprintf("%s://%s:%d", node.Scheme, node.Host, node.Port)); err != nil {
-					return nil, errors.Wrap(err, "parsing primary node URL")
+				addr := fmt.Sprintf("%s://%s:%d", node.Scheme, node.Host, node.Port)
+				if host, err = pnet.NewURIFromAddress(addr); err != nil {
+					return nil, errors.Wrapf(err, "parsing primary node URL: %s", addr)
 				}
 			} else {
 				host = c.primaryURI
@@ -930,6 +931,12 @@ func (c *Client) doRequest(host *pnet.URI, method, path string, headers map[stri
 		sleepTime time.Duration
 		rand      = rand.New(rand.NewSource(time.Now().UnixNano()))
 	)
+
+	// We have to add the service prefix to the path here (where applicable)
+	// because the pnet.URI type doesn't support the path portion of an address.
+	// Where needed, we already have a service prefix set on the Client.
+	path = c.prefix() + path
+
 	for retry := 0; ; {
 		if req, err = buildRequest(host, method, path, headers, data); err != nil {
 			return 0, nil, errors.Wrap(err, "building request")
@@ -1050,7 +1057,7 @@ func (c *Client) augmentHeaders(headers map[string]string) map[string]string {
 // FindFieldKeys looks up the IDs associated with specified keys in a field.
 // If a key does not exist, the result will not include it.
 func (c *Client) FindFieldKeys(field *Field, keys ...string) (map[string]uint64, error) {
-	path := fmt.Sprintf("%s/internal/translate/field/%s/%s/keys/find", c.prefix(), field.index.name, field.name)
+	path := fmt.Sprintf("/internal/translate/field/%s/%s/keys/find", field.index.name, field.name)
 
 	reqData, err := json.Marshal(keys)
 	if err != nil {
@@ -1082,7 +1089,7 @@ func (c *Client) FindFieldKeys(field *Field, keys ...string) (map[string]uint64,
 // CreateFieldKeys looks up the IDs associated with specified keys in a field.
 // If a key does not exist, it will be created.
 func (c *Client) CreateFieldKeys(field *Field, keys ...string) (map[string]uint64, error) {
-	path := fmt.Sprintf("%s/internal/translate/field/%s/%s/keys/create", c.prefix(), field.index.name, field.name)
+	path := fmt.Sprintf("/internal/translate/field/%s/%s/keys/create", field.index.name, field.name)
 
 	reqData, err := json.Marshal(keys)
 	if err != nil {
@@ -1114,7 +1121,7 @@ func (c *Client) CreateFieldKeys(field *Field, keys ...string) (map[string]uint6
 // FindIndexKeys looks up the IDs associated with specified column keys in an index.
 // If a key does not exist, the result will not include it.
 func (c *Client) FindIndexKeys(idx *Index, keys ...string) (map[string]uint64, error) {
-	path := fmt.Sprintf("%s/internal/translate/index/%s/keys/find", c.prefix(), idx.name)
+	path := fmt.Sprintf("/internal/translate/index/%s/keys/find", idx.name)
 
 	reqData, err := json.Marshal(keys)
 	if err != nil {
@@ -1146,7 +1153,7 @@ func (c *Client) FindIndexKeys(idx *Index, keys ...string) (map[string]uint64, e
 // CreateIndexKeys looks up the IDs associated with specified column keys in an index.
 // If a key does not exist, it will be created.
 func (c *Client) CreateIndexKeys(idx *Index, keys ...string) (map[string]uint64, error) {
-	path := fmt.Sprintf("%s/internal/translate/index/%s/keys/create", c.prefix(), idx.name)
+	path := fmt.Sprintf("/internal/translate/index/%s/keys/create", idx.name)
 
 	reqData, err := json.Marshal(keys)
 	if err != nil {
@@ -1200,7 +1207,7 @@ func (c *Client) startTransaction(id string, timeout time.Duration, exclusive bo
 		return nil, errors.Wrap(err, "marshalling transaction")
 	}
 
-	path := fmt.Sprintf("%s/transaction", c.prefix())
+	path := "/transaction"
 	status, data, err := c.httpRequest("POST", path, bod, c.augmentHeaders(defaultJSONHeaders()), true)
 	if status == http.StatusConflict && time.Now().Before(deadline) {
 		// if we're getting StatusConflict after all the usual timeouts/retries, keep retrying until the deadline
@@ -1228,7 +1235,7 @@ func (c *Client) startTransaction(id string, timeout time.Duration, exclusive bo
 }
 
 func (c *Client) FinishTransaction(id string) (*pilosa.Transaction, error) {
-	path := fmt.Sprintf("%s/transaction/%s/finish", c.prefix(), id)
+	path := fmt.Sprintf("/transaction/%s/finish", id)
 	_, data, err := c.httpRequest("POST", path, nil, c.augmentHeaders(defaultJSONHeaders()), true)
 	if err != nil && len(data) == 0 {
 		return nil, err
@@ -1251,7 +1258,7 @@ func (c *Client) FinishTransaction(id string) (*pilosa.Transaction, error) {
 }
 
 func (c *Client) Transactions() (map[string]*pilosa.Transaction, error) {
-	path := fmt.Sprintf("%s/transactions", c.prefix())
+	path := "/transactions"
 	_, respData, err := c.httpRequest("GET", path, nil, c.augmentHeaders(defaultJSONHeaders()), true)
 	if err != nil {
 		return nil, errors.Wrap(err, "getting transactions")
@@ -1266,7 +1273,7 @@ func (c *Client) Transactions() (map[string]*pilosa.Transaction, error) {
 }
 
 func (c *Client) GetTransaction(id string) (*pilosa.Transaction, error) {
-	path := fmt.Sprintf("%s/transaction/%s", c.prefix(), id)
+	path := fmt.Sprintf("/transaction/%s", id)
 	_, data, err := c.httpRequest("GET", path, nil, c.augmentHeaders(defaultJSONHeaders()), true)
 	if err != nil {
 		return nil, err
@@ -1344,9 +1351,9 @@ func makeRequestData(query string, options *QueryOptions) ([]byte, error) {
 	return r, nil
 }
 
-func makeRoaringImportPath(field *Field, shard uint64, params url.Values, pathPrefix string) string {
-	return fmt.Sprintf("%s/index/%s/field/%s/import-roaring/%d?%s",
-		pathPrefix, field.index.name, field.name, shard, params.Encode())
+func makeRoaringImportPath(field *Field, shard uint64, params url.Values) string {
+	return fmt.Sprintf("/index/%s/field/%s/import-roaring/%d?%s",
+		field.index.name, field.name, shard, params.Encode())
 }
 
 type viewImports map[string]*roaring.Bitmap

ctl/dax.go

@@ -14,12 +14,10 @@ func BuildDAXFlags(cmd *cobra.Command, srv *server.Command) {
 	flags.BoolVar(&srv.Config.Verbose, "verbose", srv.Config.Verbose, "Enable verbose logging")
 	flags.StringVar(&srv.Config.LogPath, "log-path", srv.Config.LogPath, "Log path")
 
-	flags.StringVar(&srv.Config.StorageMethod, "storage-method", srv.Config.StorageMethod, "Method to use for persistent storage.")
-	flags.StringVar(&srv.Config.StorageDSN, "storage-dsn", srv.Config.StorageDSN, "Datasource Name when using an applicable storage method.")
-
 	// MDS
 	flags.BoolVar(&srv.Config.MDS.Run, "mds.run", srv.Config.MDS.Run, "Run the MDS service in process.")
 	flags.DurationVar(&srv.Config.MDS.Config.RegistrationBatchTimeout, "mds.config.registration-batch-timeout", srv.Config.MDS.Config.RegistrationBatchTimeout, "Timeout for node registration batches.")
+	flags.StringVar(&srv.Config.MDS.Config.DataDir, "mds.config.data-dir", srv.Config.MDS.Config.DataDir, "MDS directory to use in process.")
 
 	// WriteLogger
 	flags.BoolVar(&srv.Config.WriteLogger.Run, "writelogger.run", srv.Config.WriteLogger.Run, "Run the WriteLogger service in process.")
@@ -35,5 +33,6 @@ func BuildDAXFlags(cmd *cobra.Command, srv *server.Command) {
 
 	// Computer
 	flags.BoolVar(&srv.Config.Computer.Run, "computer.run", srv.Config.Computer.Run, "Run the Computer service in process.")
+	flags.IntVar(&srv.Config.Computer.N, "computer.n", srv.Config.Computer.N, "The number of Computer services to run in process.")
 	flags.AddFlagSet(serverFlagSet(&srv.Config.Computer.Config, "computer.config"))
 }

dax/address.go

@@ -7,7 +7,7 @@ import (
 	"strings"
 )
 
-// Address is a string of the form [scheme]://[host]:[port]
+// Address is a string of the form [scheme]://[host]:[port]/[path]
 type Address string
 
 // String returns the Address as a string type.
@@ -22,7 +22,7 @@ func (a Address) Scheme() string {
 }
 
 // HostPort returns the [host]:[port] portion of the Address; in other words,
-// the Address stripped of any scheme.
+// the Address stripped of any scheme and path.
 func (a Address) HostPort() string {
 	return parse(a).hostPort()
 }
@@ -38,14 +38,20 @@ func (a Address) Port() uint16 {
 	return parse(a).port
 }
 
+// Path returns the [path] portion of the Address.
+func (a Address) Path() string {
+	return parse(a).path
+}
+
 // OverrideScheme overrides Address's current scheme with the one provided. If
-// an empty scheme is provided, OverrideScheme will return just the host:port.
+// an empty scheme is provided, OverrideScheme will return just the
+// host:port/path.
 func (a Address) OverrideScheme(scheme string) string {
 	addr := parse(a)
 	if scheme == "" {
-		return addr.hostPort()
+		return addr.hostPortPath()
 	}
-	return scheme + "://" + addr.hostPort()
+	return scheme + "://" + addr.hostPortPath()
 }
 
 // WithScheme ensures that the string returned contains the scheme portion of a
@@ -64,13 +70,14 @@ func (a Address) WithScheme(dflt string) string {
 	if addr.scheme != "" {
 		return a.String()
 	}
-	return dflt + "://" + addr.hostPort()
+	return dflt + "://" + addr.hostPortPath()
 }
 
 type addr struct {
 	scheme string
 	host   string
 	port   uint16
+	path   string
 }
 
 // parse breaks the address up into scheme://host:port. It currently assumes
@@ -80,15 +87,24 @@ func parse(a Address) addr {
 	var scheme string
 	var host string
 	var port uint16
+	var path string
 
 	aStr := string(a)
 
-	var hostPort string
+	var hostPortPath string
 	if parts := strings.Split(aStr, "://"); len(parts) > 1 {
 		scheme = parts[0]
-		hostPort = parts[1]
+		hostPortPath = parts[1]
 	} else {
-		hostPort = aStr
+		hostPortPath = aStr
+	}
+
+	var hostPort string
+	if parts := strings.SplitN(hostPortPath, "/", 2); len(parts) == 2 {
+		hostPort = parts[0]
+		path = parts[1]
+	} else {
+		hostPort = parts[0]
 	}
 
 	if parts := strings.Split(hostPort, ":"); len(parts) == 2 {
@@ -106,6 +122,7 @@ func parse(a Address) addr {
 		scheme: scheme,
 		host:   host,
 		port:   port,
+		path:   path,
 	}
 }
 
@@ -116,6 +133,21 @@ func (a addr) hostPort() string {
 	return fmt.Sprintf("%s:%d", a.host, a.port)
 }
 
+func (a addr) hostPortPath() string {
+	ret := ""
+	if a.port == 0 {
+		ret = a.host
+	} else {
+		ret = fmt.Sprintf("%s:%d", a.host, a.port)
+	}
+
+	if a.path != "" {
+		ret += "/" + a.path
+	}
+
+	return ret
+}
+
 // AddressManager is an interface for any service which needs to maintain a list
 // of addresses, and receive add/remove address requests from other services.
 type AddressManager interface {

dax/address_test.go

@@ -16,6 +16,7 @@ func TestAddress(t *testing.T) {
 			expHostPort string
 			expHost     string
 			expPort     uint16
+			expPath     string
 		}{
 			{
 				// blank address
@@ -97,6 +98,15 @@ func TestAddress(t *testing.T) {
 				expHost:     "",
 				expPort:     53308,
 			},
+			{
+				// with path:
+				addr:        "localhost:8080/foo/bar",
+				expScheme:   "",
+				expHostPort: "localhost:8080",
+				expHost:     "localhost",
+				expPort:     8080,
+				expPath:     "foo/bar",
+			},
 		}
 
 		for i, test := range tests {
@@ -105,6 +115,7 @@ func TestAddress(t *testing.T) {
 				assert.Equal(t, test.expHostPort, test.addr.HostPort())
 				assert.Equal(t, test.expHost, test.addr.Host())
 				assert.Equal(t, test.expPort, test.addr.Port())
+				assert.Equal(t, test.expPath, test.addr.Path())
 			})
 		}
 	})
@@ -130,6 +141,11 @@ func TestAddress(t *testing.T) {
 				scheme:  "",
 				expAddr: "foo:8080",
 			},
+			{
+				addr:    "http://foo:8080/bar",
+				scheme:  "",
+				expAddr: "foo:8080/bar",
+			},
 		}
 
 		for i, test := range tests {
@@ -165,6 +181,11 @@ func TestAddress(t *testing.T) {
 				scheme:  "grpc",
 				expAddr: "grpc://foo:8080",
 			},
+			{
+				addr:    "foo:8080/bar",
+				scheme:  "grpc",
+				expAddr: "grpc://foo:8080/bar",
+			},
 		}
 
 		for i, test := range tests {

dax/computer/alpha/snapshot.go

@@ -12,7 +12,6 @@ import (
 	"context"
 	"io"
 
-	featurebase "github.com/molecula/featurebase/v3"
 	"github.com/molecula/featurebase/v3/dax"
 	"github.com/molecula/featurebase/v3/dax/computer"
 	"github.com/molecula/featurebase/v3/errors"
@@ -25,10 +24,10 @@ var _ computer.SnapshotReadWriter = &alphaSnapshot{}
 // example, an http client or a locally running sub-service) to store its
 // snapshots.
 type alphaSnapshot struct {
-	ss featurebase.Snapshotter
+	ss computer.Snapshotter
 }
 
-func NewAlphaSnapshot(sser featurebase.Snapshotter) *alphaSnapshot {
+func NewAlphaSnapshot(sser computer.Snapshotter) *alphaSnapshot {
 	return &alphaSnapshot{
 		ss: sser,
 	}

dax/computer/alpha/writelog.go

@@ -15,7 +15,6 @@ import (
 	"encoding/json"
 	"io"
 
-	featurebase "github.com/molecula/featurebase/v3"
 	"github.com/molecula/featurebase/v3/dax"
 	"github.com/molecula/featurebase/v3/dax/computer"
 	"github.com/molecula/featurebase/v3/errors"
@@ -35,10 +34,10 @@ var _ computer.WriteLogWriter = &alphaWriteLog{}
 // (or perhaps "computer") is the top level package (i.e. "pilosa"). Until we
 // can correct the packaging, we will likely have weird cases like this.
 type alphaWriteLog struct {
-	wl featurebase.WriteLogger
+	wl computer.WriteLogger
 }
 
-func NewAlphaWriteLog(wler featurebase.WriteLogger) *alphaWriteLog {
+func NewAlphaWriteLog(wler computer.WriteLogger) *alphaWriteLog {
 	return &alphaWriteLog{
 		wl: wler,
 	}
@@ -126,7 +125,7 @@ func (w *alphaWriteLog) TableKeyReader(ctx context.Context, qtid dax.QualifiedTa
 }
 
 type tableKeyReader struct {
-	wl        featurebase.WriteLogger
+	wl        computer.WriteLogger
 	table     dax.TableKey
 	partition dax.PartitionNum
 	version   int
@@ -134,7 +133,7 @@ type tableKeyReader struct {
 	closer    io.Closer
 }
 
-func newTableKeyReader(wl featurebase.WriteLogger, qtid dax.QualifiedTableID, partition dax.PartitionNum, version int) *tableKeyReader {
+func newTableKeyReader(wl computer.WriteLogger, qtid dax.QualifiedTableID, partition dax.PartitionNum, version int) *tableKeyReader {
 	r := &tableKeyReader{
 		wl:        wl,
 		table:     qtid.Key(),
@@ -195,7 +194,7 @@ func (w *alphaWriteLog) FieldKeyReader(ctx context.Context, qtid dax.QualifiedTa
 }
 
 type fieldKeyReader struct {
-	wl      featurebase.WriteLogger
+	wl      computer.WriteLogger
 	table   dax.TableKey
 	field   dax.FieldName
 	version int
@@ -203,7 +202,7 @@ type fieldKeyReader struct {
 	closer  io.Closer
 }
 
-func newFieldKeyReader(wl featurebase.WriteLogger, qtid dax.QualifiedTableID, field dax.FieldName, version int) *fieldKeyReader {
+func newFieldKeyReader(wl computer.WriteLogger, qtid dax.QualifiedTableID, field dax.FieldName, version int) *fieldKeyReader {
 	r := &fieldKeyReader{
 		wl:      wl,
 		table:   qtid.Key(),
@@ -264,7 +263,7 @@ func (w *alphaWriteLog) ShardReader(ctx context.Context, qtid dax.QualifiedTable
 }
 
 type shardReader struct {
-	wl        featurebase.WriteLogger
+	wl        computer.WriteLogger
 	table     dax.TableKey
 	partition dax.PartitionNum
 	shard     dax.ShardNum
@@ -273,7 +272,7 @@ type shardReader struct {
 	closer    io.Closer
 }
 
-func newShardReader(wl featurebase.WriteLogger, qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum, version int) *shardReader {
+func newShardReader(wl computer.WriteLogger, qtid dax.QualifiedTableID, partition dax.PartitionNum, shard dax.ShardNum, version int) *shardReader {
 	r := &shardReader{
 		wl:        wl,
 		table:     qtid.Key(),

dax/computer/interfaces.go

@@ -1,4 +1,4 @@
-package pilosa
+package computer
 
 import (
 	"context"
@@ -7,16 +7,15 @@ import (
 	"github.com/molecula/featurebase/v3/dax"
 )
 
-// MDS represents the MDS methods which Computer uses. These are typically
-// implemented by both the MDS service and the MSD client.
-type MDS interface {
+// Registrar represents the methods which Computer uses to register itself with
+// MDS.
+type Registrar interface {
 	RegisterNode(ctx context.Context, node *dax.Node) error
 	CheckInNode(ctx context.Context, node *dax.Node) error
 }
 
 // WriteLogger represents the WriteLogger methods which Computer uses. These are
-// typically implemented by both the WriteLogger service and the WriteLogger
-// client.
+// typically implemented by the WriteLogger client.
 type WriteLogger interface {
 	AppendMessage(bucket string, key string, version int, msg []byte) error
 	LogReader(bucket string, key string, version int) (io.Reader, io.Closer, error)
@@ -24,8 +23,7 @@ type WriteLogger interface {
 }
 
 // Snapshotter represents the Snapshotter methods which Computer uses. These are
-// typically implemented by both the Snapshotter service and the Snapshotter
-// client.
+// typically implemented by both the Snapshotter client.
 type Snapshotter interface {
 	Read(bucket string, key string, version int) (io.ReadCloser, error)
 	Write(bucket string, key string, version int, rc io.ReadCloser) error

dax/computer/service/computer.go

@@ -0,0 +1,196 @@
+package service
+
+import (
+	"context"
+	"io"
+	"net"
+	"net/http"
+
+	featurebase "github.com/molecula/featurebase/v3"
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/computer"
+	mdsclient "github.com/molecula/featurebase/v3/dax/mds/client"
+	"github.com/molecula/featurebase/v3/dax/snapshotter"
+	snapshotterclient "github.com/molecula/featurebase/v3/dax/snapshotter/client"
+	"github.com/molecula/featurebase/v3/dax/writelogger"
+	writeloggerclient "github.com/molecula/featurebase/v3/dax/writelogger/client"
+	"github.com/molecula/featurebase/v3/errors"
+	"github.com/molecula/featurebase/v3/logger"
+	fbserver "github.com/molecula/featurebase/v3/server"
+)
+
+// Ensure type implements interface.
+var _ dax.ComputerService = (*computerService)(nil)
+
+type computerService struct {
+	addr     dax.Address
+	cfg      CommandConfig
+	key      dax.ServiceKey
+	computer *fbserver.Command
+	logger   logger.Logger
+}
+
+func New(addr dax.Address, cfg CommandConfig, logger logger.Logger) *computerService {
+	cfg.ComputerConfig.Advertise = addr.HostPort()
+
+	return &computerService{
+		addr:   addr,
+		cfg:    cfg,
+		logger: logger,
+	}
+}
+
+func (c *computerService) Start() error {
+	// We initialize the fbserver.Command at start (as opposed to in New()
+	// above) because we want to inject (via cfg.Name) the ServiceKey into the
+	// *fbserver.Command returned by newCommand() so that it can be used
+	// internally. In the future, when we have the pilosa package moved down to
+	// the computer package, we should be able to clean this up so that things
+	// happen in a reasonable order like we do with the other service types.
+	if c.computer == nil {
+		c.cfg.Name = string(c.Key())
+		c.computer = newCommand(c.addr, c.cfg)
+
+		if c.cfg.ComputerConfig.MDSAddress != "" {
+			mdsAddr := dax.Address(c.cfg.ComputerConfig.MDSAddress)
+			// Set mds (registrar) on computer.
+			if err := c.SetMDS(mdsAddr); err != nil {
+				return errors.Wrapf(err, "setting mds service on computer: %s, %v", c.cfg.Name, err)
+			}
+		}
+	}
+
+	if err := c.computer.StartNoServe(c.Address()); err != nil {
+		return errors.Wrap(err, "starting featurebase command")
+	}
+
+	if c.computer.Registrar != nil {
+		node := &dax.Node{
+			Address: c.Address(),
+			RoleTypes: []dax.RoleType{
+				dax.RoleTypeCompute,
+				dax.RoleTypeTranslate,
+			},
+		}
+
+		if err := c.computer.Registrar.RegisterNode(context.TODO(), node); err != nil {
+			return errors.Wrapf(err, "registering computer: %s", c.Address())
+		}
+	}
+	return nil
+}
+
+func (c *computerService) Stop() error {
+	return c.computer.Close()
+}
+
+func (c *computerService) Key() dax.ServiceKey {
+	return c.key
+}
+
+func (c *computerService) SetKey(key dax.ServiceKey) {
+	c.key = key
+}
+
+func (c *computerService) Address() dax.Address {
+	return dax.Address(c.addr.HostPort() + "/" + string(c.key))
+}
+
+func (c *computerService) HTTPHandler() http.Handler {
+	return c.computer.HTTPHandler()
+}
+
+func (c *computerService) SetMDS(addr dax.Address) error {
+	c.computer.Registrar = mdsclient.New(addr, c.logger)
+	return nil
+}
+
+type CommandConfig struct {
+	// Name is used to distinguish between locally running commands.
+	// For example, it's appended to DataDir so that each cmd has a
+	// separate data directory for its holder.
+	Name string
+
+	WriteLoggerRun    bool
+	WriteLoggerConfig writelogger.Config
+	SnapshotterRun    bool
+	SnapshotterConfig snapshotter.Config
+	ComputerConfig    fbserver.Config
+
+	Listener    net.Listener
+	RootDataDir string
+
+	Stderr io.Writer
+	Logger logger.Logger
+}
+
+func newCommand(addr dax.Address, cfg CommandConfig) *fbserver.Command {
+	// Set up WriteLogger.
+	// TODO(tlt): since WriteLogger is no longer a separate service (but
+	// rather just a directory path) its configuration could be moved under
+	// computer, and then get rid of WriteLogger.Run. This would become "if
+	// DataDir != ''". Let's do this after we get rid of the dax integration
+	// tests which start up separate writelogger and snapshotter containers.
+	var wlSvc *writelogger.WriteLogger
+	if cfg.WriteLoggerRun {
+		wlSvc = writelogger.New(writelogger.Config{
+			DataDir: cfg.WriteLoggerConfig.DataDir,
+			Logger:  cfg.Logger,
+		})
+	}
+
+	// Set up Snapshotter.
+	var ssSvc *snapshotter.Snapshotter
+	if cfg.SnapshotterRun {
+		ssSvc = snapshotter.New(snapshotter.Config{
+			DataDir: cfg.SnapshotterConfig.DataDir,
+			Logger:  cfg.Logger,
+		})
+	}
+
+	// Set the FeatureBase.Config values based on the top-level Config
+	// values.
+	cfg.ComputerConfig.Listener = &nopListener{}
+	cfg.ComputerConfig.Advertise = addr.HostPort()
+	cfg.ComputerConfig.GRPCListener = &nopListener{}
+	cfg.ComputerConfig.DataDir = cfg.RootDataDir + "/" + cfg.Name
+
+	var writeLoggerImpl computer.WriteLogger
+	if cfg.ComputerConfig.WriteLogger != "" {
+		writeLoggerImpl = writeloggerclient.New(dax.Address(cfg.ComputerConfig.WriteLogger))
+	} else if wlSvc != nil {
+		writeLoggerImpl = wlSvc
+	} else {
+		cfg.Logger.Warnf("No writelogger configured, dynamic scaling will not function properly.")
+	}
+
+	var snapshotterImpl computer.Snapshotter
+	if cfg.ComputerConfig.Snapshotter != "" {
+		snapshotterImpl = snapshotterclient.New(dax.Address(cfg.ComputerConfig.Snapshotter))
+	} else if ssSvc != nil {
+		snapshotterImpl = ssSvc
+	} else {
+		cfg.Logger.Warnf("No snapshotter configured.")
+	}
+
+	fbcmd := fbserver.NewCommand(cfg.Stderr,
+		fbserver.OptCommandSetConfig(&cfg.ComputerConfig),
+		fbserver.OptCommandServerOptions(
+			featurebase.OptServerIsComputeNode(true),
+			featurebase.OptServerLogger(cfg.Logger),
+		),
+		fbserver.OptCommandInjections(fbserver.Injections{
+			WriteLogger:   writeLoggerImpl,
+			Snapshotter:   snapshotterImpl,
+			IsComputeNode: true,
+		}),
+	)
+
+	return fbcmd
+}
+
+type nopListener struct{}
+
+func (n *nopListener) Accept() (net.Conn, error) { return nil, nil }
+func (n *nopListener) Close() error              { return nil }
+func (n *nopListener) Addr() net.Addr            { return nil }

dax/http/handler.go

@@ -7,16 +7,10 @@ import (
 	"runtime/debug"
 	"time"
 
-	"github.com/gorilla/mux"
-	"github.com/molecula/featurebase/v3/dax"
 	"github.com/molecula/featurebase/v3/dax/mds"
-	mdshttp "github.com/molecula/featurebase/v3/dax/mds/http"
 	"github.com/molecula/featurebase/v3/dax/queryer"
-	queryerhttp "github.com/molecula/featurebase/v3/dax/queryer/http"
 	"github.com/molecula/featurebase/v3/dax/snapshotter"
-	snapshotterhttp "github.com/molecula/featurebase/v3/dax/snapshotter/http"
 	"github.com/molecula/featurebase/v3/dax/writelogger"
-	writeloggerhttp "github.com/molecula/featurebase/v3/dax/writelogger/http"
 	"github.com/molecula/featurebase/v3/errors"
 	"github.com/molecula/featurebase/v3/logger"
 )
@@ -118,7 +112,7 @@ func OptHandlerComputer(handler http.Handler) HandlerOption {
 }
 
 // NewHandler returns a new instance of Handler with a default logger.
-func NewHandler(opts ...HandlerOption) (*Handler, error) {
+func NewHandler(router http.Handler, opts ...HandlerOption) (*Handler, error) {
 	handler := &Handler{
 		logger:       logger.NopLogger,
 		closeTimeout: time.Second * 30,
@@ -131,7 +125,7 @@ func NewHandler(opts ...HandlerOption) (*Handler, error) {
 		}
 	}
 
-	handler.Handler = newRouter(handler)
+	handler.Handler = router
 
 	handler.server = &http.Server{Handler: handler}
 
@@ -159,47 +153,6 @@ func (h *Handler) Close() error {
 	return errors.Wrap(err, "shutdown/close http server")
 }
 
-// newRouter creates a new mux http router.
-func newRouter(handler *Handler) http.Handler {
-	router := mux.NewRouter()
-
-	router.HandleFunc("/health", handler.handleGetHealth).Methods("GET").Name("GetHealth")
-
-	if handler.mds != nil {
-		pre := "/" + dax.ServicePrefixMDS
-		router.PathPrefix(pre).Handler(
-			http.StripPrefix(pre, mdshttp.Handler(handler.mds)))
-	}
-
-	if handler.writeLogger != nil {
-		pre := "/" + dax.ServicePrefixWriteLogger
-		router.PathPrefix(pre).Handler(
-			http.StripPrefix(pre, writeloggerhttp.Handler(handler.writeLogger, handler.logger)))
-	}
-
-	if handler.snapshotter != nil {
-		pre := "/" + dax.ServicePrefixSnapshotter
-		router.PathPrefix(pre).Handler(
-			http.StripPrefix(pre, snapshotterhttp.Handler(handler.snapshotter)))
-	}
-
-	if handler.queryer != nil {
-		pre := "/" + dax.ServicePrefixQueryer
-		router.PathPrefix(pre).Handler(
-			http.StripPrefix(pre, queryerhttp.Handler(handler.queryer)))
-	}
-
-	if handler.computer != nil {
-		pre := "/" + dax.ServicePrefixComputer
-		router.PathPrefix(pre).Handler(
-			http.StripPrefix(pre, handler.computer))
-	}
-
-	var h http.Handler = router
-
-	return h
-}
-
 // ServeHTTP handles an HTTP request.
 func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	defer func() {

dax/mds/client/client.go

@@ -7,41 +7,38 @@ import (
 	"encoding/json"
 	"fmt"
 	"io"
-	"log"
 	"net/http"
 
-	fb "github.com/molecula/featurebase/v3"
 	"github.com/molecula/featurebase/v3/dax"
 	"github.com/molecula/featurebase/v3/dax/mds/controller"
 	mdshttp "github.com/molecula/featurebase/v3/dax/mds/http"
 	"github.com/molecula/featurebase/v3/errors"
+	"github.com/molecula/featurebase/v3/logger"
 )
 
 const (
 	defaultScheme = "http"
-	defaultPath   = "/mds"
 )
 
-// Ensure type implements interface.
-var _ fb.MDS = (*Client)(nil)
-
 // Client is an HTTP client that operates on the MDS endpoints exposed by the
 // main MDS service.
 type Client struct {
 	address dax.Address
+	logger  logger.Logger
 }
 
 // New returns a new instance of Client.
-func New(address dax.Address) *Client {
+func New(address dax.Address, logger logger.Logger) *Client {
 	return &Client{
 		address: address,
+		logger:  logger,
 	}
 }
 
 // Health returns true if the client address returns status OK at its /health
 // endpoint.
 func (c *Client) Health() bool {
-	url := fmt.Sprintf("%s%s/health", c.address.WithScheme(defaultScheme), defaultPath)
+	url := fmt.Sprintf("%s/health", c.address.WithScheme(defaultScheme))
 
 	if resp, err := http.Get(url); err != nil {
 		return false
@@ -53,7 +50,7 @@ func (c *Client) Health() bool {
 }
 
 func (c *Client) Table(ctx context.Context, qtid dax.QualifiedTableID) (*dax.QualifiedTable, error) {
-	url := fmt.Sprintf("%s%s/table", c.address.WithScheme(defaultScheme), defaultPath)
+	url := fmt.Sprintf("%s/table", c.address.WithScheme(defaultScheme))
 
 	// Encode the request.
 	postBody, err := json.Marshal(qtid)
@@ -63,7 +60,7 @@ func (c *Client) Table(ctx context.Context, qtid dax.QualifiedTableID) (*dax.Qua
 	responseBody := bytes.NewBuffer(postBody)
 
 	// Post the request.
-	log.Printf("POST table request: url: %s", url)
+	c.logger.Debugf("POST table request: url: %s", url)
 	resp, err := http.Post(url, "application/json", responseBody)
 	if err != nil {
 		return nil, errors.Wrap(err, "posting table request")
@@ -84,7 +81,7 @@ func (c *Client) Table(ctx context.Context, qtid dax.QualifiedTableID) (*dax.Qua
 }
 
 func (c *Client) TableID(ctx context.Context, qual dax.TableQualifier, name dax.TableName) (dax.QualifiedTableID, error) {
-	url := fmt.Sprintf("%s%s/table-id", c.address.WithScheme(defaultScheme), defaultPath)
+	url := fmt.Sprintf("%s/table-id", c.address.WithScheme(defaultScheme))
 
 	dflt := dax.QualifiedTableID{}
 
@@ -121,7 +118,7 @@ func (c *Client) TableID(ctx context.Context, qual dax.TableQualifier, name dax.
 }
 
 func (c *Client) Tables(ctx context.Context, qual dax.TableQualifier, ids ...dax.TableID) ([]*dax.QualifiedTable, error) {
-	url := fmt.Sprintf("%s%s/tables", c.address.WithScheme(defaultScheme), defaultPath)
+	url := fmt.Sprintf("%s/tables", c.address.WithScheme(defaultScheme))
 
 	req := mdshttp.TablesRequest{
 		OrganizationID: qual.OrganizationID,
@@ -157,7 +154,7 @@ func (c *Client) Tables(ctx context.Context, qual dax.TableQualifier, ids ...dax
 }
 
 func (c *Client) CreateTable(ctx context.Context, qtbl *dax.QualifiedTable) error {
-	url := fmt.Sprintf("%s%s/create-table", c.address.WithScheme(defaultScheme), defaultPath)
+	url := fmt.Sprintf("%s/create-table", c.address.WithScheme(defaultScheme))
 
 	// Encode the request.
 	postBody, err := json.Marshal(qtbl)
@@ -182,7 +179,7 @@ func (c *Client) CreateTable(ctx context.Context, qtbl *dax.QualifiedTable) erro
 }
 
 func (c *Client) DropTable(ctx context.Context, qtid dax.QualifiedTableID) error {
-	url := fmt.Sprintf("%s%s/drop-table", c.address.WithScheme(defaultScheme), defaultPath)
+	url := fmt.Sprintf("%s/drop-table", c.address.WithScheme(defaultScheme))
 
 	// Encode the request.
 	postBody, err := json.Marshal(qtid)
@@ -207,7 +204,7 @@ func (c *Client) DropTable(ctx context.Context, qtid dax.QualifiedTableID) error
 }
 
 func (c *Client) CreateField(ctx context.Context, qtid dax.QualifiedTableID, fld *dax.Field) error {
-	url := fmt.Sprintf("%s%s/create-field", c.address.WithScheme(defaultScheme), defaultPath)
+	url := fmt.Sprintf("%s/create-field", c.address.WithScheme(defaultScheme))
 
 	req := mdshttp.CreateFieldRequest{
 		TableKey: qtid.Key(),
@@ -236,7 +233,7 @@ func (c *Client) CreateField(ctx context.Context, qtid dax.QualifiedTableID, fld
 }
 
 func (c *Client) DropField(ctx context.Context, qtid dax.QualifiedTableID, fldName dax.FieldName) error {
-	url := fmt.Sprintf("%s%s/drop-field", c.address.WithScheme(defaultScheme), defaultPath)
+	url := fmt.Sprintf("%s/drop-field", c.address.WithScheme(defaultScheme))
 
 	// Encode the request.
 	req := mdshttp.DropFieldRequest{
@@ -266,7 +263,7 @@ func (c *Client) DropField(ctx context.Context, qtid dax.QualifiedTableID, fldNa
 }
 
 func (c *Client) IngestShard(ctx context.Context, qtid dax.QualifiedTableID, shard dax.ShardNum) (dax.Address, error) {
-	url := fmt.Sprintf("%s%s/ingest-shard", c.address.WithScheme(defaultScheme), defaultPath)
+	url := fmt.Sprintf("%s/ingest-shard", c.address.WithScheme(defaultScheme))
 
 	var host dax.Address
 
@@ -303,7 +300,7 @@ func (c *Client) IngestShard(ctx context.Context, qtid dax.QualifiedTableID, sha
 }
 
 func (c *Client) IngestPartition(ctx context.Context, qtid dax.QualifiedTableID, partition dax.PartitionNum) (dax.Address, error) {
-	url := fmt.Sprintf("%s%s/ingest-partition", c.address.WithScheme(defaultScheme), defaultPath)
+	url := fmt.Sprintf("%s/ingest-partition", c.address.WithScheme(defaultScheme))
 
 	var host dax.Address
 
@@ -340,8 +337,8 @@ func (c *Client) IngestPartition(ctx context.Context, qtid dax.QualifiedTableID,
 }
 
 func (c *Client) ComputeNodes(ctx context.Context, qtid dax.QualifiedTableID, shards ...dax.ShardNum) ([]controller.ComputeNode, error) {
-	url := fmt.Sprintf("%s%s/compute-nodes", c.address.WithScheme(defaultScheme), defaultPath)
-	log.Printf("ComputeNodes url: %s", url)
+	url := fmt.Sprintf("%s/compute-nodes", c.address.WithScheme(defaultScheme))
+	c.logger.Debugf("ComputeNodes url: %s", url)
 
 	var nodes []controller.ComputeNode
 
@@ -378,8 +375,8 @@ func (c *Client) ComputeNodes(ctx context.Context, qtid dax.QualifiedTableID, sh
 }
 
 func (c *Client) TranslateNodes(ctx context.Context, qtid dax.QualifiedTableID, partitions ...dax.PartitionNum) ([]controller.TranslateNode, error) {
-	url := fmt.Sprintf("%s%s/translate-nodes", c.address.WithScheme(defaultScheme), defaultPath)
-	log.Printf("TranslateNodes url: %s", url)
+	url := fmt.Sprintf("%s/translate-nodes", c.address.WithScheme(defaultScheme))
+	c.logger.Debugf("TranslateNodes url: %s", url)
 
 	var nodes []controller.TranslateNode
 
@@ -416,8 +413,8 @@ func (c *Client) TranslateNodes(ctx context.Context, qtid dax.QualifiedTableID,
 }
 
 func (c *Client) RegisterNode(ctx context.Context, node *dax.Node) error {
-	url := fmt.Sprintf("%s%s/register-node", c.address.WithScheme(defaultScheme), defaultPath)
-	log.Printf("RegisterNode url: %s", url)
+	url := fmt.Sprintf("%s/register-node", c.address.WithScheme(defaultScheme))
+	c.logger.Debugf("RegisterNode url: %s", url)
 
 	req := &mdshttp.RegisterNodeRequest{
 		Address:   node.Address,
@@ -447,8 +444,8 @@ func (c *Client) RegisterNode(ctx context.Context, node *dax.Node) error {
 }
 
 func (c *Client) CheckInNode(ctx context.Context, node *dax.Node) error {
-	url := fmt.Sprintf("%s%s/check-in-node", c.address.WithScheme(defaultScheme), defaultPath)
-	log.Printf("CheckInNode url: %s", url)
+	url := fmt.Sprintf("%s/check-in-node", c.address.WithScheme(defaultScheme))
+	c.logger.Debugf("CheckInNode url: %s", url)
 
 	req := &mdshttp.CheckInNodeRequest{
 		Address:   node.Address,
@@ -476,3 +473,29 @@ func (c *Client) CheckInNode(ctx context.Context, node *dax.Node) error {
 
 	return nil
 }
+
+func (c *Client) SnapshotTable(ctx context.Context, qtid dax.QualifiedTableID) error {
+	url := fmt.Sprintf("%s/snapshot", c.address.WithScheme(defaultScheme))
+	c.logger.Debugf("Snapshot url: %s", url)
+
+	// Encode the request.
+	postBody, err := json.Marshal(qtid)
+	if err != nil {
+		return errors.Wrap(err, "marshalling post request")
+	}
+	responseBody := bytes.NewBuffer(postBody)
+
+	// Post the request.
+	resp, err := http.Post(url, "application/json", responseBody)
+	if err != nil {
+		return errors.Wrap(err, "posting translate-nodes request")
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return errors.Errorf("status code: %d: %s", resp.StatusCode, b)
+	}
+
+	return nil
+}

dax/mds/controller/balancer.go

@@ -10,7 +10,7 @@ import (
 type Balancer interface {
 	AddWorker(ctx context.Context, worker fmt.Stringer) ([]dax.WorkerDiff, error)
 	RemoveWorker(ctx context.Context, worker fmt.Stringer) ([]dax.WorkerDiff, error)
-	AddJob(ctx context.Context, job fmt.Stringer) ([]dax.WorkerDiff, error)
+	AddJobs(ctx context.Context, job ...fmt.Stringer) ([]dax.WorkerDiff, error)
 	RemoveJob(ctx context.Context, job fmt.Stringer) ([]dax.WorkerDiff, error)
 	Balance(ctx context.Context) ([]dax.WorkerDiff, error)
 	CurrentState(ctx context.Context) ([]dax.WorkerInfo, error)
@@ -46,7 +46,7 @@ func (b *NopBalancer) AddWorker(ctx context.Context, worker fmt.Stringer) ([]dax
 func (b *NopBalancer) RemoveWorker(ctx context.Context, worker fmt.Stringer) ([]dax.WorkerDiff, error) {
 	return []dax.WorkerDiff{}, nil
 }
-func (b *NopBalancer) AddJob(ctx context.Context, job fmt.Stringer) ([]dax.WorkerDiff, error) {
+func (b *NopBalancer) AddJobs(ctx context.Context, job ...fmt.Stringer) ([]dax.WorkerDiff, error) {
 	return []dax.WorkerDiff{}, nil
 }
 func (b *NopBalancer) RemoveJob(ctx context.Context, job fmt.Stringer) ([]dax.WorkerDiff, error) {

dax/mds/controller/controller.go

@@ -72,6 +72,7 @@ func New(cfg Config) *Controller {
 		logger: logger.NopLogger,
 
 		nodeChan: make(chan *dax.Node, 10),
+		stopping: make(chan struct{}),
 	}
 
 	if cfg.Logger != nil {
@@ -510,7 +511,7 @@ func (c *Controller) nodesTranslateReadOrWrite(ctx context.Context, role *dax.Tr
 				if err != nil {
 					return nil, false, NewErrInternal(err.Error())
 				}
-				diffs, err := bal.AddJob(ctx, j)
+				diffs, err := bal.AddJobs(ctx, j)
 				if err != nil {
 					return nil, false, errors.Wrap(err, "adding job")
 				}
@@ -697,7 +698,7 @@ func (c *Controller) nodesComputeReadOrWrite(ctx context.Context, role *dax.Comp
 				if err != nil {
 					return nil, false, NewErrInternal(err.Error())
 				}
-				diffs, err := bal.AddJob(ctx, j)
+				diffs, err := bal.AddJobs(ctx, j)
 				if err != nil {
 					return nil, false, errors.Wrap(err, "adding job")
 				}
@@ -811,29 +812,29 @@ func (c *Controller) CreateTable(ctx context.Context, qtbl *dax.QualifiedTable)
 		// and therefore need to be sent an updated Directive.
 		workerSet := NewAddressSet()
 
-		// Generate the list of partitions to be added.
-		partitions := make(dax.VersionedPartitions, qtbl.PartitionN)
+		// Generate the list of partitionsToAdd to be added.
+		partitionsToAdd := make(dax.VersionedPartitions, qtbl.PartitionN)
 		for partitionNum := 0; partitionNum < qtbl.PartitionN; partitionNum++ {
-			partitions[partitionNum] = dax.NewVersionedPartition(dax.PartitionNum(partitionNum), 0)
+			partitionsToAdd[partitionNum] = dax.NewVersionedPartition(dax.PartitionNum(partitionNum), 0)
 		}
 
 		// Add partitions to versionStore. Version is intentionally set to 0
 		// here as this is the initial instance of the partition.
-		if err := c.versionStore.AddPartitions(ctx, qtid, partitions...); err != nil {
+		if err := c.versionStore.AddPartitions(ctx, qtid, partitionsToAdd...); err != nil {
 			return NewErrInternal(err.Error())
 		}
 
-		for _, p := range partitions {
-			// We don't currently use the returned diff, other than to determine
-			// which worker was affected, because we send the full Directive
-			// every time.
-			diffs, err := c.TranslateBalancer.AddJob(ctx, partition(qtbl.Key(), p))
-			if err != nil {
-				return errors.Wrap(err, "adding job")
-			}
-			for _, diff := range diffs {
-				workerSet.Add(dax.Address(diff.WorkerID))
-			}
+		stringers := make([]fmt.Stringer, 0, len(partitionsToAdd))
+		for _, p := range partitionsToAdd {
+			stringers = append(stringers, partition(qtbl.Key(), p))
+		}
+
+		diffs, err := c.TranslateBalancer.AddJobs(ctx, stringers...)
+		if err != nil {
+			return errors.Wrap(err, "adding job")
+		}
+		for _, diff := range diffs {
+			workerSet.Add(dax.Address(diff.WorkerID))
 		}
 
 		// Convert the slice of addresses into a slice of addressMethod containing
@@ -866,7 +867,7 @@ func (c *Controller) CreateTable(ctx context.Context, qtbl *dax.QualifiedTable)
 		// We don't currently use the returned diff, other than to determine
 		// which worker was affected, because we send the full Directive
 		// every time.
-		diffs, err := c.TranslateBalancer.AddJob(ctx, partition(qtbl.Key(), p))
+		diffs, err := c.TranslateBalancer.AddJobs(ctx, partition(qtbl.Key(), p))
 		if err != nil {
 			return errors.Wrap(err, "adding job")
 		}
@@ -980,7 +981,7 @@ func (c *Controller) AddShards(ctx context.Context, qtid dax.QualifiedTableID, s
 		// We don't currently use the returned diff, other than to determine
 		// which worker was affected, because we send the full Directive every
 		// time.
-		diffs, err := c.ComputeBalancer.AddJob(ctx, shard(qtid.Key(), s))
+		diffs, err := c.ComputeBalancer.AddJobs(ctx, shard(qtid.Key(), s))
 		if err != nil {
 			return errors.Wrap(err, "adding job")
 		}

dax/mds/controller/http/director.go

@@ -12,10 +12,14 @@ import (
 	"time"
 
 	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/mds/controller"
 	"github.com/molecula/featurebase/v3/errors"
 	"github.com/molecula/featurebase/v3/logger"
 )
 
+// Ensure type implements interface.
+var _ controller.Director = (*Director)(nil)
+
 // Director is an http implementation of the Director interface.
 type Director struct {
 	// directivePath is the path portion of the URI to which directives should
@@ -32,7 +36,7 @@ type Director struct {
 }
 
 func NewDirector(cfg DirectorConfig) *Director {
-	var logr logger.Logger = logger.NopLogger
+	var logr = logger.NopLogger
 	if cfg.Logger != nil {
 		logr = cfg.Logger
 	}

dax/mds/controller/naive/balancer.go

@@ -4,16 +4,22 @@ package naive
 import (
 	"context"
 	"fmt"
+	"log"
 	"math"
 	"sort"
 	"strings"
 	"sync"
+	"time"
 
 	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/mds/controller"
 	"github.com/molecula/featurebase/v3/errors"
 	"github.com/molecula/featurebase/v3/logger"
 )
 
+// Ensure type implements interface.
+var _ controller.Balancer = (*Balancer)(nil)
+
 // Balancer is a naive implementation of the controller.Balancer interface. It
 // helps manage the relationships between workers and jobs. The logic it uses to
 // balance jobs across workers is very simple; it bases everything off the
@@ -47,14 +53,14 @@ type WorkerJobService interface {
 	CreateWorker(ctx context.Context, balancerName string, worker dax.Worker) error
 	DeleteWorker(ctx context.Context, balancerName string, worker dax.Worker) error
 
-	CreateJob(ctx context.Context, balancerName string, worker dax.Worker, job dax.Job) error
+	CreateJobs(ctx context.Context, balancerName string, worker dax.Worker, job ...dax.Job) error
 	DeleteJob(ctx context.Context, balancerName string, worker dax.Worker, job dax.Job) error
-	JobCount(ctx context.Context, balancerName string, worker dax.Worker) (int, error)
+	JobCounts(ctx context.Context, balancerName string, worker ...dax.Worker) (map[dax.Worker]int, error)
 	ListJobs(ctx context.Context, balancerName string, worker dax.Worker) (dax.Jobs, error)
 }
 
 type FreeJobService interface {
-	CreateFreeJob(ctx context.Context, balancerName string, job dax.Job) error
+	CreateFreeJobs(ctx context.Context, balancerName string, job ...dax.Job) error
 	DeleteFreeJob(ctx context.Context, balancerName string, job dax.Job) error
 	ListFreeJobs(ctx context.Context, balancerName string) (dax.Jobs, error)
 	MergeFreeJobs(ctx context.Context, balancerName string, jobs dax.Jobs) error
@@ -164,15 +170,30 @@ func (b *Balancer) removeWorker(ctx context.Context, worker dax.Worker) (interna
 	return diff, nil
 }
 
-// AddJob adds a job to an existing worker. If there are no existing workers,
-// the job is placed into the free list and will be assigned to a worker once
-// one becomes available.
-func (b *Balancer) AddJob(ctx context.Context, job fmt.Stringer) ([]dax.WorkerDiff, error) {
-	b.logger.Debugf("%s: AddJob(%s)", b.name, job.String())
+// AddJobs adds one or more jobs to an existing worker. If there are no existing
+// workers, the jobs are placed into the free list and will be assigned to a
+// worker once one becomes available.
+func (b *Balancer) AddJobs(ctx context.Context, jobs ...fmt.Stringer) ([]dax.WorkerDiff, error) {
+	start := time.Now()
+	defer func() {
+		log.Printf("ELAPSED: Balancer.AddJob: %v", time.Since(start))
+	}()
+
+	jobsToAdd := make([]dax.Job, 0, len(jobs))
+	for _, job := range jobs {
+		jobsToAdd = append(jobsToAdd, dax.Job(job.String()))
+	}
+
+	if len(jobsToAdd) == 1 {
+		b.logger.Debugf("%s: AddJobs (%s)", b.name, jobsToAdd[0])
+	} else {
+		b.logger.Debugf("%s: AddJobs (%d)", b.name, len(jobsToAdd))
+	}
+
 	b.mu.Lock()
 	defer b.mu.Unlock()
 
-	diff, err := b.addJob(ctx, dax.Job(job.String()))
+	diff, err := b.addJobs(ctx, jobsToAdd...)
 	if err != nil {
 		return nil, errors.Wrap(err, "adding job")
 	}
@@ -180,11 +201,11 @@ func (b *Balancer) AddJob(ctx context.Context, job fmt.Stringer) ([]dax.WorkerDi
 	return diff.output(), nil
 }
 
-func (b *Balancer) addJob(ctx context.Context, job dax.Job) (internalDiffs, error) {
+func (b *Balancer) addJobs(ctx context.Context, jobs ...dax.Job) (internalDiffs, error) {
 	if cnt, err := b.current.WorkerCount(ctx, b.name); err != nil {
 		return nil, errors.Wrap(err, "getting worker count")
 	} else if cnt == 0 {
-		if err := b.freeJobs.CreateFreeJob(ctx, b.name, job); err != nil {
+		if err := b.freeJobs.CreateFreeJobs(ctx, b.name, jobs...); err != nil {
 			return nil, errors.Wrap(err, "creating free job")
 		}
 		// TODO: we might want to inform the user that a job is in the free list
@@ -192,38 +213,59 @@ func (b *Balancer) addJob(ctx context.Context, job dax.Job) (internalDiffs, erro
 		return internalDiffs{}, nil
 	}
 
-	// Make sure this job doesn't already exist.
-	if _, ok, err := b.workerForJob(ctx, job); err != nil {
-		return nil, errors.Wrapf(err, "getting worker for job: %s", job)
-	} else if ok {
-		// The job is already being tracked.
-		return internalDiffs{}, nil
-	}
-
-	// Find the worker with the fewest number of jobs and assign it this job.
-	var lowCount int = math.MaxInt
-	var lowWorker dax.Worker
-
-	workerIDs, err := b.current.ListWorkers(ctx, b.name)
+	workerJobs, err := b.current.WorkersJobs(ctx, b.name)
 	if err != nil {
-		return nil, errors.Wrap(err, "listing workers")
+		return nil, errors.Wrapf(err, "getting workers jobs: %s", b.name)
+	}
+	jset := dax.NewSet[dax.Job]()
+	for _, workerInfo := range workerJobs {
+		jset.Merge(dax.NewSet(workerInfo.Jobs...))
 	}
 
-	for _, workerID := range workerIDs {
-		if l, err := b.current.JobCount(ctx, b.name, workerID); err != nil {
-			return nil, errors.Wrapf(err, "getting job count for worker: %s", workerID)
-		} else if l < lowCount {
-			lowCount = l
-			lowWorker = workerID
-		}
-	}
-
-	if err := b.current.CreateJob(ctx, b.name, lowWorker, job); err != nil {
-		return nil, errors.Wrap(err, "creating job")
+	workerIDs := make(dax.Workers, 0, len(workerJobs))
+	jobCounts := make(map[dax.Worker]int, 0)
+	for _, v := range workerJobs {
+		workerIDs = append(workerIDs, v.ID)
+		jobCounts[v.ID] = len(v.Jobs)
 	}
 
 	diffs := newInternalDiffs()
-	diffs.added(lowWorker, job)
+
+	jobsToCreate := make(map[dax.Worker][]dax.Job)
+
+	for _, job := range jobs {
+		// Skip any job that already exists.
+		if jset.Contains(job) {
+			continue
+		}
+
+		// Find the worker with the fewest number of jobs and assign it this job.
+		var lowCount int = math.MaxInt
+		var lowWorker dax.Worker
+
+		// We loop over workerIDs here instead of jobCounts because jobCounts is
+		// a map and it can return results in an unexpected order, which is a
+		// problem for testing.
+		for _, worker := range workerIDs {
+			jobCount := jobCounts[worker]
+			if jobCount < lowCount {
+				lowCount = jobCount
+				lowWorker = worker
+			}
+		}
+
+		jobsToCreate[lowWorker] = append(jobsToCreate[lowWorker], job)
+		jobCounts[lowWorker]++
+	}
+
+	for worker, jobs := range jobsToCreate {
+		if err := b.current.CreateJobs(ctx, b.name, worker, jobs...); err != nil {
+			return nil, errors.Wrap(err, "creating job")
+		}
+		for _, job := range jobs {
+			diffs.added(worker, job)
+		}
+	}
 
 	return diffs, nil
 }
@@ -314,11 +356,11 @@ func (b *Balancer) balance(ctx context.Context, diffs internalDiffs) (internalDi
 		return nil, errors.Wrapf(err, "listing workers: %s", b.name)
 	} else {
 		for _, worker := range workers {
-			cnt, err := b.current.JobCount(ctx, b.name, worker)
+			jobCounts, err := b.current.JobCounts(ctx, b.name, worker)
 			if err != nil {
 				return nil, errors.Wrapf(err, "getting job count: %s", worker)
 			}
-			numJobs += cnt
+			numJobs += jobCounts[worker]
 		}
 	}
 
@@ -340,10 +382,11 @@ func (b *Balancer) balance(ctx context.Context, diffs internalDiffs) (internalDi
 			numTargetJobs += 1
 		}
 
-		numCurrentJobs, err := b.current.JobCount(ctx, b.name, workerInfo.ID)
+		jobCounts, err := b.current.JobCounts(ctx, b.name, workerInfo.ID)
 		if err != nil {
 			return nil, errors.Wrapf(err, "getting job count: %s", workerInfo.ID)
 		}
+		numCurrentJobs := jobCounts[workerInfo.ID]
 
 		// If we don't need to remove jobs from this worker, then just continue
 		// on to the next worker.
@@ -364,7 +407,7 @@ func (b *Balancer) balance(ctx context.Context, diffs internalDiffs) (internalDi
 			} else {
 				diffs.merge(rj)
 			}
-			if aj, err := b.addJob(ctx, sortedJobs[i]); err != nil {
+			if aj, err := b.addJobs(ctx, sortedJobs[i]); err != nil {
 				return nil, errors.Wrapf(err, "adding job: %s", sortedJobs[i])
 			} else {
 				diffs.merge(aj)
@@ -511,7 +554,7 @@ func (b *Balancer) processFreeJobs(ctx context.Context) (internalDiffs, error) {
 		return nil, errors.Wrapf(err, "listing free jobs: %s", b.name)
 	}
 	for _, job := range jobs {
-		if aj, err := b.addJob(ctx, job); err != nil {
+		if aj, err := b.addJobs(ctx, job); err != nil {
 			return nil, errors.Wrapf(err, "adding job: %s", job)
 		} else {
 			diffs.merge(aj)

dax/mds/controller/naive/balancer_test.go

@@ -30,6 +30,12 @@ func TestBalancer(t *testing.T) {
 		db, cleanup := newBoltBalancer(t)
 		defer cleanup()
 		bal := boltdb.NewBalancer("test", db, logger.NewStandardLogger(os.Stderr))
+		// addJob is a wrapper around bal.AddJobs() which we added when the
+		// function signature of bal.AddJobs changed to take multiple jobs (and
+		// it therefore no longer satisfied the fn type in this test).
+		addJob := func(ctx context.Context, job fmt.Stringer) ([]dax.WorkerDiff, error) {
+			return bal.AddJobs(ctx, job)
+		}
 		tests := []struct {
 			fn       func(context.Context, fmt.Stringer) ([]dax.WorkerDiff, error)
 			input    string
@@ -38,7 +44,7 @@ func TestBalancer(t *testing.T) {
 		}{
 			{
 				// Add job.
-				fn:       bal.AddJob,
+				fn:       addJob,
 				input:    "p2",
 				expDiff:  []dax.WorkerDiff{},
 				expState: []dax.WorkerInfo{},
@@ -63,7 +69,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add another job out of order.
-				fn:    bal.AddJob,
+				fn:    addJob,
 				input: "p1",
 				expDiff: []dax.WorkerDiff{
 					{
@@ -81,7 +87,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add another job.
-				fn:    bal.AddJob,
+				fn:    addJob,
 				input: "p3",
 				expDiff: []dax.WorkerDiff{
 					{
@@ -99,7 +105,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add a duplicate job.
-				fn:      bal.AddJob,
+				fn:      addJob,
 				input:   "p2",
 				expDiff: []dax.WorkerDiff{},
 				expState: []dax.WorkerInfo{
@@ -127,6 +133,12 @@ func TestBalancer(t *testing.T) {
 		db, cleanup := newBoltBalancer(t)
 		defer cleanup()
 		bal := boltdb.NewBalancer("test", db, logger.NewStandardLogger(os.Stderr))
+		// addJob is a wrapper around bal.AddJobs() which we added when the
+		// function signature of bal.AddJobs changed to take multiple jobs (and
+		// it therefore no longer satisfied the fn type in this test).
+		addJob := func(ctx context.Context, job fmt.Stringer) ([]dax.WorkerDiff, error) {
+			return bal.AddJobs(ctx, job)
+		}
 		tests := []struct {
 			fn       func(context.Context, fmt.Stringer) ([]dax.WorkerDiff, error)
 			input    string
@@ -182,7 +194,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add job.
-				fn:    bal.AddJob,
+				fn:    addJob,
 				input: "p2",
 				expDiff: []dax.WorkerDiff{
 					{
@@ -204,7 +216,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add job.
-				fn:    bal.AddJob,
+				fn:    addJob,
 				input: "p3",
 				expDiff: []dax.WorkerDiff{
 					{
@@ -226,7 +238,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add job.
-				fn:    bal.AddJob,
+				fn:    addJob,
 				input: "p1",
 				expDiff: []dax.WorkerDiff{
 					{
@@ -268,7 +280,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add job.
-				fn:    bal.AddJob,
+				fn:    addJob,
 				input: "p4",
 				expDiff: []dax.WorkerDiff{
 					{
@@ -294,7 +306,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add job.
-				fn:    bal.AddJob,
+				fn:    addJob,
 				input: "p5",
 				expDiff: []dax.WorkerDiff{
 					{
@@ -320,7 +332,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add job.
-				fn:    bal.AddJob,
+				fn:    addJob,
 				input: "p0",
 				expDiff: []dax.WorkerDiff{
 					{
@@ -346,7 +358,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add job.
-				fn:    bal.AddJob,
+				fn:    addJob,
 				input: "p6",
 				expDiff: []dax.WorkerDiff{
 					{
@@ -372,7 +384,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add job.
-				fn:    bal.AddJob,
+				fn:    addJob,
 				input: "p7",
 				expDiff: []dax.WorkerDiff{
 					{
@@ -535,7 +547,7 @@ func TestBalancer(t *testing.T) {
 
 		_, err := bal.AddWorker(ctx, newStringWrapper("n1"))
 		assert.NoError(t, err)
-		_, err = bal.AddJob(ctx, newStringWrapper("p1"))
+		_, err = bal.AddJobs(ctx, newStringWrapper("p1"))
 		assert.NoError(t, err)
 
 		exp := dax.WorkerInfo{
@@ -565,7 +577,7 @@ func TestBalancer(t *testing.T) {
 		_, err = bal.AddWorker(ctx, newStringWrapper("n2"))
 		assert.NoError(t, err)
 		for i := 0; i < 12; i++ {
-			_, err = bal.AddJob(ctx, newStringWrapper(fmt.Sprintf("p%d", i)))
+			_, err = bal.AddJobs(ctx, newStringWrapper(fmt.Sprintf("p%d", i)))
 			assert.NoError(t, err)
 		}
 
@@ -661,7 +673,7 @@ func TestBalancer(t *testing.T) {
 		_, err = bal.AddWorker(ctx, newStringWrapper("n2"))
 		assert.NoError(t, err)
 		for i := 0; i < 13; i++ {
-			_, err = bal.AddJob(ctx, newStringWrapper(fmt.Sprintf("p%d", i)))
+			_, err = bal.AddJobs(ctx, newStringWrapper(fmt.Sprintf("p%d", i)))
 			assert.NoError(t, err)
 		}
 

dax/mds/controller/naive/boltdb/balancer.go

@@ -109,7 +109,7 @@ func (w *workerJobService) getWorkers(ctx context.Context, tx *boltdb.Tx, balanc
 
 		worker, err := keyWorker(k)
 		if err != nil {
-			return nil, errors.Wrapf(err, "getting worker from key: %v", k)
+			return nil, errors.Wrapf(err, "getting worker from key: %s", k)
 		}
 
 		workers = append(workers, worker)
@@ -128,7 +128,7 @@ func getWorkerInfos(ctx context.Context, tx *boltdb.Tx, balancerName string) (da
 	for k, v := c.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, v = c.Next() {
 		worker, err := keyWorker(k)
 		if err != nil {
-			return nil, errors.Wrapf(err, "getting worker from key: %v", k)
+			return nil, errors.Wrapf(err, "getting worker from key: %s", k)
 		}
 
 		jobs := dax.NewSet[dax.Job]()
@@ -212,7 +212,7 @@ func (w *workerJobService) DeleteWorker(ctx context.Context, balancerName string
 	return tx.Commit()
 }
 
-func (w *workerJobService) CreateJob(ctx context.Context, balancerName string, worker dax.Worker, job dax.Job) error {
+func (w *workerJobService) CreateJobs(ctx context.Context, balancerName string, worker dax.Worker, jobs ...dax.Job) error {
 	tx, err := w.db.BeginTx(ctx, true)
 	if err != nil {
 		return errors.Wrap(err, "beginning tx")
@@ -235,7 +235,9 @@ func (w *workerJobService) CreateJob(ctx context.Context, balancerName string, w
 		}
 	}
 
-	jobset.Add(job)
+	for _, job := range jobs {
+		jobset.Add(job)
+	}
 	val, err := encodeJobSet(jobset)
 	if err != nil {
 		return errors.Wrap(err, "encoding job set")
@@ -313,30 +315,36 @@ func (w *workerJobService) ListJobs(ctx context.Context, balancerName string, wo
 	return jobset.Sorted(), nil
 }
 
-func (w *workerJobService) JobCount(ctx context.Context, balancerName string, worker dax.Worker) (int, error) {
+func (w *workerJobService) JobCounts(ctx context.Context, balancerName string, workers ...dax.Worker) (map[dax.Worker]int, error) {
 	tx, err := w.db.BeginTx(ctx, false)
 	if err != nil {
-		return 0, errors.Wrapf(err, "getting tx: %s", balancerName)
+		return nil, errors.Wrapf(err, "getting tx: %s", balancerName)
 	}
 	defer tx.Rollback()
 
 	bkt := tx.Bucket(bucketNaiveBalancer)
 	if bkt == nil {
-		return 0, errors.Errorf(boltdb.ErrFmtBucketNotFound, bucketNaiveBalancer)
+		return nil, errors.Errorf(boltdb.ErrFmtBucketNotFound, bucketNaiveBalancer)
 	}
 
-	jobset := dax.NewSet[dax.Job]()
+	m := make(map[dax.Worker]int)
 
-	// get worker
-	wrkr := bkt.Get(workerKey(balancerName, worker))
-	if wrkr != nil {
-		jobset, err = decodeJobSet(wrkr)
-		if err != nil {
-			return 0, errors.Wrap(err, "decoding job set")
+	for _, worker := range workers {
+		jobset := dax.NewSet[dax.Job]()
+
+		// get worker
+		wrkr := bkt.Get(workerKey(balancerName, worker))
+		if wrkr != nil {
+			jobset, err = decodeJobSet(wrkr)
+			if err != nil {
+				return nil, errors.Wrap(err, "decoding job set")
+			}
 		}
+
+		m[worker] = len(jobset)
 	}
 
-	return len(jobset), nil
+	return m, nil
 }
 
 // encodeJobSet encode the jobSet into a JSON array of strings.
@@ -378,8 +386,8 @@ func newFreeJobService(db *boltdb.DB) *freeJobService {
 	}
 }
 
-func (f *freeJobService) CreateFreeJob(ctx context.Context, balancerName string, job dax.Job) error {
-	return f.MergeFreeJobs(ctx, balancerName, dax.Jobs{job})
+func (f *freeJobService) CreateFreeJobs(ctx context.Context, balancerName string, jobs ...dax.Job) error {
+	return f.MergeFreeJobs(ctx, balancerName, jobs)
 }
 
 func (f *freeJobService) DeleteFreeJob(ctx context.Context, balancerName string, job dax.Job) error {
@@ -500,7 +508,7 @@ func workerKey(bal string, worker dax.Worker) []byte {
 
 // keyWorker gets the worker out of the key.
 func keyWorker(key []byte) (dax.Worker, error) {
-	parts := strings.Split(string(key), "/")
+	parts := strings.SplitN(string(key), "/", 3)
 	if len(parts) != 3 {
 		return "", errors.New(errors.ErrUncoded, "worker key format expected: `workers/balancer/worker`")
 	}

dax/mds/controller/naive/boltdb/balancer_test.go

@@ -27,6 +27,12 @@ func TestBalancer(t *testing.T) {
 
 	t.Run("SingleWorker", func(t *testing.T) {
 		bal := boltdb.NewBalancer("test-single-worker", db, logger.NopLogger)
+		// addJob is a wrapper around bal.AddJobs() which we added when the
+		// function signature of bal.AddJobs changed to take multiple jobs (and
+		// it therefore no longer satisfied the fn type in this test).
+		addJob := func(ctx context.Context, job fmt.Stringer) ([]dax.WorkerDiff, error) {
+			return bal.AddJobs(ctx, job)
+		}
 		tests := []struct {
 			fn       func(context.Context, fmt.Stringer) ([]dax.WorkerDiff, error)
 			input    string
@@ -35,7 +41,7 @@ func TestBalancer(t *testing.T) {
 		}{
 			{
 				// Add job.
-				fn:       bal.AddJob,
+				fn:       addJob,
 				input:    "p2",
 				expDiff:  []dax.WorkerDiff{},
 				expState: []dax.WorkerInfo{},
@@ -60,7 +66,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add another job out of order.
-				fn:    bal.AddJob,
+				fn:    addJob,
 				input: "p1",
 				expDiff: []dax.WorkerDiff{
 					{
@@ -78,7 +84,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add another job.
-				fn:    bal.AddJob,
+				fn:    addJob,
 				input: "p3",
 				expDiff: []dax.WorkerDiff{
 					{
@@ -96,7 +102,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add a duplicate job.
-				fn:      bal.AddJob,
+				fn:      addJob,
 				input:   "p2",
 				expDiff: []dax.WorkerDiff{},
 				expState: []dax.WorkerInfo{
@@ -122,6 +128,12 @@ func TestBalancer(t *testing.T) {
 
 	t.Run("MultipleWorkers", func(t *testing.T) {
 		bal := boltdb.NewBalancer("test-multiple-workers", db, logger.NopLogger)
+		// addJob is a wrapper around bal.AddJobs() which we added when the
+		// function signature of bal.AddJobs changed to take multiple jobs (and
+		// it therefore no longer satisfied the fn type in this test).
+		addJob := func(ctx context.Context, job fmt.Stringer) ([]dax.WorkerDiff, error) {
+			return bal.AddJobs(ctx, job)
+		}
 		tests := []struct {
 			fn       func(context.Context, fmt.Stringer) ([]dax.WorkerDiff, error)
 			input    string
@@ -177,7 +189,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add job.
-				fn:    bal.AddJob,
+				fn:    addJob,
 				input: "p2",
 				expDiff: []dax.WorkerDiff{
 					{
@@ -199,7 +211,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add job.
-				fn:    bal.AddJob,
+				fn:    addJob,
 				input: "p3",
 				expDiff: []dax.WorkerDiff{
 					{
@@ -221,7 +233,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add job.
-				fn:    bal.AddJob,
+				fn:    addJob,
 				input: "p1",
 				expDiff: []dax.WorkerDiff{
 					{
@@ -263,7 +275,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add job.
-				fn:    bal.AddJob,
+				fn:    addJob,
 				input: "p4",
 				expDiff: []dax.WorkerDiff{
 					{
@@ -289,7 +301,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add job.
-				fn:    bal.AddJob,
+				fn:    addJob,
 				input: "p5",
 				expDiff: []dax.WorkerDiff{
 					{
@@ -315,7 +327,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add job.
-				fn:    bal.AddJob,
+				fn:    addJob,
 				input: "p0",
 				expDiff: []dax.WorkerDiff{
 					{
@@ -341,7 +353,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add job.
-				fn:    bal.AddJob,
+				fn:    addJob,
 				input: "p6",
 				expDiff: []dax.WorkerDiff{
 					{
@@ -367,7 +379,7 @@ func TestBalancer(t *testing.T) {
 			},
 			{
 				// Add job.
-				fn:    bal.AddJob,
+				fn:    addJob,
 				input: "p7",
 				expDiff: []dax.WorkerDiff{
 					{
@@ -528,7 +540,7 @@ func TestBalancer(t *testing.T) {
 
 		_, err := bal.AddWorker(ctx, newStringWrapper("n1"))
 		assert.NoError(t, err)
-		_, err = bal.AddJob(ctx, newStringWrapper("p1"))
+		_, err = bal.AddJobs(ctx, newStringWrapper("p1"))
 		assert.NoError(t, err)
 
 		exp := dax.WorkerInfo{
@@ -556,7 +568,7 @@ func TestBalancer(t *testing.T) {
 		_, err = bal.AddWorker(ctx, newStringWrapper("n2"))
 		assert.NoError(t, err)
 		for i := 0; i < 12; i++ {
-			_, err = bal.AddJob(ctx, newStringWrapper(fmt.Sprintf("p%d", i)))
+			_, err = bal.AddJobs(ctx, newStringWrapper(fmt.Sprintf("p%d", i)))
 			assert.NoError(t, err)
 		}
 
@@ -631,7 +643,7 @@ func TestBalancer(t *testing.T) {
 		_, err = bal.AddWorker(ctx, newStringWrapper("n2"))
 		assert.NoError(t, err)
 		for i := 0; i < 13; i++ {
-			_, err = bal.AddJob(ctx, newStringWrapper(fmt.Sprintf("p%d", i)))
+			_, err = bal.AddJobs(ctx, newStringWrapper(fmt.Sprintf("p%d", i)))
 			assert.NoError(t, err)
 		}
 

dax/mds/controller/stringers.go

@@ -32,6 +32,14 @@ func partition(t dax.TableKey, p dax.VersionedPartition) pUnit {
 	return pUnit{t, p}
 }
 
+func partitions(t dax.TableKey, p ...dax.VersionedPartition) []pUnit {
+	ret := make([]pUnit, 0, len(p))
+	for _, vp := range p {
+		ret = append(ret, pUnit{t, vp})
+	}
+	return ret
+}
+
 func decodePartition(j dax.Job) (pUnit, error) {
 	s := string(j)
 	parts := strings.Split(s, "|")

dax/mds/mds.go

@@ -4,14 +4,13 @@ package mds
 import (
 	"context"
 	"fmt"
-	"log"
 	"os"
 	"sync"
 	"time"
 
-	fb "github.com/molecula/featurebase/v3"
 	"github.com/molecula/featurebase/v3/dax"
 	"github.com/molecula/featurebase/v3/dax/boltdb"
+	"github.com/molecula/featurebase/v3/dax/computer"
 	"github.com/molecula/featurebase/v3/dax/mds/controller"
 	naiveboltdb "github.com/molecula/featurebase/v3/dax/mds/controller/naive/boltdb"
 	"github.com/molecula/featurebase/v3/dax/mds/poller"
@@ -23,26 +22,26 @@ import (
 
 type Config struct {
 	// Controller
-	Director controller.Director
+	Director controller.Director `toml:"-"`
 	// RegistrationBatchTimeout is the time that the controller will
 	// wait after a node registers itself to see if any more nodes
 	// will register before sending out directives to all nodes which
 	// have been registered.
-	RegistrationBatchTimeout time.Duration
+	RegistrationBatchTimeout time.Duration `toml:"registration-batch-timeout"`
 
 	// Poller
-	PollInterval time.Duration
+	PollInterval time.Duration `toml:"poll-interval"`
 
 	// Storage
-	StorageMethod string
-	StorageDSN    string
+	StorageMethod string `toml:"-"`
+	DataDir       string `toml:"-"`
 
 	// Logger
-	Logger logger.Logger
+	Logger logger.Logger `toml:"-"`
 }
 
 // Ensure type implements interface.
-var _ fb.MDS = (*MDS)(nil)
+var _ computer.Registrar = (*MDS)(nil)
 
 // MDS provides public MDS methods for an MDS service.
 type MDS struct {
@@ -52,48 +51,60 @@ type MDS struct {
 	poller     *poller.Poller
 	schemar    schemar.Schemar
 
+	// Because we stopped using a storage method interface, and always use bolt,
+	// we need to be sure to close the boltDBs that are created in mds.New()
+	// whenever mds.Close() is called. These are pointers to those DBs so we can
+	// close them.
+	schemarDB    *boltdb.DB
+	controllerDB *boltdb.DB
+
 	logger logger.Logger
 }
 
 // New returns a new instance of MDS.
 func New(cfg Config) *MDS {
 	// Set up logger.
-	var logr = logger.NopLogger
+	var logr logger.Logger = logger.StderrLogger
 	if cfg.Logger != nil {
 		logr = cfg.Logger
 	}
 
 	// Storage methods.
 	if cfg.StorageMethod != "boltdb" && cfg.StorageMethod != "" {
-		log.Printf("storagemethod %s not supported, try 'boltdb'", cfg.StorageMethod)
+		logr.Printf("storagemethod %s not supported, try 'boltdb'", cfg.StorageMethod)
 	}
-	if cfg.StorageDSN == "" {
+
+	cfg.StorageMethod = "boltdb"
+
+	if cfg.DataDir == "" {
 		dir, err := os.MkdirTemp("", "mds_*")
 		if err != nil {
 			logr.Printf("Making temp dir for MDS storage: %v", err)
 			os.Exit(1)
 		}
-		cfg.StorageDSN = fmt.Sprintf("file:%s", dir)
-		logr.Warnf("no StorageDSN given (like 'file:/path/to/directory') using temp dir at '%s'", cfg.StorageDSN)
+		cfg.DataDir = dir
+		logr.Warnf("no DataDir given (like '/path/to/directory') using temp dir at '%s'", cfg.DataDir)
 	}
-	schemarDB, err := boltdb.NewSvcBolt(cfg.StorageDSN, "schemar", schemarboltdb.SchemarBuckets...)
+
+	schemarDB, err := boltdb.NewSvcBolt(cfg.DataDir, "schemar", schemarboltdb.SchemarBuckets...)
 	if err != nil {
 		logr.Printf("Error creating schemar db: %v", err)
 		os.Exit(1)
 	}
+
 	schemar := schemarboltdb.NewSchemar(schemarDB, logr)
 
-	boltDB, err := boltdb.NewSvcBolt(cfg.StorageDSN, "balancer", naiveboltdb.NaiveBalancerBuckets...)
+	controllerDB, err := boltdb.NewSvcBolt(cfg.DataDir, "balancer", naiveboltdb.NaiveBalancerBuckets...)
 	if err != nil {
-		log.Println(errors.Wrap(err, "creating balancer bolt"))
+		logr.Printf(errors.Wrap(err, "creating balancer bolt").Error())
 		os.Exit(1)
 	}
 
 	controllerCfg := controller.Config{
 		Director:          cfg.Director,
 		Schemar:           schemar,
-		ComputeBalancer:   naiveboltdb.NewBalancer("compute", boltDB, logr),
-		TranslateBalancer: naiveboltdb.NewBalancer("translate", boltDB, logr),
+		ComputeBalancer:   naiveboltdb.NewBalancer("compute", controllerDB, logr),
+		TranslateBalancer: naiveboltdb.NewBalancer("translate", controllerDB, logr),
 
 		RegistrationBatchTimeout: cfg.RegistrationBatchTimeout,
 
@@ -101,7 +112,7 @@ func New(cfg Config) *MDS {
 		// just reusing this bolt for internal controller svcs
 		// rn... ultimately controller shouldn't know what bolt is at
 		// all
-		BoltDB: boltDB,
+		BoltDB: controllerDB,
 
 		Logger: logr,
 	}
@@ -126,6 +137,9 @@ func New(cfg Config) *MDS {
 		poller:     poller,
 		schemar:    schemar,
 
+		schemarDB:    schemarDB,
+		controllerDB: controllerDB,
+
 		logger: logr,
 	}
 }
@@ -134,8 +148,8 @@ func New(cfg Config) *MDS {
 // mds specific endpoints
 ////////////////////////////////////////////////////
 
-// Run starts MDS services, such as the Poller.
-func (m *MDS) Run() error {
+// Start starts MDS services, such as the Poller.
+func (m *MDS) Start() error {
 	// Initialize the poller (in the case where this MDS instance has restarted
 	// or is a replacement). Then start the poller.
 	if err := m.controller.InitializePoller(context.Background()); err != nil {
@@ -151,6 +165,14 @@ func (m *MDS) Run() error {
 func (m *MDS) Stop() error {
 	m.poller.Stop()
 	m.controller.Stop()
+
+	if m.schemarDB != nil {
+		m.schemarDB.Close()
+	}
+	if m.controllerDB != nil {
+		m.controllerDB.Close()
+	}
+
 	return nil
 }
 

dax/mds/service/mds.go

@@ -0,0 +1,46 @@
+package service
+
+import (
+	"net/http"
+
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/dax/mds"
+	mdshttp "github.com/molecula/featurebase/v3/dax/mds/http"
+	"github.com/molecula/featurebase/v3/errors"
+	fbnet "github.com/molecula/featurebase/v3/net"
+)
+
+// Ensure type implements interface.
+var _ dax.Service = (*mdsService)(nil)
+
+type mdsService struct {
+	uri *fbnet.URI
+	mds *mds.MDS
+}
+
+func New(uri *fbnet.URI, mds *mds.MDS) *mdsService {
+	return &mdsService{
+		uri: uri,
+		mds: mds,
+	}
+}
+
+func (m *mdsService) Start() error {
+	// Start mds service.
+	if err := m.mds.Start(); err != nil {
+		return errors.Wrap(err, "starting mds")
+	}
+	return nil
+}
+
+func (m *mdsService) Stop() error {
+	return m.mds.Stop()
+}
+
+func (m *mdsService) Address() dax.Address {
+	return dax.Address(m.uri.HostPort() + "/" + dax.ServicePrefixMDS)
+}
+
+func (m *mdsService) HTTPHandler() http.Handler {
+	return mdshttp.Handler(m.mds)
+}

dax/queryer/alpha/router.go

@@ -1,36 +0,0 @@
-package alpha
-
-import (
-	"github.com/molecula/featurebase/v3/dax"
-	"github.com/molecula/featurebase/v3/dax/queryer"
-	"github.com/molecula/featurebase/v3/errors"
-	featurebaseserver "github.com/molecula/featurebase/v3/server"
-)
-
-// Ensure type implements interface.
-var _ queryer.Router = (*Router)(nil)
-
-type Router struct {
-	computers map[dax.Address]*featurebaseserver.Command
-}
-
-func NewRouter() *Router {
-	return &Router{
-		computers: make(map[dax.Address]*featurebaseserver.Command),
-	}
-}
-
-func (r *Router) AddCmd(addr dax.Address, cmd *featurebaseserver.Command) error {
-	if cmd == nil {
-		return errors.New(errors.ErrUncoded, "cannot add nil cmd to director")
-	}
-	r.computers[addr] = cmd
-	return nil
-}
-
-func (r *Router) Importer(addr dax.Address) queryer.Importer {
-	if cmd, found := r.computers[addr]; found {
-		return queryer.NewFeatureBaseImporter(cmd.API)
-	}
-	return nil
-}

dax/queryer/client/client.go

@@ -0,0 +1,124 @@
+// Package client is an HTTP client for MDS.
+package client
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	featurebase "github.com/molecula/featurebase/v3"
+	"github.com/molecula/featurebase/v3/dax"
+	queryerhttp "github.com/molecula/featurebase/v3/dax/queryer/http"
+	"github.com/molecula/featurebase/v3/errors"
+	"github.com/molecula/featurebase/v3/logger"
+)
+
+const (
+	defaultScheme = "http"
+)
+
+// Client is an HTTP client that operates on the MDS endpoints exposed by the
+// main MDS service.
+type Client struct {
+	address dax.Address
+	logger  logger.Logger
+}
+
+// New returns a new instance of Client.
+func New(address dax.Address, logger logger.Logger) *Client {
+	return &Client{
+		address: address,
+		logger:  logger,
+	}
+}
+
+// Health returns true if the client address returns status OK at its /health
+// endpoint.
+func (c *Client) Health() bool {
+	url := fmt.Sprintf("%s/health", c.address.WithScheme(defaultScheme))
+
+	if resp, err := http.Get(url); err != nil {
+		return false
+	} else if resp.StatusCode != http.StatusOK {
+		return false
+	}
+
+	return true
+}
+
+func (c *Client) QuerySQL(ctx context.Context, qual dax.TableQualifier, sql string) (*featurebase.WireQueryResponse, error) {
+	url := fmt.Sprintf("%s/sql", c.address.WithScheme(defaultScheme))
+
+	req := &queryerhttp.SQLRequest{
+		OrganizationID: qual.OrganizationID,
+		DatabaseID:     qual.DatabaseID,
+		SQL:            sql,
+	}
+
+	// Encode the request.
+	postBody, err := json.Marshal(req)
+	if err != nil {
+		return nil, errors.Wrap(err, "marshalling post request")
+	}
+	responseBody := bytes.NewBuffer(postBody)
+
+	// Post the request.
+	c.logger.Debugf("POST query sql request: url: %s", url)
+	resp, err := http.Post(url, "application/json", responseBody)
+	if err != nil {
+		return nil, errors.Wrap(err, "posting query sql request")
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return nil, errors.Errorf("status code: %d: %s", resp.StatusCode, b)
+	}
+
+	var wireResp *featurebase.WireQueryResponse
+	if err := json.NewDecoder(resp.Body).Decode(&wireResp); err != nil {
+		return nil, errors.Wrap(err, "reading response body")
+	}
+
+	return wireResp, nil
+}
+func (c *Client) QueryPQL(ctx context.Context, qual dax.TableQualifier, table dax.TableName, pql string) (*featurebase.WireQueryResponse, error) {
+	url := fmt.Sprintf("%s/query", c.address.WithScheme(defaultScheme))
+
+	req := &queryerhttp.QueryRequest{
+		OrganizationID: qual.OrganizationID,
+		DatabaseID:     qual.DatabaseID,
+		Table:          table,
+		PQL:            pql,
+	}
+
+	// Encode the request.
+	postBody, err := json.Marshal(req)
+	if err != nil {
+		return nil, errors.Wrap(err, "marshalling post request")
+	}
+	responseBody := bytes.NewBuffer(postBody)
+
+	// Post the request.
+	c.logger.Debugf("POST query pql request: url: %s", url)
+	resp, err := http.Post(url, "application/json", responseBody)
+	if err != nil {
+		return nil, errors.Wrap(err, "posting query pql request")
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(resp.Body)
+		return nil, errors.Errorf("status code: %d: %s", resp.StatusCode, b)
+	}
+
+	var wireResp *featurebase.WireQueryResponse
+	if err := json.NewDecoder(resp.Body).Decode(&wireResp); err != nil {
+		return nil, errors.Wrap(err, "reading response body")
+	}
+
+	return wireResp, nil
+}

dax/queryer/compute_api.go

@@ -14,34 +14,23 @@ import (
 var _ featurebase.ComputeAPI = &qualifiedComputeAPI{}
 
 type qualifiedComputeAPI struct {
-	mds    MDS
-	router Router
-	qual   dax.TableQualifier
+	mds  MDS
+	qual dax.TableQualifier
 }
 
-func NewQualifiedComputeAPI(qual dax.TableQualifier, mds MDS, router Router) *qualifiedComputeAPI {
-	c := &qualifiedComputeAPI{
-		mds:    mds,
-		router: NewNopRouter(),
-		qual:   qual,
+func NewQualifiedComputeAPI(qual dax.TableQualifier, mds MDS) *qualifiedComputeAPI {
+	return &qualifiedComputeAPI{
+		mds:  mds,
+		qual: qual,
 	}
-
-	if router != nil {
-		c.router = router
-	}
-
-	return c
 }
 
-// importer is use to get the Importer based on the provided address. If the
-// computeAPI has been configured with entries in an ImporterRouter (which is a
-// map of dax.Address to in-process compute API), then it will use that.
-// Otherwise, it sets up an http client based on the provided address.
+// importer is used to get the Importer based on the provided address. We used
+// to maintain a map of different importers (pointers to computers) running
+// in-process, but since getting rid of that logic this method is currently just
+// a wrapper around NewComputeImporter. I'm leaving it like this for now in case
+// it makes sense for this to become a cache of computer clients.
 func (c *qualifiedComputeAPI) importer(addr dax.Address) (Importer, error) {
-	if imp := c.router.Importer(addr); imp != nil {
-		return imp, nil
-	}
-
 	return NewComputeImporter(addr), nil
 }
 

dax/queryer/config.go

@@ -12,10 +12,6 @@ import (
 // We initially did that with something called "Injections", but that separation
 // was a bit premature.
 type Config struct {
-	MDSAddress string `toml:"mds-address"`
-
-	MDS    MDS    `toml:"-"`
-	Router Router `toml:"-"`
-
-	Logger logger.Logger `toml:"-"`
+	MDSAddress string        `toml:"mds-address"`
+	Logger     logger.Logger `toml:"-"`
 }

dax/queryer/orchestrator.go

@@ -15,7 +15,6 @@ import (
 	"github.com/molecula/featurebase/v3/dax/mds/schemar"
 	"github.com/molecula/featurebase/v3/errors"
 	"github.com/molecula/featurebase/v3/logger"
-	"github.com/molecula/featurebase/v3/net"
 	"github.com/molecula/featurebase/v3/pql"
 	"github.com/molecula/featurebase/v3/stats"
 	"github.com/molecula/featurebase/v3/tracing"
@@ -1982,15 +1981,7 @@ func (o *orchestrator) remoteExec(ctx context.Context, node dax.Address, index s
 		EmbeddedData: embed,
 	}
 
-	scheme := node.Scheme()
-	if scheme == "" {
-		scheme = "http"
-	}
-	resp, err := o.client.QueryNode(ctx, &net.URI{
-		Scheme: scheme,
-		Host:   node.Host(),
-		Port:   node.Port(),
-	}, index, pbreq)
+	resp, err := o.client.QueryNode(ctx, node, index, pbreq)
 	if err != nil {
 		return nil, err
 	}

dax/queryer/queryer.go

@@ -32,53 +32,64 @@ import (
 type Queryer struct {
 	orchestrator *orchestrator
 
-	MDS    MDS
-	router Router
+	mds MDS
 
 	logger logger.Logger
 }
 
 // New returns a new instance of Queryer.
 func New(cfg Config) *Queryer {
-	fbClient, err := featurebase.NewInternalClient("fakehostname:8080",
-		&http.Client{},
-		featurebase.WithSerializer(proto.Serializer{}),
-		featurebase.WithPathPrefix(dax.ServicePrefixComputer),
-	)
-	if err != nil {
-		panic(err) // should be impossible
-	}
-
-	var logr = logger.NopLogger
-	if cfg.Logger != nil {
-		logr = cfg.Logger
-	}
-
 	q := &Queryer{
-		MDS:    NewNopMDS(),
-		router: NewNopRouter(),
-		orchestrator: &orchestrator{
-			schema:   NewSchemaInfoAPI(cfg.MDS),
-			trans:    NewMDSTranslator(cfg.MDS),
-			topology: &MDSTopology{mds: cfg.MDS},
-			// TODO(jaffee) using default http.Client probably bad... need to set some timeouts.
-			client: fbClient,
-			stats:  stats.NopStatsClient,
-			logger: logr,
-		},
-		logger: logr,
+		mds:          NewNopMDS(),
+		orchestrator: nil,
+		logger:       logger.NopLogger,
 	}
 
-	if cfg.MDS != nil {
-		q.MDS = cfg.MDS
-	}
-	if cfg.Router != nil {
-		q.router = cfg.Router
+	if cfg.Logger != nil {
+		q.logger = cfg.Logger
 	}
 
 	return q
 }
 
+func (q *Queryer) SetMDS(mds MDS) error {
+	q.mds = mds
+
+	// fbClient is an instance of internal client. It's used in one place in the
+	// orchestrator (o.client.QueryNode()), but in that case, the host is
+	// replaces with the actual host (another computer node) to connect to.
+	// That's why we set it up with a dummy host here.
+	fbClient, err := featurebase.NewInternalClient("fakehostname:8080",
+		&http.Client{},
+		featurebase.WithSerializer(proto.Serializer{}),
+		featurebase.WithPathPrefix("should-not-be-used"),
+	)
+	if err != nil {
+		return errors.Wrap(err, "setting up internal client")
+	}
+
+	q.orchestrator = &orchestrator{
+		schema:   NewSchemaInfoAPI(q.mds),
+		trans:    NewMDSTranslator(q.mds),
+		topology: &MDSTopology{mds: q.mds},
+		// TODO(jaffee) using default http.Client probably bad... need to set some timeouts.
+		client: fbClient,
+		stats:  stats.NopStatsClient,
+		logger: q.logger,
+	}
+
+	return nil
+}
+
+func (q *Queryer) Start() error {
+	if q.mds == nil {
+		return errors.New(errors.ErrUncoded, "queryer requires mds to be configured")
+	} else if q.orchestrator == nil {
+		return errors.New(errors.ErrUncoded, "queryer requires orchestrator to be configured")
+	}
+	return nil
+}
+
 func (q *Queryer) QuerySQL(ctx context.Context, qual dax.TableQualifier, sql string) (*featurebase.WireQueryResponse, error) {
 	start := time.Now()
 
@@ -112,16 +123,16 @@ func (q *Queryer) QuerySQL(ctx context.Context, qual dax.TableQualifier, sql str
 	}
 
 	// ComputeAPI
-	capi := NewQualifiedComputeAPI(qual, q.MDS, q.router)
+	capi := NewQualifiedComputeAPI(qual, q.mds)
 
 	// SchemaAPI
-	sapi := NewQualifiedSchemaAPI(qual, q.MDS)
+	sapi := NewQualifiedSchemaAPI(qual, q.mds)
 
 	// Orchestrator
-	orch := newQualifiedOrchestrator(q.orchestrator, qual, q.MDS)
+	orch := newQualifiedOrchestrator(q.orchestrator, qual, q.mds)
 
 	// Importer
-	imp := newBatchImporter(idkmds.NewImporter(q.MDS, nil), qual, q.MDS)
+	imp := newBatchImporter(idkmds.NewImporter(q.mds, nil), qual, q.mds)
 
 	// TODO(tlt): this obviously doesn't work; we don't have an API here. We
 	// need a dax-compatible implementation of the SystemAPI (or at least a
@@ -321,7 +332,7 @@ func (q *Queryer) indexToQualifiedTableKey(ctx context.Context, qual dax.TableQu
 		return dax.TableKey(index), nil
 	}
 
-	qtid, err := q.MDS.TableID(ctx, qual, dax.TableName(index))
+	qtid, err := q.mds.TableID(ctx, qual, dax.TableName(index))
 	if err != nil {
 		return "", errors.Wrap(err, "converting index to qualified table id")
 	}

dax/queryer/router.go

@@ -1,23 +0,0 @@
-package queryer
-
-import (
-	"github.com/molecula/featurebase/v3/dax"
-)
-
-type Router interface {
-	Importer(addr dax.Address) Importer
-}
-
-// Ensure type implements interface.
-var _ Router = &NopRouter{}
-
-// NopRouter is a no-op implementation of the Router interface.
-type NopRouter struct{}
-
-func NewNopRouter() *NopRouter {
-	return &NopRouter{}
-}
-
-func (d *NopRouter) Importer(addr dax.Address) Importer {
-	return nil
-}

dax/queryer/service/queryer.go

@@ -0,0 +1,55 @@
+package service
+
+import (
+	"net/http"
+
+	"github.com/molecula/featurebase/v3/dax"
+	mdsclient "github.com/molecula/featurebase/v3/dax/mds/client"
+	"github.com/molecula/featurebase/v3/dax/queryer"
+	queryerhttp "github.com/molecula/featurebase/v3/dax/queryer/http"
+	"github.com/molecula/featurebase/v3/errors"
+	"github.com/molecula/featurebase/v3/logger"
+	fbnet "github.com/molecula/featurebase/v3/net"
+)
+
+// Ensure type implements interface.
+var _ dax.Service = (*queryerService)(nil)
+
+type queryerService struct {
+	uri     *fbnet.URI
+	queryer *queryer.Queryer
+	logger  logger.Logger
+}
+
+func New(uri *fbnet.URI, queryer *queryer.Queryer, logger logger.Logger) *queryerService {
+	return &queryerService{
+		uri:     uri,
+		queryer: queryer,
+		logger:  logger,
+	}
+}
+
+func (q *queryerService) Start() error {
+	// Start queryer service.
+	if err := q.queryer.Start(); err != nil {
+		return errors.Wrap(err, "starting queryer")
+	}
+	return nil
+}
+
+func (q *queryerService) Stop() error {
+	return nil
+}
+
+func (q *queryerService) Address() dax.Address {
+	return dax.Address(q.uri.HostPort() + "/" + dax.ServicePrefixQueryer)
+}
+
+func (q *queryerService) HTTPHandler() http.Handler {
+	return queryerhttp.Handler(q.queryer)
+}
+
+func (q *queryerService) SetMDS(addr dax.Address) error {
+	q.queryer.SetMDS(mdsclient.New(addr, q.logger))
+	return nil
+}

dax/queryer/translator.go

@@ -28,11 +28,11 @@ func NewMDSTranslator(mds MDS) *MDSTranslator {
 
 func fbClient(address dax.Address) (*featurebase_client.Client, error) {
 	// Set up a FeatureBase client with address.
-	return featurebase_client.NewClient(address.String(),
+	return featurebase_client.NewClient(address.HostPort(),
 		featurebase_client.OptClientRetries(2),
 		featurebase_client.OptClientTotalPoolSize(1000),
 		featurebase_client.OptClientPoolSizePerRoute(400),
-		featurebase_client.OptClientPathPrefix(dax.ServicePrefixComputer),
+		featurebase_client.OptClientPathPrefix(address.Path()),
 		//featurebase_client.OptClientStatsClient(m.stats),
 	)
 }
@@ -261,7 +261,7 @@ func (m *MDSTranslator) TranslateFieldListIDs(ctx context.Context, index, field
 
 func makeTranslateIDsRequest(fbClient *featurebase_client.Client, table, field string, ids []uint64) ([]string, error) {
 	method := "POST"
-	path := "/" + dax.ServicePrefixComputer + "/internal/translate/ids"
+	path := "/internal/translate/ids"
 	headers := map[string]string{
 		"Content-Type": "application/x-protobuf",
 		"Accept":       "application/x-protobuf",

dax/server/config.go

@@ -10,6 +10,7 @@ import (
 	"strings"
 	"time"
 
+	"github.com/molecula/featurebase/v3/dax/mds"
 	"github.com/molecula/featurebase/v3/dax/queryer"
 	"github.com/molecula/featurebase/v3/dax/snapshotter"
 	"github.com/molecula/featurebase/v3/dax/writelogger"
@@ -32,6 +33,12 @@ type Config struct {
 	// route to an interface that Bind is listening on.
 	Advertise string `toml:"advertise"`
 
+	// Seed is used to seed the default rand.Source. If Seed is 0 (i.e. not set)
+	// the default rand.Source will be seeded using the current time. Note: this
+	// is not very useful at the moment because Table.CreateID() uses package
+	// crypto/rand which doesn't honor this seed.
+	Seed int64 `toml:"seed"`
+
 	// Verbose toggles verbose logging which can be useful for debugging.
 	Verbose bool `toml:"verbose"`
 
@@ -43,19 +50,11 @@ type Config struct {
 	Snapshotter SnapshotterOptions `toml:"snapshotter"`
 	Queryer     QueryerOptions     `toml:"queryer"`
 	Computer    ComputerOptions    `toml:"computer"`
-
-	// Storage methods.
-	StorageMethod string `toml:"storage-method"`
-	StorageDSN    string `toml:"storage-dsn"`
 }
 
 type MDSOptions struct {
-	Run    bool      `toml:"run"`
-	Config MDSConfig `toml:"config"`
-}
-
-type MDSConfig struct {
-	RegistrationBatchTimeout time.Duration `toml:"registration-batch-timeout"`
+	Run    bool       `toml:"run"`
+	Config mds.Config `toml:"config"`
 }
 
 type WriteLoggerOptions struct {
@@ -75,6 +74,7 @@ type QueryerOptions struct {
 
 type ComputerOptions struct {
 	Run    bool            `toml:"run"`
+	N      int             `toml:"n"`
 	Config fbserver.Config `toml:"config"`
 }
 
@@ -82,15 +82,15 @@ type ComputerOptions struct {
 func NewConfig() *Config {
 	c := &Config{
 		MDS: MDSOptions{
-			Config: MDSConfig{
+			Config: mds.Config{
 				RegistrationBatchTimeout: time.Second * 3,
+				StorageMethod:            defaultStorageMethod,
 			},
 		},
 		Bind: ":" + defaultBindPort,
 		Computer: ComputerOptions{
 			Config: *fbserver.NewConfig(),
 		},
-		StorageMethod: defaultStorageMethod,
 	}
 	return c
 }

dax/server/server.go

@@ -12,6 +12,7 @@ import (
 	"crypto/tls"
 	"encoding/json"
 	"io"
+	"log"
 	"math/rand"
 	"net"
 	"os"
@@ -23,21 +24,16 @@ import (
 
 	featurebase "github.com/molecula/featurebase/v3"
 	"github.com/molecula/featurebase/v3/dax"
+	computersvc "github.com/molecula/featurebase/v3/dax/computer/service"
 	daxhttp "github.com/molecula/featurebase/v3/dax/http"
 	"github.com/molecula/featurebase/v3/dax/mds"
-	mdsclient "github.com/molecula/featurebase/v3/dax/mds/client"
-	controlleralpha "github.com/molecula/featurebase/v3/dax/mds/controller/alpha"
 	controllerhttp "github.com/molecula/featurebase/v3/dax/mds/controller/http"
+	mdssvc "github.com/molecula/featurebase/v3/dax/mds/service"
 	"github.com/molecula/featurebase/v3/dax/queryer"
-	queryeralpha "github.com/molecula/featurebase/v3/dax/queryer/alpha"
-	"github.com/molecula/featurebase/v3/dax/snapshotter"
-	snapshotterclient "github.com/molecula/featurebase/v3/dax/snapshotter/client"
-	"github.com/molecula/featurebase/v3/dax/writelogger"
-	writeloggerclient "github.com/molecula/featurebase/v3/dax/writelogger/client"
+	queryersvc "github.com/molecula/featurebase/v3/dax/queryer/service"
 	"github.com/molecula/featurebase/v3/errors"
 	"github.com/molecula/featurebase/v3/logger"
 	fbnet "github.com/molecula/featurebase/v3/net"
-	featurebaseserver "github.com/molecula/featurebase/v3/server"
 )
 
 // Command represents the state of the dax server command.
@@ -60,39 +56,32 @@ type Command struct {
 	advertiseURI *fbnet.URI
 	tlsConfig    *tls.Config
 
-	// registerFns is a list of functions to call once the service is up and
-	// running. This is typically used to register the service with MDS.
-	registerFns []registerFn
-
-	// checkInFn is a function to call periodically in order to check-in with a
-	// monitorinig service such as MDS.
-	checkInFn checkInFn
-
 	logger    logger.Logger
 	logOutput io.Writer
-}
 
-type registerFn func() error
-type checkInFn func() error
+	svcmgr *dax.ServiceManager
+}
 
 type CommandOption func(c *Command) error
 
 func OptCommandConfig(config *Config) CommandOption {
 	return func(c *Command) error {
 		defer c.Config.MustValidate()
-		if c.Config != nil {
-			// c.Config.Etcd = config.Etcd
-			// c.Config.Auth = config.Auth
-			// c.Config.TLS = config.TLS
-			// c.Config.Controller = config.Controller
-			// c.Config.WriteLogger = config.WriteLogger
-			return nil
-		}
 		c.Config = config
 		return nil
 	}
 }
 
+// OptCommandServiceManager allows the ability to pass in a ServiceManage that
+// has been initialized outside of the Command. This is useful for testing where
+// we want to controll the service manager during a test run.
+func OptCommandServiceManager(svcmgr *dax.ServiceManager) CommandOption {
+	return func(c *Command) error {
+		c.svcmgr = svcmgr
+		return nil
+	}
+}
+
 // NewCommand returns a new instance of Command.
 func NewCommand(stderr io.Writer, opts ...CommandOption) *Command {
 	c := &Command{
@@ -100,9 +89,9 @@ func NewCommand(stderr io.Writer, opts ...CommandOption) *Command {
 
 		stderr: stderr,
 
-		registerFns: make([]registerFn, 0),
-
 		done: make(chan struct{}),
+
+		svcmgr: dax.NewServiceManager(),
 	}
 
 	for _, opt := range opts {
@@ -119,17 +108,17 @@ func NewCommand(stderr io.Writer, opts ...CommandOption) *Command {
 // Start starts the DAX server.
 func (m *Command) Start() (err error) {
 	// Seed random number generator
-	rand.Seed(time.Now().UTC().UnixNano())
+	seed := m.Config.Seed
+	if seed == 0 {
+		seed = time.Now().UTC().UnixNano()
+	}
+	log.Printf("Random seed: %d", seed)
+	rand.Seed(seed)
 
 	if err := m.setupServer(); err != nil {
 		return errors.Wrap(err, "setting up server")
 	}
 
-	// // Initialize server.
-	// if err = m.Server.Open(); err != nil {
-	// 	return errors.Wrap(err, "opening server")
-	// }
-
 	// Serve HTTP.
 	go func() {
 		if err := m.Handler.Serve(); err != nil {
@@ -138,42 +127,17 @@ func (m *Command) Start() (err error) {
 	}()
 	m.logger.Printf("listening as %s\n", m.listenURI)
 
-	// Register the service(s) by calling any registerFn they have implemented.
-	for i := range m.registerFns {
-		if err := m.registerFns[i](); err != nil {
-			return errors.Wrap(err, "calling register function")
-		}
+	if err := m.setupServices(); err != nil {
+		return errors.Wrap(err, "setting up services")
 	}
 
-	// Start the "check-in" background process which periodically checks in with
-	// MDS.
-	go m.checkIn()
+	if err := m.svcmgr.StartAll(); err != nil {
+		return errors.Wrap(err, "starting all services")
+	}
 
 	return nil
 }
 
-// checkIn calls the CheckIn function set on m.checkInFn every interval period.
-// If the interval period is 0, the check-in is disabled.
-func (m *Command) checkIn() {
-	interval := m.Config.Computer.Config.CheckInInterval
-
-	if interval == 0 || m.checkInFn == nil {
-		return
-	}
-
-	for {
-		select {
-		case <-m.done:
-			return
-		case <-time.After(interval):
-			m.logger.Debugf("node check-in in last %s, address: %s", interval, m.Config.Advertise)
-			if err := m.checkInFn(); err != nil {
-				m.logger.Errorf("checking in node: %s, %v", m.Config.Advertise, err)
-			}
-		}
-	}
-}
-
 // Wait waits for the server to be closed or interrupted.
 func (m *Command) Wait() error {
 	// First SIGKILL causes server to shut down gracefully.
@@ -209,6 +173,16 @@ func (m *Command) Close() error {
 	}
 }
 
+// URI returns the advertise URI at which the command can be reached.
+func (m *Command) URI() *fbnet.URI {
+	return m.advertiseURI
+}
+
+// Address returns the advertise address at which the command can be reached.
+func (m *Command) Address() dax.Address {
+	return dax.Address(m.advertiseURI.Normalize())
+}
+
 // // ParseConfig parses s into a Config.
 // func ParseConfig(s string) (Config, error) {
 // 	var c Config
@@ -236,6 +210,11 @@ func (m *Command) setupServer() error {
 	if err := m.setupLogger(); err != nil {
 		return errors.Wrap(err, "setting up logger")
 	}
+
+	if m.svcmgr != nil {
+		m.svcmgr.Logger = m.logger
+	}
+
 	conf, err := json.MarshalIndent(m.Config, "", "\t")
 	if err != nil {
 		return errors.Wrap(err, "marshalling config")
@@ -286,175 +265,106 @@ func (m *Command) setupServer() error {
 		daxhttp.OptHandlerLogger(m.logger),
 	}
 
-	// Set up WriteLogger.
-	var wlSvc *writelogger.WriteLogger
-	if m.Config.WriteLogger.Run {
-		wlSvc = writelogger.New(writelogger.Config{
-			DataDir: m.Config.WriteLogger.Config.DataDir,
-			Logger:  m.logger,
-		})
-		handlerOpts = append(handlerOpts, daxhttp.OptHandlerWriteLogger(wlSvc))
+	drouter := m.svcmgr.HTTPHandler()
+
+	// Set up Handler based on which services are running in process.
+	m.Handler, err = daxhttp.NewHandler(drouter, handlerOpts...)
+	if err != nil {
+		return errors.Wrap(err, "new handler")
 	}
 
-	// Set up Snapshotter.
-	var ssSvc *snapshotter.Snapshotter
-	if m.Config.Snapshotter.Run {
-		ssSvc = snapshotter.New(snapshotter.Config{
-			DataDir: m.Config.Snapshotter.Config.DataDir,
-			Logger:  m.logger,
-		})
-		handlerOpts = append(handlerOpts, daxhttp.OptHandlerSnapshotter(ssSvc))
-	}
+	return nil
+}
 
+// setupServices uses the configuration to set up the configured services.
+func (m *Command) setupServices() error {
 	// Set up MDS.
-	var mdsSvc *mds.MDS
-
-	// alphaDirector is used in the case where both the `mds` and `computer`
-	// services are running in the same process. It maintains the mapping
-	// between computer address and its API.
-	alphaDirector := controlleralpha.NewDirector()
-
-	alphaRouter := queryeralpha.NewRouter()
-
 	if m.Config.MDS.Run {
-		mdsSvcCfg := mds.Config{
+		mdsCfg := mds.Config{
 			RegistrationBatchTimeout: m.Config.MDS.Config.RegistrationBatchTimeout,
-			StorageMethod:            m.Config.StorageMethod,
-			StorageDSN:               m.Config.StorageDSN,
+			StorageMethod:            m.Config.MDS.Config.StorageMethod,
+			DataDir:                  m.Config.MDS.Config.DataDir,
 			Logger:                   m.logger,
-		}
-
-		// If the computer service is being run locally (in process) with MDS,
-		// then we want to use an implementation of the controller.Director
-		// interface which calls the interface methods *directly* on the compute
-		// node service (as opposed to going over http).
-		if m.Config.Computer.Run {
-			mdsSvcCfg.Director = alphaDirector
-		} else {
-			mdsSvcCfg.Director = controllerhttp.NewDirector(
+			Director: controllerhttp.NewDirector(
 				controllerhttp.DirectorConfig{
-					DirectivePath:       dax.ServicePrefixComputer + "/directive",
-					SnapshotRequestPath: dax.ServicePrefixComputer + "/snapshot",
+					DirectivePath:       "directive",
+					SnapshotRequestPath: "snapshot",
 					Logger:              m.logger,
-				})
+				}),
 		}
 
-		mdsSvc = mds.New(mdsSvcCfg)
-		handlerOpts = append(handlerOpts, daxhttp.OptHandlerMDS(mdsSvc))
-
-		// Start mds services.
-		if err := mdsSvc.Run(); err != nil {
-			return errors.Wrap(err, "running mds")
+		m.svcmgr.MDS = mdssvc.New(m.advertiseURI, mds.New(mdsCfg))
+		if err := m.svcmgr.MDSStart(); err != nil {
+			return errors.Wrap(err, "starting mds service")
 		}
 	}
 
 	// Set up Queryer.
 	if m.Config.Queryer.Run {
-		qryrSvcCfg := queryer.Config{
+		qryrCfg := queryer.Config{
 			Logger: m.logger,
 		}
 
-		var qryrSvcMDS queryer.MDS
-		var mdsRunning bool
+		m.svcmgr.Queryer = queryersvc.New(m.advertiseURI, queryer.New(qryrCfg), m.logger)
 
-		// This intentionally gives precedence to an MDSAddress over an MDS
-		// sub-service running in the same process.
+		var mdsAddr dax.Address
 		if m.Config.Queryer.Config.MDSAddress != "" {
-			qryrSvcMDS = mdsclient.New(dax.Address(m.Config.Queryer.Config.MDSAddress))
-		} else if m.Config.MDS.Run {
-			qryrSvcMDS = mdsSvc
-			mdsRunning = true
+			mdsAddr = dax.Address(m.Config.Queryer.Config.MDSAddress + "/" + dax.ServicePrefixMDS)
+		} else if m.svcmgr.MDS != nil {
+			mdsAddr = m.svcmgr.MDS.Address()
 		} else {
-			return errors.Errorf("queryer can't run without MDS")
-		}
-		qryrSvcCfg.MDS = qryrSvcMDS
-
-		// If the computer service is being run locally (in process) with MDS,
-		// then we want to use an importer which bypasses http requests and
-		// instead calls the respective services directly.
-		if mdsRunning && m.Config.Computer.Run {
-			qryrSvcCfg.Router = alphaRouter
+			return errors.Errorf("queryer requires MDS")
 		}
 
-		qryrSvc := queryer.New(qryrSvcCfg)
-		handlerOpts = append(handlerOpts, daxhttp.OptHandlerQueryer(qryrSvc))
+		// Set MDS
+		if err := m.svcmgr.Queryer.SetMDS(mdsAddr); err != nil {
+			return errors.Wrap(err, "setting mds")
+		}
+
+		// Start queryer.
+		if err := m.svcmgr.QueryerStart(); err != nil {
+			return errors.Wrap(err, "starting queryer service")
+		}
 	}
 
+	// rootDataDir holds the initial value in Config.DataDir. Because we change
+	// that value for every computer instance, we need to know what it started
+	// out as. A better solution might be to make a copy of Computer.Config on
+	// every iteration and create the new Command based on the copy (which can
+	// have a unique DataDir).
+	rootDataDir := m.Config.Computer.Config.DataDir
+
 	// Set up Computer.
 	if m.Config.Computer.Run {
-		// Set the FeatureBase.Config values based on the top-level Config
-		// values.
-		m.Config.Computer.Config.Listener = m.ln
-		m.Config.Computer.Config.Bind = uri.HostPort()
-		m.Config.Computer.Config.Advertise = m.advertiseURI.HostPort()
-
-		var mdsImpl featurebase.MDS
-		if m.Config.Computer.Config.MDSAddress != "" {
-			mdsImpl = mdsclient.New(dax.Address(m.Config.Computer.Config.MDSAddress))
-		} else if mdsSvc != nil {
-			mdsImpl = mdsSvc
-		} else {
-			return errors.Errorf("computer requires MDS")
+		n := m.Config.Computer.N
+		if n == 0 {
+			n = 1
 		}
 
-		var writeLoggerImpl featurebase.WriteLogger
-		if m.Config.Computer.Config.WriteLogger != "" {
-			writeLoggerImpl = writeloggerclient.New(dax.Address(m.Config.Computer.Config.WriteLogger))
-		} else if wlSvc != nil {
-			writeLoggerImpl = wlSvc
-		} else {
-			m.logger.Warnf("No writelogger configured, dynamic scaling will not function properly.")
+		for i := 0; i < n; i++ {
+			m.logger.Printf("Set up computer (%d)", i)
+			cfg := computersvc.CommandConfig{
+				WriteLoggerRun:    m.Config.WriteLogger.Run,
+				WriteLoggerConfig: m.Config.WriteLogger.Config,
+				SnapshotterRun:    m.Config.Snapshotter.Run,
+				SnapshotterConfig: m.Config.Snapshotter.Config,
+				ComputerConfig:    m.Config.Computer.Config,
+
+				Listener:    m.ln,
+				RootDataDir: rootDataDir,
+
+				Stderr: m.stderr,
+				Logger: m.logger,
+			}
+
+			if cfg.ComputerConfig.MDSAddress == "" && m.svcmgr.MDS != nil {
+				cfg.ComputerConfig.MDSAddress = string(m.svcmgr.MDS.Address())
+			}
+
+			// Add new computer service.
+			_ = m.svcmgr.AddComputer(
+				computersvc.New(dax.Address(m.advertiseURI.HostPort()), cfg, m.logger))
 		}
-
-		var snapshotterImpl featurebase.Snapshotter
-		if m.Config.Computer.Config.Snapshotter != "" {
-			snapshotterImpl = snapshotterclient.New(dax.Address(m.Config.Computer.Config.Snapshotter))
-		} else if ssSvc != nil {
-			snapshotterImpl = ssSvc
-		} else {
-			m.logger.Warnf("No snapshotter configured.")
-		}
-
-		fbcmd := featurebaseserver.NewCommand(m.stderr,
-			featurebaseserver.OptCommandSetConfig(&m.Config.Computer.Config),
-			featurebaseserver.OptCommandServerOptions(
-				featurebase.OptServerIsComputeNode(true),
-				featurebase.OptServerLogger(m.logger),
-			),
-			featurebaseserver.OptCommandInjections(featurebaseserver.Injections{
-				MDS:           mdsImpl,
-				WriteLogger:   writeLoggerImpl,
-				Snapshotter:   snapshotterImpl,
-				IsComputeNode: true,
-			}),
-		)
-
-		// Register the API with the local Director.
-		if err := alphaDirector.AddCmd(dax.Address(m.advertiseURI.HostPort()), fbcmd); err != nil {
-			return errors.Wrap(err, "adding cmd to director")
-		}
-
-		// Register the API with the local Router.
-		if err := alphaRouter.AddCmd(dax.Address(m.advertiseURI.HostPort()), fbcmd); err != nil {
-			return errors.Wrap(err, "adding cmd to router")
-		}
-
-		if err := fbcmd.StartNoServe(); err != nil {
-			return errors.Wrap(err, "start featurebase command")
-		}
-
-		// Add the cmd.Register function to the list of functions to call after
-		// setup.
-		m.registerFns = append(m.registerFns, fbcmd.Register)
-		m.checkInFn = fbcmd.CheckIn
-
-		handlerOpts = append(handlerOpts, daxhttp.OptHandlerComputer(fbcmd.HTTPHandler()))
-	}
-
-	// Set up Handler based on which services are running in process.
-	m.Handler, err = daxhttp.NewHandler(handlerOpts...)
-	if err != nil {
-		return errors.Wrap(err, "new handler")
 	}
 
 	return nil

dax/server/test/managed.go

@@ -0,0 +1,244 @@
+// Copyright 2021 Molecula Corp. All rights reserved.
+package test
+
+import (
+	"encoding/json"
+	"fmt"
+	"io"
+	"log"
+	"net/http"
+	"os"
+	"testing"
+	"time"
+
+	"github.com/molecula/featurebase/v3/dax"
+	computersvc "github.com/molecula/featurebase/v3/dax/computer/service"
+	"github.com/molecula/featurebase/v3/dax/mds"
+	mdssvc "github.com/molecula/featurebase/v3/dax/mds/service"
+	"github.com/molecula/featurebase/v3/dax/queryer"
+	queryersvc "github.com/molecula/featurebase/v3/dax/queryer/service"
+	"github.com/molecula/featurebase/v3/dax/server"
+	"github.com/molecula/featurebase/v3/errors"
+	fbtest "github.com/molecula/featurebase/v3/test"
+	"github.com/stretchr/testify/assert"
+)
+
+// ManagedCommand represents a test wrapper for server.Command.
+type ManagedCommand struct {
+	*server.Command
+
+	svcmgr *dax.ServiceManager
+
+	started bool
+}
+
+// Manage returns the ServiceManager for the ManagedCommand.
+func (mc *ManagedCommand) Manage() *dax.ServiceManager {
+	return mc.svcmgr
+}
+
+// Address returns the advertise address at which the command can be reached.
+func (mc *ManagedCommand) Address() dax.Address {
+	uri := mc.URI()
+	return dax.Address(uri.String())
+}
+
+// Start starts the embedded command.
+func (mc *ManagedCommand) Start() error {
+	if mc.started {
+		return nil
+	}
+
+	if err := mc.Command.Start(); err != nil {
+		return errors.Wrap(err, "starting command")
+	}
+	mc.started = true
+
+	return nil
+}
+
+// Close closes the embedded  command.
+func (mc *ManagedCommand) Close() error {
+	return mc.Command.Close()
+}
+
+// NewMDS adds a new MDSService to the ManagedCommands ServiceManager.
+func (mc *ManagedCommand) NewMDS(cfg mds.Config) dax.ServiceKey {
+	uri := mc.URI()
+	cfg.Logger = mc.svcmgr.Logger
+	mc.svcmgr.MDS = mdssvc.New(uri, mds.New(cfg))
+
+	return dax.ServicePrefixMDS
+}
+
+// NewQueryer adds a new QueryerService to the ManagedCommands ServiceManager.
+func (mc *ManagedCommand) NewQueryer(cfg queryer.Config) dax.ServiceKey {
+	uri := mc.URI()
+	logger := mc.svcmgr.Logger
+	cfg.Logger = logger
+	mc.svcmgr.Queryer = queryersvc.New(uri, queryer.New(cfg), logger)
+
+	var mdsAddr dax.Address
+	if cfg.MDSAddress != "" {
+		mdsAddr = dax.Address(cfg.MDSAddress + "/" + dax.ServicePrefixMDS)
+	} else if mc.svcmgr.MDS != nil {
+		mdsAddr = mc.svcmgr.MDS.Address()
+	}
+
+	// Set MDS
+	if err := mc.svcmgr.Queryer.SetMDS(mdsAddr); err != nil {
+		logger.Panicf(errors.Wrap(err, "setting mds").Error())
+	}
+
+	return dax.ServicePrefixQueryer
+}
+
+// NewComputer adds a new ComputerService to the ManagedCommands ServiceManager.
+func (mc *ManagedCommand) NewComputer() dax.ServiceKey {
+	cfg := computersvc.CommandConfig{
+		WriteLoggerRun:    mc.Config.WriteLogger.Run,
+		WriteLoggerConfig: mc.Config.WriteLogger.Config,
+		SnapshotterRun:    mc.Config.Snapshotter.Run,
+		SnapshotterConfig: mc.Config.Snapshotter.Config,
+		ComputerConfig:    mc.Config.Computer.Config,
+
+		RootDataDir: mc.Config.Computer.Config.DataDir,
+
+		Stderr: os.Stderr,
+		Logger: mc.svcmgr.Logger,
+	}
+
+	cfg.ComputerConfig.MDSAddress = mc.svcmgr.MDS.Address().String()
+
+	// Add new computer service.
+	return mc.svcmgr.AddComputer(
+		computersvc.New(mc.Address(), cfg, cfg.Logger))
+}
+
+// Healthy returns true if the provided service's /health endpoint returns 200
+// OK. This means that the service has been added to the ServiceManager and
+// started, and that its http handler has been dynamically added.
+func (mc *ManagedCommand) Healthy(key dax.ServiceKey) bool {
+	if key == "" {
+		return false
+	}
+
+	addr := mc.Address()
+	url := fmt.Sprintf("%s/%s/health", addr.WithScheme("http"), key)
+	log.Println("HEALTH URL:", url)
+	res, err := http.Get(url)
+	if err != nil {
+		return false
+	} else if res.StatusCode != http.StatusOK {
+		return false
+	}
+
+	return true
+}
+
+// WaitForApplied is a test helper function which retries a computer's
+// /directive endpoint a specified number of times, along with a sleep time in
+// between tries, until the computer returns applied=true.
+func (mc *ManagedCommand) WaitForApplied(t *testing.T, key dax.ServiceKey, n int, sleep time.Duration) {
+	t.Helper()
+
+	addr := mc.Address()
+	url := fmt.Sprintf("%s/%s/directive", addr.WithScheme("http"), key)
+	log.Println("WAIT URL:", url)
+
+	for i := 0; i < n; i++ {
+		resp, err := http.Get(url)
+		assert.NoError(t, err)
+		assert.Equal(t, http.StatusOK, resp.StatusCode)
+
+		body := resp.Body
+		defer body.Close()
+
+		got := struct {
+			Applied bool `json:"applied"`
+		}{}
+		assert.NoError(t, json.NewDecoder(body).Decode(&got))
+
+		if got.Applied {
+			return
+		}
+
+		t.Logf("Wait (%d/%d): url: %s (sleep: %s)\n", i, n, url, sleep.String())
+		if i < n-1 {
+			time.Sleep(sleep)
+		}
+	}
+
+	// Getting to here means the directive endpoint never returned successfully,
+	// so we need to stop the test.
+	t.Fatal("WaitForApplied timed out")
+}
+
+// NewManagedCommand returns a new instance of Command.
+func NewManagedCommand(tb fbtest.DirCleaner, opts ...server.CommandOption) *ManagedCommand {
+	path := tb.TempDir()
+
+	svcmgr := dax.NewServiceManager()
+	opts = append(opts, server.OptCommandServiceManager(svcmgr))
+
+	mc := &ManagedCommand{}
+	output := io.Discard
+	if testing.Verbose() {
+		output = os.Stderr
+	}
+	mc.Command = server.NewCommand(output, opts...)
+	mc.svcmgr = svcmgr
+
+	mc.Config.Bind = "http://localhost:0"
+	mc.Config.MDS.Config.DataDir = path + "/mds"
+	mc.Config.Computer.Config.DataDir = path
+	mc.Config.WriteLogger.Config.DataDir = path + "/wl"
+	mc.Config.Snapshotter.Config.DataDir = path + "/sn"
+
+	return mc
+}
+
+// DefaultConfig includes a single instance of each service type.
+func DefaultConfig() *server.Config {
+	cfg := server.NewConfig()
+	cfg.Verbose = true
+	cfg.MDS.Run = true
+	cfg.MDS.Config.RegistrationBatchTimeout = 0
+	cfg.Queryer.Run = true
+	cfg.Computer.Run = true
+	cfg.Computer.N = 1
+	cfg.WriteLogger.Run = true
+	cfg.Snapshotter.Run = true
+	return cfg
+}
+
+// MustRunManagedCommand starts an in-process set of Services based on the
+// provided configuration. If no configuration is provided, it will use the
+// DefaultConfig which consists of one instance of each service type.
+func MustRunManagedCommand(tb testing.TB, opts ...server.CommandOption) *ManagedCommand {
+	// If no opts are passed, use the default configuration which includes a
+	// single instance of each service type. This is really just meant to keep
+	// test code a bit cleaner when it's not necessary to have a custom service
+	// configuration.
+	var basic bool
+	if len(opts) == 0 {
+		opts = []server.CommandOption{
+			server.OptCommandConfig(DefaultConfig()),
+		}
+		basic = true
+	}
+
+	mc := NewManagedCommand(tb, opts...)
+
+	if err := mc.Start(); err != nil {
+		tb.Fatalf("starting managed command: %v", err)
+	}
+
+	if basic {
+		assert.True(tb, mc.Healthy(dax.ServicePrefixMDS))
+		assert.True(tb, mc.Healthy(dax.ServicePrefixQueryer))
+		assert.True(tb, mc.Healthy(dax.ServicePrefixComputer+"0"))
+	}
+
+	return mc
+}

dax/service_manager.go

@@ -0,0 +1,392 @@
+package dax
+
+import (
+	"fmt"
+	"net/http"
+	"sync"
+
+	"github.com/gorilla/mux"
+	"github.com/molecula/featurebase/v3/errors"
+	"github.com/molecula/featurebase/v3/logger"
+)
+
+// ServiceKey is a unique key used to identify one service managed by the
+// ServiceManager. These typically align with the ServicePrefix* values.
+type ServiceKey string
+
+// ServiceManager manages the various services running in process. It is used to
+// do things like start/stop services, and it dynamically builds the http router
+// depending on the state of all services.
+type ServiceManager struct {
+	mu sync.RWMutex
+
+	// MDS
+	MDS        MDSService
+	mdsStarted bool
+
+	// Queryer
+	Queryer        QueryerService
+	queryerStarted bool
+
+	// Computers
+	computerID int
+	computers  map[ServiceKey]*computerServiceState
+
+	drouter *dynamicRouter
+
+	Logger logger.Logger
+}
+
+type computerServiceState struct {
+	service ComputerService
+	started bool
+}
+
+// NewServiceManager returns a new ServiceManager with default values.
+func NewServiceManager() *ServiceManager {
+	return &ServiceManager{
+		computers: map[ServiceKey]*computerServiceState{},
+		drouter:   &dynamicRouter{},
+		Logger:    logger.NopLogger,
+	}
+}
+
+// HTTPHandler returns the current http.Handler for ServiceManager based on the
+// state of its services.
+func (s *ServiceManager) HTTPHandler() http.Handler {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+	s.resetRouter()
+	return s.drouter
+}
+
+// StartAll starts all services which have been added to ServiceManager.
+func (s *ServiceManager) StartAll() error {
+	// MDS
+	if err := s.MDSStart(); err != nil {
+		return errors.Wrap(err, "starting mds")
+	}
+
+	// Queryer
+	if err := s.QueryerStart(); err != nil {
+		return errors.Wrap(err, "starting queryer")
+	}
+
+	// Computer(s)
+	for key := range s.computers {
+		if err := s.ComputerStart(key); err != nil {
+			return errors.Wrapf(err, "starting computer (%s)", key)
+		}
+	}
+
+	return nil
+}
+
+// MDSStart starts the MDS service.
+func (s *ServiceManager) MDSStart() error {
+	if s.MDS == nil {
+		return nil
+	}
+
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	if s.mdsStarted {
+		return nil
+	}
+
+	s.mdsStarted = true
+	s.resetRouter()
+
+	if err := s.MDS.Start(); err != nil {
+		s.mdsStarted = false
+		return errors.Wrap(err, "starting mds")
+	}
+
+	return nil
+}
+
+// MDSStop stops the MDS service.
+func (s *ServiceManager) MDSStop() error {
+	if s.MDS == nil {
+		return nil
+	}
+
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	if !s.mdsStarted {
+		return nil
+	}
+
+	s.mdsStarted = false
+	s.resetRouter()
+
+	if err := s.MDS.Stop(); err != nil {
+		s.mdsStarted = true
+		return errors.Wrap(err, "stopping controller")
+	}
+
+	return nil
+}
+
+// QueryerStart starts the Queryer service.
+func (s *ServiceManager) QueryerStart() error {
+	if s.Queryer == nil {
+		return nil
+	}
+
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	if s.queryerStarted {
+		return nil
+	}
+
+	s.queryerStarted = true
+	s.resetRouter()
+
+	if err := s.Queryer.Start(); err != nil {
+		s.queryerStarted = false
+		return errors.Wrap(err, "starting queryer")
+	}
+
+	return nil
+}
+
+// QueryerStop stops the Queryer service.
+func (s *ServiceManager) QueryerStop() error {
+	if s.Queryer == nil {
+		return nil
+	}
+
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	if !s.queryerStarted {
+		return nil
+	}
+
+	s.queryerStarted = false
+	s.resetRouter()
+
+	if err := s.Queryer.Stop(); err != nil {
+		s.queryerStarted = true
+		return errors.Wrap(err, "stopping queryer")
+	}
+
+	return nil
+}
+
+// Computer returns the ComputerService specified by the provided key.
+func (s *ServiceManager) Computer(key ServiceKey) ComputerService {
+	serviceState, ok := s.computers[key]
+	if !ok {
+		return nil
+	}
+	return serviceState.service
+}
+
+// ComputerStart starts the Computer service specified by the provided key.
+func (s *ServiceManager) ComputerStart(key ServiceKey) error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	serviceState, ok := s.computers[key]
+	if !ok {
+		return errors.Errorf("computer to be started does not exist: %s", key)
+	}
+
+	if serviceState.started {
+		return nil
+	}
+
+	serviceState.started = true
+
+	if err := serviceState.service.Start(); err != nil {
+		serviceState.started = false
+		return errors.Wrapf(err, "starting computer (%s)", key)
+	}
+
+	// resetRouter is called *after* service.Start() for computer (but not other
+	// service types) because currently, the handler returned by
+	// server.Command.HTTPHandler() doesn't get initialized until startup. A
+	// task for the future will be to tease out the computer http routes so that
+	// they're available prior to startup.
+	s.resetRouter()
+
+	return nil
+}
+
+// ComputerStop stops the Computer service specified by the provided key.
+func (s *ServiceManager) ComputerStop(key ServiceKey) error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	serviceState, ok := s.computers[key]
+	if !ok {
+		return errors.Errorf("computer to be stopped does not exist: %s", key)
+	}
+
+	if !serviceState.started {
+		return nil
+	}
+
+	serviceState.started = false
+	s.resetRouter()
+
+	if err := serviceState.service.Stop(); err != nil {
+		serviceState.started = true
+		return errors.Wrapf(err, "stopping computer (%s)", key)
+	}
+
+	return nil
+}
+
+// Computers returns a map (keyed by ServiceKey) of all computers registered
+// with ServiceManager.
+func (s *ServiceManager) Computers() map[ServiceKey]ComputerService {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+	m := make(map[ServiceKey]ComputerService)
+
+	for k, v := range s.computers {
+		m[k] = v.service
+	}
+
+	return m
+}
+
+// AddComputer adds the provided ComputerService to ServiceManager. It assigns
+// the service a unique ServiceKey.
+func (s *ServiceManager) AddComputer(cs ComputerService) ServiceKey {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	key := ServiceKey(fmt.Sprintf("%s%d", ServicePrefixComputer, s.computerID))
+	s.computers[key] = &computerServiceState{
+		service: cs,
+	}
+	cs.SetKey(key)
+
+	s.computerID++
+
+	return key
+}
+
+// RemoveComputer removes the ComputerService specified by the provided key.
+func (s *ServiceManager) RemoveComputer(key ServiceKey) bool {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	if _, ok := s.computers[key]; ok {
+		delete(s.computers, key)
+		s.resetRouter()
+		return true
+	}
+
+	return false
+}
+
+func getHealth(w http.ResponseWriter, req *http.Request) {
+	w.WriteHeader(http.StatusOK)
+}
+
+// Must be called with at least a read lock held (because that's required of buildRouter).
+func (s *ServiceManager) resetRouter() {
+	s.drouter.Swap(s.buildRouter())
+}
+
+// Must be called with at least a read lock held?
+func (s *ServiceManager) buildRouter() *mux.Router {
+	router := mux.NewRouter()
+	router.HandleFunc("/health", getHealth).Methods("GET").Name("GetHealth")
+
+	// MDS.
+	if s.MDS != nil && s.mdsStarted {
+		pre := "/" + ServicePrefixMDS
+		router.PathPrefix(pre + "/").Handler(
+			http.StripPrefix(pre, s.MDS.HTTPHandler()))
+	}
+
+	// Computers.
+	for k, serviceState := range s.computers {
+		// Skip any computer which have not been started.
+		if !serviceState.started {
+			continue
+		}
+
+		pre := "/" + string(k)
+		router.PathPrefix(pre + "/").Handler(
+			http.StripPrefix(pre, serviceState.service.HTTPHandler()))
+	}
+
+	// Queryer.
+	if s.Queryer != nil {
+		pre := "/" + ServicePrefixQueryer
+		router.PathPrefix(pre + "/").Handler(
+			http.StripPrefix(pre, s.Queryer.HTTPHandler()))
+	}
+
+	return router
+}
+
+//////////////////////////////////////////
+
+// Service is an interface implemented by any service which is part of
+// ServiceManager.
+type Service interface {
+	Start() error
+	Stop() error
+	Address() Address
+	HTTPHandler() http.Handler
+}
+
+// MultiService is a service type which can have multiple instances within
+// ServicesManager.
+type MultiService interface {
+	Service
+
+	SetKey(ServiceKey)
+	Key() ServiceKey
+}
+
+type MDSService interface {
+	Service
+}
+
+type ComputerService interface {
+	MultiService
+
+	SetMDS(Address) error
+}
+
+type QueryerService interface {
+	Service
+
+	SetMDS(Address) error
+}
+
+//////////////////////////////////////////
+
+// dynamicRouter is used to dynamically swap out http routers as service states
+// withing ServiceManager change.
+type dynamicRouter struct {
+	mu     sync.RWMutex
+	router *mux.Router
+}
+
+func (dr *dynamicRouter) Swap(new *mux.Router) {
+	dr.mu.Lock()
+	defer dr.mu.Unlock()
+
+	dr.router = new
+}
+
+func (dr *dynamicRouter) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+	dr.mu.RLock()
+	router := dr.router
+	dr.mu.RUnlock()
+
+	router.ServeHTTP(w, r)
+}

dax/test/datagen/container.go

@@ -1,80 +0,0 @@
-package datagen
-
-import (
-	"strconv"
-
-	"github.com/molecula/featurebase/v3/dax"
-	"github.com/molecula/featurebase/v3/dax/test/docker"
-)
-
-var ImageName = docker.Getenv("DATAGEN_DOCKER_IMAGE", "dax/datagen")
-
-var (
-	Seed                      = "123456"
-	Target                    = "mds"
-	Source                    = "custom"
-	StartFrom                 = 1
-	EndAt                     = 1000
-	BatchSize                 = 100
-	PilosaIndex               = ""
-	FeatureBaseOrganizationID = ""
-	FeatureBaseDatabaseID     = ""
-	FeatureBaseTableName      = ""
-
-	MDSAddress = "mds:8080"
-)
-
-const (
-	NetworkName = "datagen-network"
-)
-
-type Option func(*Container)
-
-func WithEndAt(endAt int) Option {
-	return func(c *Container) {
-		c.endAt = endAt
-	}
-}
-
-type Container struct {
-	name       string
-	mdsAddress dax.Address
-	endAt      int
-}
-
-func NewContainer(name string, mdsAddress dax.Address) *Container {
-	return &Container{
-		name:       name,
-		mdsAddress: mdsAddress,
-	}
-}
-
-func (c *Container) Hostname() string {
-	return c.name
-}
-
-func (c *Container) Env() map[string]string {
-	env := map[string]string{
-		"GEN_MDS_ADDRESS":            c.mdsAddress.String(),
-		"GEN_SEED":                   docker.Getenv("GEN_SEED", Seed),
-		"GEN_TARGET":                 Target,
-		"GEN_SOURCE":                 Source,
-		"GEN_START_FROM":             strconv.Itoa(StartFrom),
-		"GEN_END_AT":                 strconv.Itoa(c.endAt),
-		"GEN_PILOSA_BATCH_SIZE":      strconv.Itoa(BatchSize),
-		"GEN_FEATUREBASE_ORG_ID":     docker.Getenv("GEN_FEATUREBASE_ORG_ID", FeatureBaseOrganizationID),
-		"GEN_FEATUREBASE_DB_ID":      docker.Getenv("GEN_FEATUREBASE_DB_ID", FeatureBaseDatabaseID),
-		"GEN_FEATUREBASE_TABLE_NAME": docker.Getenv("GEN_FEATUREBASE_TABLE_NAME", FeatureBaseTableName),
-		"GEN_CUSTOM_CONFIG":          docker.Getenv("GEN_CUSTOM_CONFIG", ""),
-	}
-
-	return env
-}
-
-func (c *Container) Cmd() []string {
-	return []string{"datagen"}
-}
-
-func (c *Container) ExposedPorts() []string {
-	return []string{}
-}

dax/test/dax/image.go

@@ -1,24 +0,0 @@
-package dax
-
-import (
-	"strings"
-	"testing"
-
-	"github.com/molecula/featurebase/v3/dax/test/docker"
-	"github.com/stretchr/testify/require"
-)
-
-func imagePull(t *testing.T, imageName ...string) {
-	t.Helper()
-	for _, img := range imageName {
-		// If the images is something other than one at docker.io, then don't
-		// perform the `docker pull`. This allows a develper to swap out the
-		// ImageName with a local image.
-		if !strings.HasPrefix(img, "docker.io/") {
-			continue
-		}
-		if err := docker.ImagePull(img); err != nil {
-			require.NoError(t, err)
-		}
-	}
-}

dax/test/dax/testdata/basic.yaml

@@ -1,52 +0,0 @@
-fields:
-  - name: "an_id"
-    type: "uint" # (default IDField (non-mutex))
-    distribution: "sequential"
-    min: 0
-    max: 2000000
-    repeat: false # if false, data generation stops when we hit >= max. only available with sequential
-    step: 1
-  - name: "an_int"
-    type: "int" # (default IntField)
-    distribution: "uniform" # uniform or zipfian
-    min: 0
-    max: 500
-  - name: "a_random_string"
-    type: "string" # (default StringField (non-mutex))
-    generator_type: "random-string" # used to generate random strings rather than pulling from known set
-    min_len: 3
-    max_len: 3
-    charset: "AB" # set of possible characters to pull from when generating random string
-  - name: "an_id_set"
-    type: "uint-set" # (default IDArrayField)
-    min: 0
-    max: 1000
-    distribution: "uniform"
-    min_num: 1
-    max_num: 6
-  - name: "a_string_set"
-    type: "string-set" # (default StringArrayField)
-    generator_type: "random-string" # used to generate random strings rather than pulling from known. "distribution" is ignored.
-    min_len: 4
-    max_len: 4
-    charset: "0123456789ABCDEF"
-    min_num: 0 # minimum number of strings in each value (default 0)
-    max_num: 10 # max number of strings (default to cardinality of source)
-
-# idk_params describe how data from "fields" should be ingested by IDK
-idk_params:
-  primary_key_config:
-    field: "id" # if this is a single field named "id" then we'll use uint IDs, if it's empty we'll autogen ids, and if it's anything else we'll do string keys... yes this is a bit hacky, needs to be cleaned up.
-  # fields is keyed by names of fields from top level "fields". It is
-  # not required that all fields appear here, those that don't will
-  # use the default ingestion.
-  fields:
-    an_id:
-      - type: "ID"
-        mutex: false
-        name: "id"
-    a_string_set:
-      - type: "StringArray"
-    a_random_string:
-      - type: "String"
-        mutex: true

dax/test/docker/client.go

@@ -1,37 +0,0 @@
-package docker
-
-import (
-	"context"
-	"os"
-
-	"github.com/docker/docker/api/types"
-	"github.com/docker/docker/client"
-)
-
-// DefaultClient is the default docker Client
-var DefaultClient *client.Client
-
-func init() {
-	cli, err := client.NewClientWithOpts()
-	if err != nil {
-		panic(err)
-	}
-	if err := client.FromEnv(cli); err != nil {
-		panic(err)
-	}
-	cli.RegistryLogin(context.Background(), types.AuthConfig{})
-
-	DefaultClient = cli
-}
-
-func Getenv(key, fallback string) string {
-	value := os.Getenv(key)
-	if len(value) == 0 {
-		return fallback
-	}
-	return value
-}
-
-func Setenv(key, value string) error {
-	return os.Setenv(key, value)
-}

dax/test/docker/docker.go

@@ -1,521 +0,0 @@
-package docker
-
-import (
-	"bufio"
-	"context"
-	"fmt"
-	"io"
-	"log"
-	"os/exec"
-	"strings"
-	"syscall"
-	"time"
-
-	"github.com/docker/docker/api/types"
-	"github.com/docker/docker/api/types/container"
-	"github.com/docker/docker/api/types/filters"
-	"github.com/docker/docker/api/types/mount"
-	"github.com/docker/docker/api/types/network"
-	"github.com/docker/docker/api/types/strslice"
-	"github.com/docker/docker/api/types/volume"
-	"github.com/docker/docker/pkg/stdcopy"
-	"github.com/docker/go-connections/nat"
-	"github.com/molecula/featurebase/v3/errors"
-	v1 "github.com/opencontainers/image-spec/specs-go/v1"
-)
-
-type Composer struct {
-	network    string
-	containers []Container
-
-	// volumes is a map of container to volume(s).
-	volumes map[string][]Volume
-}
-
-type Volume struct {
-	Type   string
-	Source string
-	Target string
-}
-
-// WithVolume creates a volume and registers it with the provided containers.
-// Note that WithVolume must be called before WithService for the applicable
-// containers.
-func (c *Composer) WithVolume(volume Volume, containers ...Container) *Composer {
-	if c.volumes == nil {
-		c.volumes = make(map[string][]Volume)
-	}
-
-	// delete previous existing networks if any to avoid creation errors
-	vol, _ := VolumeByName(volume.Source)
-	if vol != nil {
-		VolumeRemove(vol.Name)
-		VolumePrune(vol.Name)
-	}
-
-	// TODO(jaffee) I don't understand what the other volume type is
-	// being used for, but if you try to pass a bind mount here it
-	// panics, so I put a janky "if" around it.
-	if volume.Type != "bind" {
-		_, err := VolumeCreate(volume.Source)
-		if err != nil {
-			panic(err)
-		}
-	}
-
-	// Register volume with container(s).
-	for _, cont := range containers {
-		c.volumes[cont.Hostname()] = append(c.volumes[cont.Hostname()], volume)
-	}
-
-	return c
-}
-
-func (c *Composer) WithNetwork(networkName string) *Composer {
-	// delete previous existing networks if any to avoid creation errors
-	net, _ := NetworkByName(networkName)
-	NetworkRemove(net.Name)
-	NetworkPrune(net.Name)
-
-	_, err := NetworkCreate(networkName)
-	if err != nil {
-		panic(err)
-	}
-
-	for _, c := range c.containers {
-		if err = NetworkConnect(networkName, c.Hostname()); err != nil {
-			panic(err)
-		}
-	}
-	c.network = networkName
-
-	return c
-}
-
-func (c *Composer) WithService(imageName string, container ...Container) *Composer {
-	for _, cc := range container {
-		// remove previous containers with the same name
-		pc, _ := ContainerByName(cc.Hostname())
-		for _, n := range pc.Names {
-			ContainerStop(n)
-			ContainerRemove(n)
-			ContainerPrune(n)
-		}
-
-		_, err := ContainerCreate(imageName, cc, c.volumes[cc.Hostname()])
-		if err != nil {
-			panic(err)
-		}
-		c.containers = append(c.containers, cc)
-		if c.network != "" {
-			if err = NetworkConnect(c.network, cc.Hostname()); err != nil {
-				panic(err)
-			}
-		}
-	}
-
-	return c
-}
-
-func (c *Composer) Up() error {
-	for _, cc := range c.containers {
-		if err := ContainerStartWithLogging(cc.Hostname()); err != nil {
-			return err
-		}
-	}
-
-	return nil
-}
-
-func (c *Composer) Down() error {
-	var errs []error
-
-	for _, cc := range c.containers {
-		name := cc.Hostname()
-
-		if err := ContainerStop(name); err != nil {
-			log.Printf("Composer.Down error: ContainerStop: %s: %v", name, err)
-			errs = append(errs, err)
-		}
-
-		if err := ContainerRemove(name); err != nil {
-			log.Printf("Composer.Down error: ContainerRemove: %s: %v", name, err)
-			errs = append(errs, err)
-		}
-	}
-
-	if c.network != "" {
-		if err := NetworkRemove(c.network); err != nil {
-			log.Printf("Composer.Down error: NetworkRemove: %v", err)
-			errs = append(errs, err)
-		}
-	}
-
-	if len(errs) == 0 {
-		return nil
-	}
-
-	var errString strings.Builder
-
-	for i := range errs {
-		errString.WriteString(fmt.Sprintf("(%d) ", i))
-		errString.WriteString(errs[i].Error())
-		errString.WriteString(" ")
-	}
-
-	return errors.New(errors.ErrUncoded, errString.String())
-}
-
-type Container interface {
-	Hostname() string
-	ExposedPorts() []string
-	Env() map[string]string
-	Cmd() []string
-}
-
-// ImagePull pulls the selected image from internet
-func ImagePull(imageName string) error {
-	// TODO temporal. Problems with authorization
-	cmd := exec.Command("docker", "pull", imageName)
-	if err := cmd.Run(); err != nil {
-		if exiterr, ok := err.(*exec.ExitError); ok {
-			if status, ok := exiterr.Sys().(syscall.WaitStatus); ok && status.ExitStatus() > 0 {
-				return err
-			}
-		} else {
-			return err
-		}
-	}
-	return nil
-}
-
-func ContainerCreate(imageName string, c Container, volumes []Volume) (string, error) {
-	exposedPorts := make(nat.PortSet)
-	for _, p := range c.ExposedPorts() {
-		exposedPorts[nat.Port(p)] = struct{}{}
-	}
-
-	var platform *v1.Platform
-
-	mnts := make([]mount.Mount, 0)
-	for _, v := range volumes {
-		typ := v.Type
-		if typ == "" {
-			typ = string(mount.TypeVolume)
-		}
-		mnts = append(mnts, mount.Mount{
-			Type:   mount.Type(typ),
-			Source: v.Source,
-			Target: v.Target,
-		})
-	}
-
-	resp, err := DefaultClient.ContainerCreate(context.Background(),
-		&container.Config{
-			Image:        imageName,
-			Env:          envToSlice(c.Env()),
-			Hostname:     c.Hostname(),
-			ExposedPorts: exposedPorts,
-			Cmd:          strslice.StrSlice(c.Cmd()),
-		},
-		&container.HostConfig{
-			// AutoRemove:      true,
-			NetworkMode:     "bridge",
-			PublishAllPorts: true,
-			Mounts:          mnts,
-		},
-		&network.NetworkingConfig{}, platform, c.Hostname())
-	if err != nil {
-		return "", err
-	}
-	return resp.ID, err
-}
-
-func ContainerByName(containerName string) (types.Container, error) {
-	f := filters.NewArgs()
-	f.Add("name", containerName)
-
-	l, err := DefaultClient.ContainerList(context.Background(), types.ContainerListOptions{
-		Limit:   1,
-		Filters: f,
-	})
-	if err != nil {
-		return types.Container{}, err
-	}
-	if len(l) == 0 {
-		return types.Container{}, fmt.Errorf("Container %s not found", containerName)
-	}
-
-	return l[0], nil
-}
-
-func ContainerPrune(containerName string) error {
-	f := filters.NewArgs()
-	f.Add("name", containerName)
-
-	_, err := DefaultClient.ContainersPrune(context.Background(), f)
-	return err
-}
-
-func ContainerStart(containerName string) error {
-	c, err := ContainerByName(containerName)
-	if err != nil {
-		return err
-	}
-	return DefaultClient.ContainerStart(context.Background(), c.ID, types.ContainerStartOptions{})
-}
-
-func ContainerStartWithLogging(containerName string) error {
-	if err := ContainerStart(containerName); err != nil {
-		return err
-	}
-
-	go ContainerLogs(containerName)
-
-	return nil
-}
-
-func ContainerWait(containerName string) error {
-	c, err := ContainerByName(containerName)
-	if err != nil {
-		return err
-	}
-
-	var cond container.WaitCondition = container.WaitConditionNotRunning
-
-	statusCh, errCh := DefaultClient.ContainerWait(context.Background(), c.ID, cond)
-	_ = errCh
-
-	status := <-statusCh
-
-	log.Printf("ContainerWait status code: %d", status.StatusCode)
-	if status.Error != nil {
-		log.Printf("ContainerWait error: %s", status.Error.Message)
-	}
-
-	if err != nil {
-		return errors.WithMessagef(err, "ContainerWait(%s: %s) error status code: %v", c.ID, containerName, status)
-	}
-	return nil
-}
-
-func ContainerRestart(containerName string) error {
-	c, err := ContainerByName(containerName)
-	if err != nil {
-		return err
-	}
-
-	return DefaultClient.ContainerRestart(context.Background(), c.ID, nil)
-}
-
-func ContainerPauseAndResume(containerName string, timeout time.Duration) (err error) {
-	c, err := ContainerByName(containerName)
-	if err != nil {
-		return err
-	}
-
-	if err = DefaultClient.ContainerPause(context.Background(), c.ID); err != nil {
-		return errors.WithMessagef(err, "ContainerPauseAndResume(%s: %s) error", c.ID, containerName)
-	}
-
-	time.AfterFunc(timeout, func() {
-		if err = DefaultClient.ContainerUnpause(context.Background(), c.ID); err != nil {
-			err = errors.WithMessagef(err, "ContainerUnpause(%s: %s) error", c.ID, containerName)
-		}
-	})
-
-	return err
-}
-
-func ContainerLogs(containerName string) error {
-	c, err := ContainerByName(containerName)
-	if err != nil {
-		return err
-	}
-
-	reader, err := DefaultClient.ContainerLogs(context.Background(), c.ID,
-		types.ContainerLogsOptions{
-			ShowStderr: true,
-			ShowStdout: true,
-			Follow:     true,
-		})
-	if err != nil {
-		return err
-	}
-	defer reader.Close()
-
-	r, w := io.Pipe()
-
-	go func() {
-		stdcopy.StdCopy(w, w, reader)
-	}()
-
-	br := bufio.NewReader(r)
-	for {
-		line, err := br.ReadString('\n')
-
-		if err == io.EOF {
-			break
-		}
-		if err != nil {
-			return err
-		}
-
-		fmt.Print("LOGS from ", containerName, ": ", line)
-	}
-
-	return nil
-}
-
-func ContainerStop(containerName string) error {
-	c, err := ContainerByName(containerName)
-	if err != nil {
-		return err
-	}
-
-	return DefaultClient.ContainerStop(context.Background(), c.ID, nil)
-}
-
-func ContainerRemove(containerName string) error {
-	c, err := ContainerByName(containerName)
-	if err != nil {
-		return err
-	}
-
-	return DefaultClient.ContainerRemove(
-		context.Background(),
-		c.ID,
-		types.ContainerRemoveOptions{
-			Force:         true,
-			RemoveVolumes: true,
-		},
-	)
-}
-
-func VolumeCreate(volumeName string) (string, error) {
-	res, err := DefaultClient.
-		VolumeCreate(
-			context.Background(),
-			volume.VolumeCreateBody{
-				Name: volumeName,
-			},
-		)
-
-	return res.Name, err
-}
-
-func VolumeByName(volumeName string) (*types.Volume, error) {
-	f := filters.NewArgs()
-	f.Add("name", volumeName)
-
-	n, err := DefaultClient.VolumeList(context.Background(), f)
-	if err != nil {
-		return nil, err
-	}
-	if len(n.Volumes) == 0 {
-		return nil, fmt.Errorf("volume %s not found", volumeName)
-	}
-
-	return n.Volumes[0], nil
-}
-
-func VolumeRemove(volumeName string) error {
-	n, err := VolumeByName(volumeName)
-	if err != nil {
-		return err
-	}
-
-	return DefaultClient.VolumeRemove(context.Background(), n.Name, false)
-}
-
-func VolumePrune(volumeName string) error {
-	f := filters.NewArgs()
-	f.Add("name", volumeName)
-
-	_, err := DefaultClient.VolumesPrune(context.Background(), f)
-	return err
-}
-
-func NetworkCreate(networkName string) (string, error) {
-	res, err := DefaultClient.
-		NetworkCreate(
-			context.Background(),
-			networkName,
-			types.NetworkCreate{
-				CheckDuplicate: true,
-			},
-		)
-
-	return res.ID, err
-}
-
-func NetworkByName(networkName string) (types.NetworkResource, error) {
-	f := filters.NewArgs()
-	f.Add("name", networkName)
-
-	n, err := DefaultClient.NetworkList(context.Background(), types.NetworkListOptions{
-		Filters: f,
-	})
-	if err != nil {
-		return types.NetworkResource{}, err
-	}
-	if len(n) == 0 {
-		return types.NetworkResource{}, fmt.Errorf("network %s not found", networkName)
-	}
-
-	return n[0], nil
-}
-
-func NetworkRemove(networkName string) error {
-	n, err := NetworkByName(networkName)
-	if err != nil {
-		return err
-	}
-
-	return DefaultClient.NetworkRemove(context.Background(), n.ID)
-}
-
-func NetworkConnect(networkName, containerName string) error {
-	n, err := NetworkByName(networkName)
-	if err != nil {
-		return err
-	}
-
-	c, err := ContainerByName(containerName)
-	if err != nil {
-		return err
-	}
-
-	return DefaultClient.NetworkConnect(context.Background(), n.ID, c.ID, &network.EndpointSettings{})
-}
-
-func NetworkDisconnect(networkName, containerName string) error {
-	n, err := NetworkByName(networkName)
-	if err != nil {
-		return err
-	}
-
-	c, err := ContainerByName(containerName)
-	if err != nil {
-		return err
-	}
-
-	return DefaultClient.NetworkDisconnect(context.Background(), n.ID, c.ID, true)
-}
-
-func NetworkPrune(networkName string) error {
-	f := filters.NewArgs()
-	f.Add("name", networkName)
-
-	_, err := DefaultClient.NetworksPrune(context.Background(), f)
-	return err
-}
-
-func envToSlice(env map[string]string) []string {
-	var out = make([]string, 0, len(env))
-	for k, v := range env {
-		out = append(out, fmt.Sprintf("%s=%s", k, v))
-	}
-
-	return out
-}

dax/test/featurebase/container.go

@@ -1,75 +0,0 @@
-package featurebase
-
-import (
-	"fmt"
-
-	"github.com/molecula/featurebase/v3/dax/test/docker"
-)
-
-var ImageName = docker.Getenv("FEATUREBASE_DOCKER_IMAGE", "dax/featurebase-test")
-
-const (
-	DataDir     = "/data"
-	NetworkName = "featurebase-network"
-
-	HTTPPort            = "8080"
-	GRPCPort            = "20101"
-	AdvertisePeerAddr   = "2379"
-	AdvertiseClientAddr = "2380"
-)
-
-// Ensure type implements interface.
-var _ docker.Container = &Container{}
-
-type Container struct {
-	name    string
-	replica int
-	cmd     []string
-	env     map[string]string
-}
-
-func NewContainer(name string, env map[string]string) *Container {
-	// peers is just "self" because we don't want to use etcd as a cluster in
-	// the case of dumb compute nodes.
-	// peers := fmt.Sprintf("%s=http://%s:%s", name, name, AdvertisePeerAddr)
-
-	c := &Container{
-		name:    name,
-		replica: 1,
-		cmd: []string{
-			"/featurebase",
-			"-test.run=TestRunMain",
-			fmt.Sprintf("-test.coverprofile=/results/coverage-%s.out", name),
-			"dax",
-		},
-		env: map[string]string{
-			"FEATUREBASE_BIND":      "0.0.0.0:" + HTTPPort,
-			"FEATUREBASE_ADVERTISE": name + ":" + HTTPPort,
-		},
-	}
-
-	// Apply given env vars.
-	for k, v := range env {
-		c.env[k] = v
-	}
-
-	return c
-}
-
-func (c *Container) Hostname() string {
-	return c.name
-}
-
-func (c *Container) ExposedPorts() []string {
-	// We don't expose HTTPPort, because it's already exposed by default by
-	// featurebase docker image.
-	return []string{GRPCPort, AdvertisePeerAddr, AdvertiseClientAddr}
-}
-
-func (c *Container) Cmd() []string {
-	return c.cmd
-}
-
-func (c *Container) Env() map[string]string {
-	return c.env
-}

dax/test/inspector/inspector.go

@@ -1,217 +0,0 @@
-package inspector
-
-import (
-	"bytes"
-	"context"
-	"encoding/json"
-	"fmt"
-	"io"
-	"log"
-	"strings"
-
-	"github.com/docker/docker/api/types"
-	"github.com/docker/docker/client"
-	"github.com/docker/docker/pkg/stdcopy"
-	"github.com/molecula/featurebase/v3/dax"
-	mdshttp "github.com/molecula/featurebase/v3/dax/mds/http"
-	queryerhttp "github.com/molecula/featurebase/v3/dax/queryer/http"
-	"github.com/molecula/featurebase/v3/dax/test/docker"
-	"github.com/molecula/featurebase/v3/errors"
-)
-
-type Inspector struct {
-	cli *client.Client
-
-	// containers is a map of container name to container ID.
-	containers map[string]string
-}
-
-func NewInspector() (*Inspector, error) {
-	cli, err := client.NewClientWithOpts(client.FromEnv)
-	if err != nil {
-		return nil, errors.Wrap(err, "getting new client with opts")
-	}
-
-	containers, err := cli.ContainerList(context.Background(), types.ContainerListOptions{})
-	if err != nil {
-		return nil, errors.Wrap(err, "getting container list")
-	}
-
-	m := make(map[string]string)
-	for _, container := range containers {
-		if len(container.Names) == 0 {
-			continue
-		}
-		m[container.Names[0]] = container.ID[:10]
-	}
-
-	return &Inspector{
-		cli:        cli,
-		containers: m,
-	}, nil
-}
-
-type ExecRequest struct {
-	address dax.Address
-	path    string
-	data    interface{}
-}
-
-func NewExecRequest(address dax.Address, path string, data interface{}) *ExecRequest {
-	return &ExecRequest{
-		address: address,
-		path:    path,
-		data:    data,
-	}
-}
-
-func (e *ExecRequest) Cmd() []string {
-	address := e.address
-	if address == "" {
-		address = "http://localhost:8080"
-	}
-	uri := fmt.Sprintf("%s/%s", address, e.path)
-
-	if e.data == nil {
-		return []string{
-			"curl",
-			uri,
-		}
-	}
-
-	var dArgs string
-	switch v := e.data.(type) {
-	case string:
-		dArgs = v
-	case *dax.Table,
-		*dax.QualifiedTable,
-		mdshttp.RegisterNodeRequest,
-		mdshttp.SnapshotFieldKeysRequest,
-		mdshttp.SnapshotShardRequest,
-		mdshttp.TranslateNodesRequest,
-		queryerhttp.QueryRequest,
-		queryerhttp.SQLRequest,
-		dax.QualifiedTableID:
-		b, err := json.Marshal(v)
-		if err != nil {
-			panic(err) // FIX THIS
-		}
-		dArgs = string(b)
-	default:
-		log.Printf("unhandled return type: %T", e.data)
-		dArgs = "{}"
-	}
-
-	return []string{
-		"curl",
-		"-d " + dArgs,
-		uri,
-	}
-}
-
-type ExecResponse struct {
-	stdOut   string
-	stdErr   string
-	exitCode int
-}
-
-func (e *ExecResponse) Out() string {
-	return strings.TrimSpace(e.stdOut)
-}
-
-func (e *ExecResponse) Err() string {
-	return strings.TrimSpace(e.stdErr)
-}
-
-func (i *Inspector) Close() error {
-	if i.cli != nil {
-		return i.cli.Close()
-	}
-	return nil
-}
-
-// ExecResp is a helper method which runs exec() then resp().
-func (i *Inspector) ExecResp(ctx context.Context, container docker.Container, command []string) (ExecResponse, error) {
-	var execResp ExecResponse
-
-	exec, err := i.exec(ctx, container, command)
-	if err != nil {
-		return execResp, err
-	}
-
-	return i.resp(context.Background(), exec.ID)
-}
-
-func (i *Inspector) exec(ctx context.Context, container docker.Container, command []string) (types.IDResponse, error) {
-	hostName := container.Hostname()
-
-	containerID, ok := i.containers[slash(hostName)]
-	if !ok {
-		return types.IDResponse{}, errors.Errorf("invalid container: %s", hostName)
-	}
-
-	config := types.ExecConfig{
-		AttachStderr: true,
-		AttachStdout: true,
-		Cmd:          command,
-	}
-
-	return i.cli.ContainerExecCreate(ctx, containerID, config)
-}
-
-func (i *Inspector) resp(ctx context.Context, id string) (ExecResponse, error) {
-	var execResp ExecResponse
-
-	resp, err := i.cli.ContainerExecAttach(ctx, id, types.ExecStartCheck{})
-	if err != nil {
-		return execResp, err
-	}
-	defer resp.Close()
-
-	// read the output
-	var outBuf, errBuf bytes.Buffer
-	outputDone := make(chan error)
-
-	go func() {
-		// StdCopy demultiplexes the stream into two buffers
-		_, err = stdcopy.StdCopy(&outBuf, &errBuf, resp.Reader)
-		outputDone <- err
-	}()
-
-	select {
-	case err := <-outputDone:
-		if err != nil {
-			return execResp, err
-		}
-		break
-
-	case <-ctx.Done():
-		return execResp, ctx.Err()
-	}
-
-	stdout, err := io.ReadAll(&outBuf)
-	if err != nil {
-		return execResp, err
-	}
-	stderr, err := io.ReadAll(&errBuf)
-	if err != nil {
-		return execResp, err
-	}
-
-	res, err := i.cli.ContainerExecInspect(ctx, id)
-	if err != nil {
-		return execResp, err
-	}
-
-	execResp.exitCode = res.ExitCode
-	execResp.stdOut = string(stdout)
-	execResp.stdErr = string(stderr)
-	return execResp, nil
-}
-
-// slash is a helper functions which addresses the fact that the lower level
-// docker inspect functions actually store the container name with a leading
-// slash. See this for more info: https://github.com/moby/moby/issues/6705
-func slash(s string) string {
-	return "/" + s
-}

dax/versioned_partition.go

@@ -8,6 +8,10 @@ type PartitionNum int
 // PartitionNums is a slice of PartitionNum.
 type PartitionNums []PartitionNum
 
+func (p PartitionNums) Len() int           { return len(p) }
+func (p PartitionNums) Less(i, j int) bool { return p[i] < p[j] }
+func (p PartitionNums) Swap(i, j int)      { p[i], p[j] = p[j], p[i] }
+
 // String returns the PartitionNum as a string.
 func (p PartitionNum) String() string {
 	return fmt.Sprintf("%d", p)

disco/disco.go

@@ -155,6 +155,57 @@ func (n *nopDisCo) DeleteNode(context.Context, string) error {
 	return nil
 }
 
+// Ensure type implements interface.
+var _ DisCo = &inMemDisCo{}
+
+func NewInMemDisCo(id string) *inMemDisCo {
+	return &inMemDisCo{
+		id: id,
+	}
+}
+
+// inMemDisCo represents a DisCo that is aware of itself.
+type inMemDisCo struct {
+	id string
+}
+
+// Close no-op.
+func (n *inMemDisCo) Close() error {
+	return nil
+}
+
+// Start is a no-op implementation of the DisCo Start method.
+func (n *inMemDisCo) Start(ctx context.Context) (InitialClusterState, error) {
+	return InitialClusterStateNew, nil
+}
+
+// ID is a no-op implementation of the DisCo ID method.
+func (n *inMemDisCo) ID() string {
+	return n.id
+}
+
+// IsLeader is a no-op implementation of the DisCo IsLeader method.
+func (n *inMemDisCo) IsLeader() bool {
+	return true
+}
+
+// Leader is a no-op implementation of the DisCo Leader method.
+func (n *inMemDisCo) Leader() *Peer {
+	return nil
+}
+
+// Peers is a no-op implementation of the DisCo Peers method.
+func (n *inMemDisCo) Peers() []*Peer {
+	return nil
+}
+
+// DeleteNode a no-op implementation of the DisCo DeleteNode method.
+func (n *inMemDisCo) DeleteNode(context.Context, string) error {
+	return nil
+}
+
+////////////////////////////////////////////////
+
 // NopSharder represents a Sharder that doesn't do anything.
 var NopSharder Sharder = &nopSharder{}
 

disco/noder.go

@@ -81,9 +81,9 @@ func (n *localNoder) PrimaryNodeID(hasher Hasher) string {
 	return primaryNode.ID
 }
 
-// ClusterState is a no-op implementation of the Stator ClusterState method.
+// ClusterState for localNoder just assumes the cluster is normal.
 func (n *localNoder) ClusterState(context.Context) (ClusterState, error) {
-	return ClusterStateUnknown, nil
+	return ClusterStateNormal, nil
 }
 
 func (n *localNoder) SetState(ctx context.Context, state NodeState) error {

executor.go

@@ -19,14 +19,15 @@ import (
 
 	"github.com/gomem/gomem/pkg/dataframe"
 	"github.com/lib/pq"
-	"github.com/featurebasedb/featurebase/v3/disco"
-	"github.com/featurebasedb/featurebase/v3/pql"
-	"github.com/featurebasedb/featurebase/v3/proto"
-	"github.com/featurebasedb/featurebase/v3/roaring"
-	"github.com/featurebasedb/featurebase/v3/shardwidth"
-	"github.com/featurebasedb/featurebase/v3/task"
-	"github.com/featurebasedb/featurebase/v3/testhook"
-	"github.com/featurebasedb/featurebase/v3/tracing"
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/disco"
+	"github.com/molecula/featurebase/v3/pql"
+	"github.com/molecula/featurebase/v3/proto"
+	"github.com/molecula/featurebase/v3/roaring"
+	"github.com/molecula/featurebase/v3/shardwidth"
+	"github.com/molecula/featurebase/v3/task"
+	"github.com/molecula/featurebase/v3/testhook"
+	"github.com/molecula/featurebase/v3/tracing"
 	"github.com/pkg/errors"
 	"golang.org/x/sync/errgroup"
 )
@@ -6033,7 +6034,8 @@ func (e *executor) remoteExec(ctx context.Context, node *disco.Node, index strin
 		MaxMemory:    maxMemory,
 	}
 
-	resp, err := e.client.QueryNode(ctx, &node.URI, index, pbreq)
+	addr := dax.Address(node.URI.String())
+	resp, err := e.client.QueryNode(ctx, addr, index, pbreq)
 	if err != nil {
 		return nil, err
 	}

go.mod

@@ -20,10 +20,6 @@ require (
 	github.com/confluentinc/confluent-kafka-go v1.9.1
 	github.com/davecgh/go-spew v1.1.1
 	github.com/denisenkom/go-mssqldb v0.11.0
-	github.com/docker/distribution v2.8.1+incompatible // indirect
-	github.com/docker/docker v20.10.17+incompatible
-	github.com/docker/go-connections v0.4.0
-	github.com/docker/go-units v0.4.0 // indirect
 	github.com/felixge/fgprof v0.9.2
 	github.com/getsentry/sentry-go v0.13.0
 	github.com/glycerine/vprint v0.0.0-20200730000117-76cea49a68ea
@@ -44,7 +40,6 @@ require (
 	github.com/jedib0t/go-pretty v4.3.0+incompatible
 	github.com/lib/pq v1.10.5
 	github.com/molecula/apophenia v0.0.0-20190827192002-68b7a14a478b
-	github.com/opencontainers/image-spec v1.0.2
 	github.com/opentracing/opentracing-go v1.2.0
 	github.com/pelletier/go-toml v1.9.5
 	github.com/pkg/errors v0.9.1
@@ -147,12 +142,9 @@ require (
 	github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8 // indirect
 	github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3 // indirect
 	github.com/mitchellh/mapstructure v1.5.0 // indirect
-	github.com/moby/term v0.0.0-20210619224110-3f7ff695adc6 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.2 // indirect
-	github.com/morikuni/aec v1.0.0 // indirect
 	github.com/oklog/ulid v1.3.1 // indirect
-	github.com/opencontainers/go-digest v1.0.0 // indirect
 	github.com/pierrec/lz4/v4 v4.1.15 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
 	github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
@@ -196,7 +188,6 @@ require (
 	gopkg.in/ini.v1 v1.62.0 // indirect
 	gopkg.in/natefinch/lumberjack.v2 v2.0.0 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
-	gotest.tools/v3 v3.0.2 // indirect
 	nhooyr.io/websocket v1.8.6 // indirect
 )
 

go.sum

@@ -39,8 +39,6 @@ cloud.google.com/go/storage v1.6.0/go.mod h1:N7U0C8pVQ/+NIKOBQyamJIeKQKkZ+mxpohl
 cloud.google.com/go/storage v1.8.0/go.mod h1:Wv1Oy7z6Yz3DshWRJFhqM/UCfaWIRTdp0RXyy7KQOVs=
 cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9ullr3+Kg0=
 dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
-github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 h1:UQHMgLO+TxOElx5B5HZ4hJQsoJ/PvUvKRhJHDQXO8P8=
-github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
 github.com/Azure/go-autorest/autorest v0.9.0/go.mod h1:xyHB1BMZT0cuDHU7I0+g046+BFDTQ8rEZB0s4Yfa6bI=
 github.com/Azure/go-autorest/autorest/adal v0.5.0/go.mod h1:8Z9fGy2MpX0PvDjB1pEgQTmVqjGhiHBW7RJJEciWzS0=
 github.com/Azure/go-autorest/autorest/date v0.1.0/go.mod h1:plvfp3oPSKwf2DNjlBjWF/7vwR+cUD/ELuzDCXwHUVA=
@@ -233,14 +231,6 @@ github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZm
 github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw=
 github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
 github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
-github.com/docker/distribution v2.8.1+incompatible h1:Q50tZOPR6T/hjNsyc9g8/syEs6bk8XXApsHjKukMl68=
-github.com/docker/distribution v2.8.1+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w=
-github.com/docker/docker v20.10.17+incompatible h1:JYCuMrWaVNophQTOrMMoSwudOVEfcegoZZrleKc1xwE=
-github.com/docker/docker v20.10.17+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
-github.com/docker/go-connections v0.4.0 h1:El9xVISelRB7BuFusrZozjnkIM5YnzCViNKohAFqRJQ=
-github.com/docker/go-connections v0.4.0/go.mod h1:Gbd7IOopHjR8Iph03tsViu4nIes5XhDvyHbTtUxmeec=
-github.com/docker/go-units v0.4.0 h1:3uh0PgVws3nIA0Q+MwDC8yjEPf9zjRfZZWXZYDct3Tw=
-github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
 github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96/go.mod h1:Qh8CwZgvJUkLughtfhJv5dyTYa91l1fOUCrgjqmcifM=
 github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
 github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=
@@ -821,8 +811,6 @@ github.com/mitchellh/mapstructure v1.4.2/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RR
 github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY=
 github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
 github.com/mitchellh/reflectwalk v1.0.0/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw=
-github.com/moby/term v0.0.0-20210619224110-3f7ff695adc6 h1:dcztxKSvZ4Id8iPpHERQBbIJfabdt4wUm5qy3wOL2Zc=
-github.com/moby/term v0.0.0-20210619224110-3f7ff695adc6/go.mod h1:E2VnQOmVuvZB6UYnnDB0qG5Nq/1tD9acaOpo6xmt0Kw=
 github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
 github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
 github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
@@ -834,8 +822,6 @@ github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjY
 github.com/molecula/apophenia v0.0.0-20190827192002-68b7a14a478b h1:cZADDaNYM7xn/nklO3g198JerGQjadFuA0ofxBJgK0Y=
 github.com/molecula/apophenia v0.0.0-20190827192002-68b7a14a478b/go.mod h1:uXd1BiH7xLmgkhVmspdJLENv6uGWrTL/MQX2TN7Yz9s=
 github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe/go.mod h1:wL8QJuTMNUDYhXwkmfOly8iTdp5TEcJFWZD2D7SIkUc=
-github.com/morikuni/aec v1.0.0 h1:nP9CBfwrvYnBRgY6qfDQkygYDmYwOilePFkwzv4dU8A=
-github.com/morikuni/aec v1.0.0/go.mod h1:BbKIizmSmc5MMPqRYbxO4ZU0S0+P200+tUnFx7PXmsc=
 github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
 github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f h1:KUppIJq7/+SVif2QVs3tOP0zanoHgBEVAwHxUSIzRqU=
@@ -874,10 +860,6 @@ github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1y
 github.com/onsi/gomega v1.10.2/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo=
 github.com/onsi/gomega v1.16.0/go.mod h1:HnhC7FXeEQY45zxNK3PPoIUhzk/80Xly9PcubAlGdZY=
 github.com/op/go-logging v0.0.0-20160315200505-970db520ece7/go.mod h1:HzydrMdWErDVzsI23lYNej1Htcns9BCg93Dk0bBINWk=
-github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
-github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
-github.com/opencontainers/image-spec v1.0.2 h1:9yCKha/T5XdGtO0q9Q9a6T5NUCsTn/DrBg0D7ufOcFM=
-github.com/opencontainers/image-spec v1.0.2/go.mod h1:BtxoFyWECRxE4U/7sNtV5W15zMzWCbyJoFRP3s7yZA0=
 github.com/opentracing-contrib/go-observer v0.0.0-20170622124052-a52f23424492/go.mod h1:Ngi6UdF0k5OKD5t5wlmGhe/EDKPoUM3BXZSSfIuJbis=
 github.com/opentracing/basictracer-go v1.0.0/go.mod h1:QfBfYuafItcjQuMwinw9GhYKwFXS9KnPs5lxoYwgW74=
 github.com/opentracing/opentracing-go v1.0.2/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
@@ -1478,7 +1460,6 @@ golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBc
 golang.org/x/sys v0.0.0-20210514084401-e8d321eab015/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
@@ -1538,7 +1519,6 @@ golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBn
 golang.org/x/tools v0.0.0-20190531172133-b3315ee88b7d/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
 golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
 golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
-golang.org/x/tools v0.0.0-20190624222133-a101b041ded4/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
 golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
 golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20190823170909-c4a336ef6a2f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
@@ -1787,8 +1767,6 @@ gorm.io/driver/sqlserver v1.0.4/go.mod h1:ciEo5btfITTBCj9BkoUVDvgQbUdLWQNqdFY5OG
 gorm.io/gorm v1.9.19/go.mod h1:0HFTzE/SqkGTzK6TlDPPQbAYCluiVvhzoA1+aVyzenw=
 gorm.io/gorm v1.20.0/go.mod h1:0HFTzE/SqkGTzK6TlDPPQbAYCluiVvhzoA1+aVyzenw=
 gorm.io/gorm v1.20.6/go.mod h1:0HFTzE/SqkGTzK6TlDPPQbAYCluiVvhzoA1+aVyzenw=
-gotest.tools/v3 v3.0.2 h1:kG1BFyqVHuQoVQiR1bWGnfz/fmHvvuiSPIV7rvl360E=
-gotest.tools/v3 v3.0.2/go.mod h1:3SzNCllyD9/Y+b5r9JIKQ474KzkZyqLqEfYqMsX94Bk=
 honnef.co/go/tools v0.0.0-20180728063816-88497007e858/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=

holder.go

@@ -140,6 +140,14 @@ type Holder struct {
 	// directive is the latest directive applied to the node.
 	directive *dax.Directive
 
+	// directiveApplied is used for testing (in an attempt to avoid sleeps). It
+	// should be removed once we sort out the logic between MDS and computer
+	// nodes. For example, MDS really needs to send out directives
+	// asynchronously and allow a computer to load data from
+	// snapshotter/writelogger; then MDS should only start directing queries to
+	// that computer once it has completed applying the snapshot.
+	directiveApplied bool
+
 	versionStore dax.VersionStore
 }
 
@@ -173,9 +181,27 @@ func (h *Holder) SetDirective(d *dax.Directive) {
 	// of the existing directive's version.
 	if h.directive == nil || d.Version > h.directive.Version {
 		h.directive = d
+		h.directiveApplied = false
 	}
 }
 
+// DirectiveApplied returns true if the Holder's latest directive has been fully
+// applied and is safe for queries. This is primarily used in testing and will
+// likely evolve to something smarter.
+func (h *Holder) DirectiveApplied() bool {
+	h.mu.RLock()
+	defer h.mu.RUnlock()
+	return h.directiveApplied
+}
+
+// SetDirectiveApplied sets the value of directiveApplied. See the node on the
+// DirectiveApplied method.
+func (h *Holder) SetDirectiveApplied(a bool) {
+	h.mu.Lock()
+	defer h.mu.Unlock()
+	h.directiveApplied = a
+}
+
 func (h *Holder) StartTransaction(ctx context.Context, id string, timeout time.Duration, exclusive bool) (*Transaction, error) {
 	return h.transactionManager.Start(ctx, id, timeout, exclusive)
 }

http_handler.go

@@ -621,6 +621,7 @@ func newRouter(handler *Handler) http.Handler {
 	router.HandleFunc("/internal/oauth-config", handler.handleOAuthConfig).Methods("GET").Name("GetOAuthConfig")
 
 	router.HandleFunc("/health", handler.handleGetHealth).Methods("GET").Name("GetHealth")
+	router.HandleFunc("/directive", handler.handleGetDirective).Methods("GET").Name("GetDirective")
 	router.HandleFunc("/directive", handler.handlePostDirective).Methods("POST").Name("PostDirective")
 	router.HandleFunc("/snapshot/shard-data", handler.handlePostSnapshotShardData).Methods("POST").Name("PostShapshotShardData")
 	router.HandleFunc("/snapshot/table-keys", handler.handlePostSnapshotTableKeys).Methods("POST").Name("PostShapshotTableKeys")
@@ -4000,6 +4001,30 @@ func getTokens(r *http.Request) (string, string) {
 	return access, refresh
 }
 
+func (h *Handler) handleGetDirective(w http.ResponseWriter, r *http.Request) {
+	if !validHeaderAcceptJSON(r.Header) {
+		http.Error(w, "JSON only acceptable response", http.StatusNotAcceptable)
+		return
+	}
+
+	applied, err := h.api.DirectiveApplied(r.Context())
+	if err != nil {
+		http.Error(w, "getting directive applied error: "+err.Error(), http.StatusInternalServerError)
+		return
+	}
+
+	resp := struct {
+		Applied bool `json:"applied"`
+	}{
+		Applied: applied,
+	}
+
+	w.Header().Set("Content-Type", "application/json")
+	if err := json.NewEncoder(w).Encode(resp); err != nil {
+		h.logger.Errorf("write status response error: %s", err)
+	}
+}
+
 func (h *Handler) handlePostDirective(w http.ResponseWriter, r *http.Request) {
 	if !validHeaderAcceptJSON(r.Header) {
 		http.Error(w, "JSON only acceptable response", http.StatusNotAcceptable)

idk/Makefile

@@ -164,7 +164,7 @@ start-all: testenv build-wait
 	$(DOCKER_COMPOSE) run -T  wait pilosa curl --silent --fail http://pilosa:10101/status
 	$(DOCKER_COMPOSE) run -T  wait pilosa-tls curl --silent --cacert /certs/ca.crt --key /certs/theclient.key --cert /certs/theclient.crt --fail https://pilosa-tls:10111/status
 	$(DOCKER_COMPOSE) run -T  wait pilosa-auth curl --silent --fail http://pilosa-auth:10105/version
-	$(DOCKER_COMPOSE) run -T wait dax curl --silent --fail http://dax:8080/computer/status
+	$(DOCKER_COMPOSE) run -T wait dax curl --silent --fail http://dax:8080/computer0/status
 	$(DOCKER_COMPOSE) run -T  wait kafka nc -z kafka 9092
 	$(DOCKER_COMPOSE) run -T  wait schema-registry curl --silent --fail http://schema-registry:8081/config
 

idk/docker-compose.yml

@@ -139,14 +139,12 @@ services:
     build:
       context: ..
       dockerfile: Dockerfile-dax
-    image: idk_dax:latest
     environment:
       FEATUREBASE_BIND: 0.0.0.0:8080
       FEATUREBASE_VERBOSE: "true"
-      FEATUREBASE_STORAGE_METHOD: boltdb
-      FEATUREBASE_STORAGE_DSN: file:/dax-data/mds.boldtb
       FEATUREBASE_QUERYER_RUN: "true"
       FEATUREBASE_MDS_RUN: "true"
+      FEATUREBASE_MDS_CONFIG_DATA_DIR: /dax-data/mds
       FEATUREBASE_WRITELOGGER_RUN: "true"
       FEATUREBASE_WRITELOGGER_CONFIG_DATA_DIR: /dax-data/wl
       FEATUREBASE_SNAPSHOTTER_RUN: "true"

idk/ingest.go

@@ -986,8 +986,18 @@ func (m *Main) setupClient() (*tls.Config, error) {
 			pilosaclient.OptClientPoolSizePerRoute(400),
 		)
 	}
+
 	if m.useMDS() {
-		opts = append(opts, pilosaclient.OptClientPathPrefix(dax.ServicePrefixComputer))
+		// We should only have one "pilosa host" here. Get the path from that
+		// and use it as the client path prefix.
+		var prefix string
+		for i := range m.PilosaHosts {
+			addr := dax.Address(m.PilosaHosts[i])
+			prefix = addr.Path()
+			m.PilosaHosts = []string{addr.HostPort()}
+			break
+		}
+		opts = append(opts, pilosaclient.OptClientPathPrefix(prefix))
 	}
 
 	m.client, err = pilosaclient.NewClient(m.PilosaHosts, opts...)
@@ -1001,7 +1011,7 @@ func (m *Main) setupClient() (*tls.Config, error) {
 
 		// MDS doesn't auto-create a table based on IDK ingest; the table must
 		// already exist.
-		mdsClient := mdsclient.New(dax.Address(m.MDSAddress))
+		mdsClient := mdsclient.New(dax.Address(m.MDSAddress), m.log)
 		qual := dax.NewTableQualifier(m.OrganizationID, m.DatabaseID)
 		qtid, err := mdsClient.TableID(ctx, qual, m.TableName)
 		if err != nil {
@@ -1014,7 +1024,7 @@ func (m *Main) setupClient() (*tls.Config, error) {
 
 		m.Qtbl = qtbl
 		m.Index = string(qtbl.Key())
-		m.SchemaManager = mds.NewSchemaManager(dax.Address(m.MDSAddress), qual)
+		m.SchemaManager = mds.NewSchemaManager(dax.Address(m.MDSAddress), qual, m.log)
 
 		m.NewImporterFn = func() pilosabatch.Importer {
 			return mds.NewImporter(mdsClient, qtbl)

idk/ingest_test.go

@@ -56,10 +56,10 @@ func configureTestFlagsMDS(main *Main, address dax.Address, qtbl *dax.QualifiedT
 	main.DatabaseID = qtbl.Qualifier().DatabaseID
 	main.TableName = qtbl.Name
 	main.Qtbl = qtbl
-	main.SchemaManager = mds.NewSchemaManager(address, qtbl.Qualifier())
+	main.SchemaManager = mds.NewSchemaManager(address, qtbl.Qualifier(), logger.StderrLogger)
 	main.Index = string(qtbl.Key())
 
-	mdsClient := mdsclient.New(dax.Address(address))
+	mdsClient := mdsclient.New(dax.Address(address), logger.StderrLogger)
 	main.NewImporterFn = func() batch.Importer {
 		return mds.NewImporter(mdsClient, qtbl)
 	}
@@ -1755,7 +1755,7 @@ func TestBatchTargetMDS(t *testing.T) {
 		mdsHost = "dax:8080"
 	}
 
-	mdsAddress := dax.Address(mdsHost)
+	mdsAddress := dax.Address(mdsHost + "/" + dax.ServicePrefixMDS)
 	orgID := dax.OrganizationID("acme")
 	dbID := dax.DatabaseID("db1")
 
@@ -1867,7 +1867,7 @@ func TestBatchTargetMDS(t *testing.T) {
 				ctx := context.Background()
 
 				// Create the table in MDS Schemar.
-				mdsClient := mdsclient.New(mdsAddress)
+				mdsClient := mdsclient.New(mdsAddress, logger.StderrLogger)
 				if err := mdsClient.CreateTable(ctx, qtbl); err != nil {
 					t.Fatalf("creating table: %v", err)
 				}

idk/mds/importer.go

@@ -38,11 +38,11 @@ func NewImporter(mds MDS, qtbl *dax.QualifiedTable) *importer {
 // have a client for.
 func (m *importer) fbClient(address dax.Address) (*featurebaseclient.Client, error) {
 	// Set up a FeatureBase client with address.
-	return featurebaseclient.NewClient(address.String(),
+	return featurebaseclient.NewClient(address.HostPort(),
 		featurebaseclient.OptClientRetries(2),
 		featurebaseclient.OptClientTotalPoolSize(1000),
 		featurebaseclient.OptClientPoolSizePerRoute(400),
-		featurebaseclient.OptClientPathPrefix(dax.ServicePrefixComputer),
+		featurebaseclient.OptClientPathPrefix(address.Path()),
 		//featurebaseclient.OptClientStatsClient(m.stats),
 	)
 }

idk/mds/schemamanager.go

@@ -9,6 +9,7 @@ import (
 	"github.com/molecula/featurebase/v3/dax"
 	mdsclient "github.com/molecula/featurebase/v3/dax/mds/client"
 	"github.com/molecula/featurebase/v3/errors"
+	"github.com/molecula/featurebase/v3/logger"
 )
 
 // Ensure type implements interface.
@@ -18,12 +19,14 @@ import (
 type schemaManager struct {
 	client *mdsclient.Client
 	qual   dax.TableQualifier
+	logger logger.Logger
 }
 
-func NewSchemaManager(mdsAddress dax.Address, qual dax.TableQualifier) *schemaManager {
+func NewSchemaManager(mdsAddress dax.Address, qual dax.TableQualifier, logger logger.Logger) *schemaManager {
 	return &schemaManager{
-		client: mdsclient.New(mdsAddress),
+		client: mdsclient.New(mdsAddress, logger),
 		qual:   qual,
+		logger: logger,
 	}
 }
 

internal_client.go

@@ -21,6 +21,12 @@ import (
 	fbcontext "github.com/molecula/featurebase/v3/context"
 
 	"github.com/hashicorp/go-retryablehttp"
+	"github.com/molecula/featurebase/v3/authn"
+	"github.com/molecula/featurebase/v3/dax"
+	"github.com/molecula/featurebase/v3/disco"
+	"github.com/molecula/featurebase/v3/logger"
+	pnet "github.com/molecula/featurebase/v3/net"
+	"github.com/molecula/featurebase/v3/tracing"
 	"github.com/pkg/errors"
 	"golang.org/x/oauth2"
 )
@@ -588,11 +594,12 @@ func (c *InternalClient) Nodes(ctx context.Context) ([]*disco.Node, error) {
 func (c *InternalClient) Query(ctx context.Context, index string, queryRequest *QueryRequest) (*QueryResponse, error) {
 	span, ctx := tracing.StartSpanFromContext(ctx, "InternalClient.Query")
 	defer span.Finish()
-	return c.QueryNode(ctx, c.defaultURI, index, queryRequest)
+	addr := dax.Address(c.defaultURI.String())
+	return c.QueryNode(ctx, addr, index, queryRequest)
 }
 
 // QueryNode executes query against the index, sending the request to the node specified.
-func (c *InternalClient) QueryNode(ctx context.Context, uri *pnet.URI, index string, queryRequest *QueryRequest) (*QueryResponse, error) {
+func (c *InternalClient) QueryNode(ctx context.Context, addr dax.Address, index string, queryRequest *QueryRequest) (*QueryResponse, error) {
 	span, ctx := tracing.StartSpanFromContext(ctx, "QueryNode")
 	defer span.Finish()
 
@@ -607,7 +614,7 @@ func (c *InternalClient) QueryNode(ctx context.Context, uri *pnet.URI, index str
 	}
 
 	// Create HTTP request.
-	u := uri.Path(fmt.Sprintf("%s/index/%s/query", c.prefix(), index))
+	u := fmt.Sprintf("%s/index/%s/query", addr.WithScheme("http"), index)
 	req, err := http.NewRequest("POST", u, bytes.NewReader(buf))
 	if err != nil {
 		return nil, errors.Wrap(err, "creating request")

server.go

@@ -658,11 +658,16 @@ func (s *Server) Open() error {
 	// Set node ID.
 	s.nodeID = s.disCo.ID()
 
+	nodeState := disco.NodeStateUnknown
+	if s.cluster.isComputeNode {
+		nodeState = disco.NodeStateStarted
+	}
+
 	node := &disco.Node{
 		ID:        s.nodeID,
 		URI:       s.uri,
 		GRPCURI:   s.grpcURI,
-		State:     disco.NodeStateUnknown,
+		State:     nodeState,
 		IsPrimary: s.IsPrimary(),
 	}
 

server/config.go

@@ -84,7 +84,7 @@ type Config struct {
 	// This is for use by test infrastructure, where it's useful to
 	// be able to dynamically generate the bindings by actually binding
 	// to :0, and avoid "address already in use" errors.
-	GRPCListener *net.TCPListener
+	GRPCListener net.Listener
 
 	// Advertise is the address advertised by the server to other nodes
 	// in the cluster. It should be reachable by all other nodes and should

server/server.go

@@ -26,17 +26,15 @@ import (
 	"syscall"
 	"time"
 
-	"github.com/molecula/featurebase/v3/dax"
-	"github.com/molecula/featurebase/v3/systemlayer"
-	"golang.org/x/sync/errgroup"
-
 	pilosa "github.com/molecula/featurebase/v3"
 	"github.com/molecula/featurebase/v3/authn"
 	"github.com/molecula/featurebase/v3/authz"
 	"github.com/molecula/featurebase/v3/batch"
 	"github.com/molecula/featurebase/v3/boltdb"
+	"github.com/molecula/featurebase/v3/dax"
 	"github.com/molecula/featurebase/v3/dax/computer"
 	"github.com/molecula/featurebase/v3/dax/computer/alpha"
+	"github.com/molecula/featurebase/v3/disco"
 	"github.com/molecula/featurebase/v3/encoding/proto"
 	petcd "github.com/molecula/featurebase/v3/etcd"
 	"github.com/molecula/featurebase/v3/gcnotify"
@@ -49,10 +47,12 @@ import (
 	"github.com/molecula/featurebase/v3/statik"
 	"github.com/molecula/featurebase/v3/stats"
 	"github.com/molecula/featurebase/v3/statsd"
+	"github.com/molecula/featurebase/v3/systemlayer"
 	"github.com/molecula/featurebase/v3/syswrap"
 	"github.com/molecula/featurebase/v3/testhook"
 	"github.com/pelletier/go-toml"
 	"github.com/pkg/errors"
+	"golang.org/x/sync/errgroup"
 )
 
 type loggerLogger interface {
@@ -77,9 +77,9 @@ type Command struct {
 	logger         loggerLogger
 	queryLogger    loggerLogger
 
-	mds         pilosa.MDS
-	writeLogger pilosa.WriteLogger
-	snapshotter pilosa.Snapshotter
+	Registrar   computer.Registrar
+	writeLogger computer.WriteLogger
+	snapshotter computer.Snapshotter
 
 	Handler      pilosa.HandlerI
 	httpHandler  http.Handler
@@ -150,9 +150,6 @@ func OptCommandSetConfig(config *Config) CommandOption {
 // OptCommandInjections injects the interface implementations.
 func OptCommandInjections(inj Injections) CommandOption {
 	return func(c *Command) error {
-		if inj.MDS != nil {
-			c.mds = inj.MDS
-		}
 		if inj.WriteLogger != nil {
 			c.writeLogger = inj.WriteLogger
 		}
@@ -165,9 +162,8 @@ func OptCommandInjections(inj Injections) CommandOption {
 }
 
 type Injections struct {
-	MDS           pilosa.MDS
-	WriteLogger   pilosa.WriteLogger
-	Snapshotter   pilosa.Snapshotter
+	WriteLogger   computer.WriteLogger
+	Snapshotter   computer.Snapshotter
 	IsComputeNode bool
 }
 
@@ -269,10 +265,7 @@ func (m *Command) setupResourceLimits() error {
 }
 
 // StartNoServe starts the pilosa server, but doesn't serve on the http handler.
-func (m *Command) StartNoServe() (err error) {
-	// Seed random number generator
-	rand.Seed(time.Now().UTC().UnixNano())
-
+func (m *Command) StartNoServe(addr dax.Address) (err error) {
 	// setupServer
 	err = m.setupServer()
 	if err != nil {
@@ -288,41 +281,46 @@ func (m *Command) StartNoServe() (err error) {
 		return errors.Wrap(err, "opening server")
 	}
 
+	// Start the "check-in" background process which periodically checks in with
+	// MDS.
+	go m.checkIn(addr)
+
 	return nil
 }
 
-// Register registers the node with the MDS service using whatever MDS
-// implementation was injected during setup.
-func (m *Command) Register() (err error) {
-	if m.mds == nil {
-		return errors.New("no MDS implementation with which to register")
+// checkIn calls the CheckIn function set on m.checkInFn every interval period.
+// If the interval period is 0, the check-in is disabled.
+func (m *Command) checkIn(addr dax.Address) {
+	interval := m.Config.CheckInInterval
+
+	if interval == 0 {
+		return
 	}
 
-	node := &dax.Node{
-		Address: dax.Address(m.Config.Advertise),
-		RoleTypes: []dax.RoleType{
-			dax.RoleTypeCompute,
-			dax.RoleTypeTranslate,
-		},
-	}
-	return m.mds.RegisterNode(context.Background(), node)
-}
+	for {
+		select {
+		case <-m.done:
+			return
+		case <-time.After(interval):
+			m.logger.Debugf("node check-in in last %s, address: %s", interval, m.Config.Advertise)
 
-// CheckIn is called periodically to check in with the MDS service using
-// whatever MDS implementation was injected during setup.
-func (m *Command) CheckIn() (err error) {
-	if m.mds == nil {
-		return errors.New("no MDS implementation with which to check-in")
-	}
+			if m.Registrar == nil {
+				m.logger.Printf("no MDS implementation with which to check-in on node: %s", m.Config.Advertise)
+			}
 
-	node := &dax.Node{
-		Address: dax.Address(m.Config.Advertise),
-		RoleTypes: []dax.RoleType{
-			dax.RoleTypeCompute,
-			dax.RoleTypeTranslate,
-		},
+			node := &dax.Node{
+				Address: addr,
+				RoleTypes: []dax.RoleType{
+					dax.RoleTypeCompute,
+					dax.RoleTypeTranslate,
+				},
+			}
+
+			if err := m.Registrar.CheckInNode(context.Background(), node); err != nil {
+				m.logger.Errorf("checking in node: %s, %v", node.Address, err)
+			}
+		}
 	}
-	return m.mds.CheckInNode(context.Background(), node)
 }
 
 // Start starts the pilosa server - it returns once the server is running.
@@ -569,9 +567,6 @@ func (m *Command) setupServer() error {
 		snap = alpha.NewAlphaSnapshot(m.snapshotter)
 	}
 
-	m.Config.Etcd.Id = m.Config.Name // TODO(twg) rethink this
-	e := petcd.NewEtcd(m.Config.Etcd, m.logger, m.Config.Cluster.ReplicaN, version)
-
 	executionPlannerFn := func(e pilosa.Executor, api *pilosa.API, sql string) sql3.CompilePlanner {
 		fapi := &pilosa.FeatureBaseSchemaAPI{API: api}
 		fsapi := &pilosa.FeatureBaseSystemAPI{API: api}
@@ -605,7 +600,6 @@ func (m *Command) setupServer() error {
 		pilosa.OptServerMaxQueryMemory(m.Config.MaxQueryMemory),
 		pilosa.OptServerQueryHistoryLength(m.Config.QueryHistoryLength),
 		pilosa.OptServerPartitionAssigner(m.Config.Cluster.PartitionToNodeAssignment),
-		pilosa.OptServerDisCo(e, e, e, e),
 		pilosa.OptServerExecutionPlannerFn(executionPlannerFn),
 		pilosa.OptServerWriteLogReader(wlr),
 		pilosa.OptServerWriteLogWriter(wlw),
@@ -613,6 +607,25 @@ func (m *Command) setupServer() error {
 		pilosa.OptServerIsDataframeEnabled(m.Config.Dataframe.Enable),
 	}
 
+	if m.isComputeNode {
+		nodeID := "localcmd"
+		serverOptions = append(serverOptions,
+			pilosa.OptServerDisCo(
+				disco.NewInMemDisCo(nodeID),
+				disco.NewLocalNoder([]*disco.Node{
+					{ID: nodeID, URI: *advertiseURI, IsPrimary: true, State: disco.NodeStateStarted},
+				}),
+				disco.InMemSharder,
+				disco.InMemSchemator,
+			),
+			pilosa.OptServerNodeID(nodeID),
+		)
+	} else {
+		m.Config.Etcd.Id = m.Config.Name // TODO(twg) rethink this
+		e := petcd.NewEtcd(m.Config.Etcd, m.logger, m.Config.Cluster.ReplicaN, version)
+		serverOptions = append(serverOptions, pilosa.OptServerDisCo(e, e, e, e))
+	}
+
 	if m.Config.LookupDBDSN != "" {
 		serverOptions = append(serverOptions, pilosa.OptServerLookupDB(m.Config.LookupDBDSN))
 	}

sql3/test/defs/defs_bool.go

@@ -9,7 +9,6 @@ var boolTests = TableTest{
 			srcHdr("_id", fldTypeID),
 			srcHdr("a_bool", fldTypeBool),
 		),
-		srcRows(),
 	),
 	SQLTests: []SQLTest{
 		{

sql3/test/defs/defs_create_table.go

@@ -55,7 +55,6 @@ var alterTable = TableTest{
 			srcHdr("_id", fldTypeID),
 			srcHdr("a_int", fldTypeInt),
 		),
-		srcRows(),
 	),
 	SQLTests: []SQLTest{
 		{

sql3/test/defs/defs_inserts.go

@@ -13,7 +13,6 @@ var insertTest = TableTest{
 			srcHdr("event", fldTypeStringSet),
 			srcHdr("ievent", fldTypeIDSet),
 		),
-		nil,
 	),
 	SQLTests: []SQLTest{
 		{

sql3/test/defs/defs_keyed.go

@@ -1,5 +1,7 @@
 package defs
 
+var Keyed TableTest = keyed
+
 var keyed = TableTest{
 	Table: tbl(
 		"keyed",
@@ -17,6 +19,10 @@ var keyed = TableTest{
 			srcRow("three", int64(33), []int64{11, 32, 33}, int64(301), "str3", []string{"a3", "b3", "c3"}),
 			srcRow("four", int64(44), []int64{41, 42, 43}, int64(401), "str4", []string{"a4", "b4", "c4"}),
 		),
+		srcRows(
+			srcRow("five", int64(55), []int64{51, 52, 53}, int64(501), "str5", []string{"a5", "b5", "c5"}),
+			srcRow("six", int64(66), []int64{61, 62, 63}, int64(601), "str6", []string{"a6", "b6", "c6"}),
+		),
 	),
 	SQLTests: []SQLTest{
 		{
@@ -40,6 +46,16 @@ var keyed = TableTest{
 				row("three", int64(33), []int64{11, 32, 33}, int64(301), "str3", []string{"a3", "b3", "c3"}),
 				row("four", int64(44), []int64{41, 42, 43}, int64(401), "str4", []string{"a4", "b4", "c4"}),
 			),
+			ExpRowsPlus1: rowSets(
+				rows(
+					row("one", int64(11), []int64{11, 12, 13}, int64(101), "str1", []string{"a1", "b1", "c1"}),
+					row("two", int64(22), []int64{11, 12, 23}, int64(201), "str2", []string{"a2", "b2", "c2"}),
+					row("three", int64(33), []int64{11, 32, 33}, int64(301), "str3", []string{"a3", "b3", "c3"}),
+					row("four", int64(44), []int64{41, 42, 43}, int64(401), "str4", []string{"a4", "b4", "c4"}),
+					row("five", int64(55), []int64{51, 52, 53}, int64(501), "str5", []string{"a5", "b5", "c5"}),
+					row("six", int64(66), []int64{61, 62, 63}, int64(601), "str6", []string{"a6", "b6", "c6"}),
+				),
+			),
 			Compare:        CompareExactUnordered,
 			SortStringKeys: true,
 		},
@@ -128,19 +144,20 @@ var keyed = TableTest{
 				row(int64(12), int64(2)),
 			),
 		},
-		{
-			name:  "topn",
-			Table: "keyed",
-			PQLs:  []string{"TopN(an_id_set, n=2)"},
-			ExpHdrs: hdrs(
-				hdr("an_id_set", fldTypeID),
-				hdr("count", fldTypeID),
-			),
-			ExpRows: rows(
-				row(int64(11), int64(3)),
-				row(int64(12), int64(2)),
-			),
-		},
+		// TODO(tlt): figure out why this sometimes fails on multi-node setups
+		// {
+		// 	name:  "topn",
+		// 	Table: "keyed",
+		// 	PQLs:  []string{"TopN(an_id_set, n=2)"},
+		// 	ExpHdrs: hdrs(
+		// 		hdr("an_id_set", fldTypeID),
+		// 		hdr("count", fldTypeID),
+		// 	),
+		// 	ExpRows: rows(
+		// 		row(int64(11), int64(3)),
+		// 		row(int64(12), int64(2)),
+		// 	),
+		// },
 		{
 			name:  "rows",
 			Table: "keyed",
@@ -197,7 +214,7 @@ var keyed = TableTest{
 		},
 		{
 			name:  "unionrows",
-			Table: "unkeyed",
+			Table: "keyed",
 			PQLs:  []string{"Count(UnionRows(Rows(field=an_id_set)))"},
 			ExpHdrs: hdrs(
 				hdr("count", fldTypeID),

sql3/test/defs/defs_keyed_insert.go

@@ -14,7 +14,6 @@ var keyedInsertTest = TableTest{
 			srcHdr("event", fldTypeStringSet),
 			srcHdr("ievent", fldTypeIDSet),
 		),
-		nil,
 	),
 	SQLTests: []SQLTest{
 		{

sql3/test/defs/defs_timestamp_literals.go

@@ -13,7 +13,6 @@ var timestampLiterals = TableTest{
 			srcHdr("event", fldTypeStringSet),
 			srcHdr("ievent", fldTypeIDSet),
 		),
-		srcRows(),
 	),
 	SQLTests: []SQLTest{
 		{

sql3/test/defs/types.go

@@ -97,11 +97,14 @@ func (tt TableTest) CreateTable() string {
 	return tt.Table.createTable()
 }
 
-func (tt TableTest) InsertInto(t *testing.T) string {
+func (tt TableTest) InsertInto(t *testing.T, rowSets ...int) string {
 	if !tt.HasTable() {
 		return ""
 	}
-	return tt.Table.insertInto(t)
+	if len(rowSets) == 0 {
+		rowSets = []int{0}
+	}
+	return tt.Table.insertInto(t, rowSets)
 }
 
 type SQLTest struct {
@@ -109,6 +112,7 @@ type SQLTest struct {
 	SQLs           []string
 	ExpHdrs        []*featurebase.WireQueryField
 	ExpRows        [][]interface{}
+	ExpRowsPlus1   [][][]interface{}
 	ExpErr         string
 	Compare        compareMethod
 	SortStringKeys bool
@@ -127,12 +131,13 @@ func (s SQLTest) Name(i int) string {
 }
 
 type PQLTest struct {
-	name    string
-	PQLs    []string
-	Table   string
-	ExpHdrs []*featurebase.WireQueryField
-	ExpRows [][]interface{}
-	ExpErr  string
+	name         string
+	PQLs         []string
+	Table        string
+	ExpHdrs      []*featurebase.WireQueryField
+	ExpRows      [][]interface{}
+	ExpRowsPlus1 [][][]interface{}
+	ExpErr       string
 }
 
 // Name returns a string name which can be used to distingish test runs. It
@@ -153,7 +158,7 @@ type sourceColumn struct {
 	options string
 }
 
-func tbl(name string, columns []sourceColumn, rows []sourceRow) source {
+func tbl(name string, columns []sourceColumn, rows ...[]sourceRow) source {
 	return source{
 		name:    name,
 		columns: columns,
@@ -242,7 +247,7 @@ func (sr sourceRows) insertTuples(t *testing.T) string {
 type source struct {
 	name    string
 	columns []sourceColumn
-	rows    []sourceRow
+	rows    [][]sourceRow
 }
 
 func (s source) createTable() string {
@@ -264,9 +269,11 @@ func (s source) createTable() string {
 	return ct
 }
 
-func (s source) insertInto(t *testing.T) string {
+func (s source) insertInto(t *testing.T, rowSets []int) string {
 	ii := "INSERT INTO " + s.name + " VALUES "
-	ii += sourceRows(s.rows).insertTuples(t)
+	for _, rowSet := range rowSets {
+		ii += sourceRows(s.rows[rowSet]).insertTuples(t)
+	}
 	return ii
 }
 
@@ -291,6 +298,10 @@ func rows(rows ...[]interface{}) [][]interface{} {
 	return rows
 }
 
+func rowSets(rowSets ...[][]interface{}) [][][]interface{} {
+	return rowSets
+}
+
 func row(cells ...interface{}) []interface{} {
 	return cells
 }