featurebase/client
Travis Turner a44b622aa0 Introduce ServiceManager and Refactor DAX Integration tests (#2320)
* Introduce ServiceManager and Refactor DAX Integration tests

The ServiceManager provides an interface with which to manage
featurebase (dax) services (mds, queryer, computer). It replaces the
confusing interface implementations in /dax/server/server.go (which
optionally used pointers to in-process objects to satisfy an interface)
with (for now) http implementations. The thought is that even if we're
running all services in-process, we should communicate between services
over http in order to mirror what we would do in a production
environment where the services are running on different nodes.

This batch of commits does quit a lot, most of which is captured here:

- Added `path` support to `dax.Address`. Address is now a string of the form [scheme]://[host]:[port]/[path].
- Added `Holder.directiveApplied` to determine (in tests) if the computer has completed applying the latest directive. This is somewhat temporary until we improve the mds-to-computer logic.
- Removed the "service prefix" code which was prepending client URL paths with the prefix. Instead, the serviceType (mds, queryer, computer[n] is now part of `dax.Address`).
- Removed, from the dax config, the top level `StorageMethod` and `StorageDSN` and now just have `MDS.Config.DataDir`.
- Added `Computer.Config.N` to specify the number of computers to run in-process.
- Moved the `pilosa.MDS` interface to `computer.Registrar`. This is an example of getting the interfaces defined in the right packages.
- Added `SnapshotTable()` method to the mds client (to align with its API).
- Changed `Balancer.AddJob()` to `Balancer.AddJobs()` to support, for example, adding 256 partitions in a single call. Refactored some of the naive Balancer to account for this.
- Added a `Seed` to the top-level config. It's not really useful because of package `crypto/rand`.
- Added an in-memory implementation of the DisCo interface and disabled etcd in a computer service.
- Create sepearte data-dirs for each in-process computer.
- Disabled grpc in dax.
- Modified the sql3 test definition format to support multiple insert steps and separate query results (to align with those steps).

* Changes necessary to get multiple computer instance running in-process

For now the config looks like this:

```
[computer]
run = true
n = 4
```

but we can probably just change that to be something like:

```
[computer]
run = 4
```

*Issues found running multiple "computers" in-process*
- grpc was trying to bind on the same port
  - changed GRPCListener from `*net.TCPListener` to `net.Listener`
  - created a nopListener and set to that for now (i.e. disabled grpc)
- etcd was starting more than once
  - changed dax to use in-memory implementations of the disco interfaces (i.e. stop using etcd)
- IDAllocator (which uses boltdb) was trying to open the `idalloc.db` file more than once
  - realized we have to set separate data-dirs for each holder. that fixed it.

* Port dax integration tests to ManagedCommand

* Modify Balancer-related methods like AddJob to AddJobs

There were (and still are) a lot of places where we were adding on job
at a time, even when we had a long list of jobs to add. This resulted in
every job add (for example adding 1 of 256 shards) taking ~40ms, or over
10s to create a keyed table. One reason was because each job add was
making multiple boltdb transactions.

* Port over more dax integration test stuff

* Add DirectiveApplied to signify that snapshot/writes have loaded.

We use this in tests to avoid using sleeps.
This should be considered temporary; we're going to need a more robust
solution for determining when a computer node is ready to serve complete
data.

* Finish porting dax integration tests

* Improve godocs

* Remove docker-based DAX integration tests.

* go mod tidy

* Move test/managed.go to avoid package conflicts

* Modify IDK integration tests to work with ServiceManager changes

This is really just computer -> computer0
And the MDS DataDir config change.

* cleanup found during review

* echo $CI_COMMIT_REF_SLUG in CI

* remove docker image arg, use build instead

(cherry picked from commit 2843f218bc)
2022-12-12 09:01:20 -08:00
..
csv Updated dependency paths to reflect new repo location 2022-09-06 09:39:22 -07:00
docs Updated code to latest version for open-sourcing. 2022-09-02 13:23:39 -07:00
types Add DAX - full list of squashed commits below 2022-12-12 09:01:20 -08:00
api.go Add DAX - full list of squashed commits below 2022-12-12 09:01:20 -08:00
client.go Introduce ServiceManager and Refactor DAX Integration tests (#2320) 2022-12-12 09:01:20 -08:00
client_it_test.go Add DAX - full list of squashed commits below 2022-12-12 09:01:20 -08:00
client_test.go Updated dependency paths to reflect new repo location 2022-09-06 09:39:22 -07:00
cluster.go Updated dependency paths to reflect new repo location 2022-09-06 09:39:22 -07:00
cluster_test.go Updated dependency paths to reflect new repo location 2022-09-06 09:39:22 -07:00
doc.go Updated dependency paths to reflect new repo location 2022-09-06 09:39:22 -07:00
error.go Updated code to latest version for open-sourcing. 2022-09-02 13:23:39 -07:00
importer.go Add DAX - full list of squashed commits below 2022-12-12 09:01:20 -08:00
logimport.go Updated code to latest version for open-sourcing. 2022-09-02 13:23:39 -07:00
logimport_test.go use testhook test cleanup 2022-09-30 11:25:27 -07:00
main_test.go added testify dependency 2022-09-30 11:31:37 -07:00
orm.go Add DAX - full list of squashed commits below 2022-12-12 09:01:20 -08:00
orm_test.go Add DAX - full list of squashed commits below 2022-12-12 09:01:20 -08:00
README.md Updated code to latest version for open-sourcing. 2022-09-02 13:23:39 -07:00
record.go Updated code to latest version for open-sourcing. 2022-09-02 13:23:39 -07:00
record_test.go Updated dependency paths to reflect new repo location 2022-09-06 09:39:22 -07:00
response.go Updated dependency paths to reflect new repo location 2022-09-06 09:39:22 -07:00
response_test.go Updated dependency paths to reflect new repo location 2022-09-06 09:39:22 -07:00
shardnodes.go Updated dependency paths to reflect new repo location 2022-09-06 09:39:22 -07:00
tracer.go Updated code to latest version for open-sourcing. 2022-09-02 13:23:39 -07:00
validate.go Updated code to latest version for open-sourcing. 2022-09-02 13:23:39 -07:00
validate_test.go Updated code to latest version for open-sourcing. 2022-09-02 13:23:39 -07:00
version.go Updated code to latest version for open-sourcing. 2022-09-02 13:23:39 -07:00

Go Client for Pilosa

Go client for Pilosa high performance distributed index.

Usage

If you have the pilosa repo in your GOPATH, you can import the library in your code using:

import "github.com/pilosa/pilosa/v2/client"

Quick overview

Assuming Pilosa server is running at localhost:10101 (the default):

package main

import (
	"fmt"

	"github.com/pilosa/pilosa/v2/client"
)

func main() {
	// Create the default client
	cli := client.DefaultClient()

	// Retrieve the schema
	schema, err := cli.Schema()

	// Create an Index object
	myindex := schema.Index("myindex")

	// Create a Field object
	myfield := myindex.Field("myfield")

	// make sure the index and the field exists on the server
	err := cli.SyncSchema(schema)

	// Send a Set query. If err is non-nil, response will be nil.
	response, err := cli.Query(myfield.Set(5, 42))

	// Send a Row query. If err is non-nil, response will be nil.
	response, err = cli.Query(myfield.Row(5))

	// Get the result
	result := response.Result()
	// Act on the result
	if result != nil {
		columns := result.Row().Columns
		fmt.Println("Got columns: ", columns)
	}

	// You can batch queries to improve throughput
	response, err = cli.Query(myindex.BatchQuery(
		myfield.Row(5),
		myfield.Row(10)))
	if err != nil {
		fmt.Println(err)
	}

	for _, result := range response.Results() {
		// Act on the result
		fmt.Println(result.Row().Columns)
	}
}

Documentation

Data Model and Queries

See: Data Model and Queries

Executing Queries

See: Server Interaction

Other Documentation