mirror of
https://github.com/featurebasedb/featurebase.git
synced 2026-08-28 10:54:59 +00:00
2843f218bc
* Introduce ServiceManager and Refactor DAX Integration tests The ServiceManager provides an interface with which to manage featurebase (dax) services (mds, queryer, computer). It replaces the confusing interface implementations in /dax/server/server.go (which optionally used pointers to in-process objects to satisfy an interface) with (for now) http implementations. The thought is that even if we're running all services in-process, we should communicate between services over http in order to mirror what we would do in a production environment where the services are running on different nodes. This batch of commits does quit a lot, most of which is captured here: - Added `path` support to `dax.Address`. Address is now a string of the form [scheme]://[host]:[port]/[path]. - Added `Holder.directiveApplied` to determine (in tests) if the computer has completed applying the latest directive. This is somewhat temporary until we improve the mds-to-computer logic. - Removed the "service prefix" code which was prepending client URL paths with the prefix. Instead, the serviceType (mds, queryer, computer[n] is now part of `dax.Address`). - Removed, from the dax config, the top level `StorageMethod` and `StorageDSN` and now just have `MDS.Config.DataDir`. - Added `Computer.Config.N` to specify the number of computers to run in-process. - Moved the `pilosa.MDS` interface to `computer.Registrar`. This is an example of getting the interfaces defined in the right packages. - Added `SnapshotTable()` method to the mds client (to align with its API). - Changed `Balancer.AddJob()` to `Balancer.AddJobs()` to support, for example, adding 256 partitions in a single call. Refactored some of the naive Balancer to account for this. - Added a `Seed` to the top-level config. It's not really useful because of package `crypto/rand`. - Added an in-memory implementation of the DisCo interface and disabled etcd in a computer service. - Create sepearte data-dirs for each in-process computer. - Disabled grpc in dax. - Modified the sql3 test definition format to support multiple insert steps and separate query results (to align with those steps). * Changes necessary to get multiple computer instance running in-process For now the config looks like this: ``` [computer] run = true n = 4 ``` but we can probably just change that to be something like: ``` [computer] run = 4 ``` *Issues found running multiple "computers" in-process* - grpc was trying to bind on the same port - changed GRPCListener from `*net.TCPListener` to `net.Listener` - created a nopListener and set to that for now (i.e. disabled grpc) - etcd was starting more than once - changed dax to use in-memory implementations of the disco interfaces (i.e. stop using etcd) - IDAllocator (which uses boltdb) was trying to open the `idalloc.db` file more than once - realized we have to set separate data-dirs for each holder. that fixed it. * Port dax integration tests to ManagedCommand * Modify Balancer-related methods like AddJob to AddJobs There were (and still are) a lot of places where we were adding on job at a time, even when we had a long list of jobs to add. This resulted in every job add (for example adding 1 of 256 shards) taking ~40ms, or over 10s to create a keyed table. One reason was because each job add was making multiple boltdb transactions. * Port over more dax integration test stuff * Add DirectiveApplied to signify that snapshot/writes have loaded. We use this in tests to avoid using sleeps. This should be considered temporary; we're going to need a more robust solution for determining when a computer node is ready to serve complete data. * Finish porting dax integration tests * Improve godocs * Remove docker-based DAX integration tests. * go mod tidy * Move test/managed.go to avoid package conflicts * Modify IDK integration tests to work with ServiceManager changes This is really just computer -> computer0 And the MDS DataDir config change. * cleanup found during review * echo $CI_COMMIT_REF_SLUG in CI * remove docker image arg, use build instead
194 lines
8.7 KiB
Modula-2
194 lines
8.7 KiB
Modula-2
module github.com/molecula/featurebase/v3
|
|
|
|
replace github.com/go-avro/avro => github.com/pilosa/avro v0.0.0-20200626214113-bc1bf9fd41c1
|
|
|
|
replace github.com/gomem/gomem => github.com/tgruben/gomem v0.0.0-20221021111114-79fdc77dcf61
|
|
|
|
replace robpike.io/ivy => github.com/tgruben/ivy v0.0.0-20221107170120-634b546dcdac
|
|
|
|
require (
|
|
github.com/CAFxX/gcnotifier v0.0.0-20220409005548-0153238b886a
|
|
github.com/DataDog/datadog-go v4.8.3+incompatible
|
|
github.com/HdrHistogram/hdrhistogram-go v1.1.2 // indirect
|
|
github.com/Microsoft/go-winio v0.5.2 // indirect
|
|
github.com/alexbrainman/odbc v0.0.0-20211220213544-9c9a2e61c5e2
|
|
github.com/aws/aws-sdk-go v1.42.39
|
|
github.com/beevik/ntp v0.3.0
|
|
github.com/benbjohnson/immutable v0.4.0
|
|
github.com/cespare/xxhash v1.1.0
|
|
github.com/chzyer/readline v1.5.1
|
|
github.com/confluentinc/confluent-kafka-go v1.9.1
|
|
github.com/davecgh/go-spew v1.1.1
|
|
github.com/denisenkom/go-mssqldb v0.11.0
|
|
github.com/felixge/fgprof v0.9.2
|
|
github.com/getsentry/sentry-go v0.13.0
|
|
github.com/glycerine/vprint v0.0.0-20200730000117-76cea49a68ea
|
|
github.com/go-avro/avro v0.0.0-20171219232920-444163702c11
|
|
github.com/go-openapi/strfmt v0.21.2 // indirect
|
|
github.com/go-sql-driver/mysql v1.6.0
|
|
github.com/go-test/deep v1.0.7
|
|
github.com/gogo/protobuf v1.3.2
|
|
github.com/golang-jwt/jwt v3.2.2+incompatible
|
|
github.com/golang/protobuf v1.5.2
|
|
github.com/google/go-cmp v0.5.8
|
|
github.com/gopherjs/gopherjs v0.0.0-20200217142428-fce0ec30dd00 // indirect
|
|
github.com/gorilla/handlers v1.3.0
|
|
github.com/gorilla/mux v1.8.0
|
|
github.com/gorilla/securecookie v1.1.1
|
|
github.com/hashicorp/go-retryablehttp v0.7.1
|
|
github.com/improbable-eng/grpc-web v0.15.0
|
|
github.com/jedib0t/go-pretty v4.3.0+incompatible
|
|
github.com/lib/pq v1.10.5
|
|
github.com/molecula/apophenia v0.0.0-20190827192002-68b7a14a478b
|
|
github.com/opentracing/opentracing-go v1.2.0
|
|
github.com/pelletier/go-toml v1.9.5
|
|
github.com/pkg/errors v0.9.1
|
|
github.com/prometheus/client_golang v1.12.2
|
|
github.com/prometheus/client_model v0.2.0
|
|
github.com/prometheus/prom2json v1.3.1
|
|
github.com/rakyll/statik v0.1.7
|
|
github.com/remyoudompheng/bigfft v0.0.0-20200410134404-eec4a21b6bb0 // indirect
|
|
github.com/ricochet2200/go-disk-usage/du v0.0.0-20210707232629-ac9918953285
|
|
github.com/rs/cors v1.8.2 // indirect
|
|
github.com/satori/go.uuid v1.2.1-0.20180404165556-75cca531ea76
|
|
github.com/segmentio/kafka-go v0.4.29
|
|
github.com/shirou/gopsutil/v3 v3.22.5
|
|
github.com/spf13/cobra v1.2.1
|
|
github.com/spf13/pflag v1.0.5
|
|
github.com/spf13/viper v1.8.1
|
|
github.com/stretchr/testify v1.8.0
|
|
github.com/uber/jaeger-client-go v2.25.0+incompatible
|
|
github.com/uber/jaeger-lib v2.4.1+incompatible // indirect
|
|
github.com/zeebo/blake3 v0.2.3
|
|
go.etcd.io/bbolt v1.3.6
|
|
go.etcd.io/etcd v3.3.27+incompatible
|
|
go.etcd.io/etcd/api/v3 v3.5.5
|
|
go.etcd.io/etcd/client/pkg/v3 v3.5.5
|
|
go.etcd.io/etcd/client/v3 v3.5.5
|
|
go.etcd.io/etcd/server/v3 v3.5.5
|
|
golang.org/x/exp v0.0.0-20220827204233-334a2380cb91
|
|
golang.org/x/mod v0.7.0
|
|
golang.org/x/oauth2 v0.0.0-20220608161450-d0670ef3b1eb
|
|
golang.org/x/sync v0.1.0
|
|
golang.org/x/time v0.0.0-20211116232009-f0f3c7e86c11
|
|
gopkg.in/DataDog/dd-trace-go.v1 v1.38.1
|
|
gopkg.in/yaml.v2 v2.4.0
|
|
modernc.org/mathutil v1.5.0
|
|
modernc.org/strutil v1.1.3
|
|
sigs.k8s.io/yaml v1.2.0
|
|
vitess.io/vitess v3.0.0-rc.3.0.20190602171040-12bfde34629c+incompatible
|
|
)
|
|
|
|
require (
|
|
github.com/PaesslerAG/gval v1.0.0
|
|
github.com/PaesslerAG/jsonpath v0.1.1
|
|
github.com/apache/arrow/go/v10 v10.0.0-20221021053532-2f627c213fc3
|
|
github.com/gomem/gomem v0.1.0
|
|
github.com/google/uuid v1.3.0
|
|
github.com/jaffee/commandeer v0.6.0
|
|
github.com/linkedin/goavro/v2 v2.11.1
|
|
google.golang.org/grpc v1.49.0
|
|
google.golang.org/protobuf v1.28.1
|
|
robpike.io/ivy v0.2.9
|
|
)
|
|
|
|
require (
|
|
github.com/DataDog/datadog-go/v5 v5.1.0 // indirect
|
|
github.com/DataDog/gostackparse v0.5.0 // indirect
|
|
github.com/JohnCGriffin/overflow v0.0.0-20211019200055-46fa312c352c // indirect
|
|
github.com/andybalholm/brotli v1.0.4 // indirect
|
|
github.com/apache/thrift v0.16.0 // indirect
|
|
github.com/asaskevich/govalidator v0.0.0-20200907205600-7a23bdc65eef // indirect
|
|
github.com/beorn7/perks v1.0.1 // indirect
|
|
github.com/cenkalti/backoff/v4 v4.1.3 // indirect
|
|
github.com/cespare/xxhash/v2 v2.1.2 // indirect
|
|
github.com/coreos/go-semver v0.3.0 // indirect
|
|
github.com/coreos/go-systemd/v22 v22.3.2 // indirect
|
|
github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f // indirect
|
|
github.com/dustin/go-humanize v1.0.0 // indirect
|
|
github.com/form3tech-oss/jwt-go v3.2.3+incompatible // indirect
|
|
github.com/fsnotify/fsnotify v1.4.9 // indirect
|
|
github.com/go-ole/go-ole v1.2.6 // indirect
|
|
github.com/go-openapi/errors v0.19.8 // indirect
|
|
github.com/go-stack/stack v1.8.0 // indirect
|
|
github.com/gobwas/httphead v0.0.0-20200921212729-da3d93bc3c58 // indirect
|
|
github.com/gobwas/pool v0.2.1 // indirect
|
|
github.com/gobwas/ws v1.0.4 // indirect
|
|
github.com/goccy/go-json v0.9.11 // indirect
|
|
github.com/golang-sql/civil v0.0.0-20190719163853-cb61b32ac6fe // indirect
|
|
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b // indirect
|
|
github.com/golang/snappy v0.0.4 // indirect
|
|
github.com/google/btree v1.0.1 // indirect
|
|
github.com/google/flatbuffers v2.0.8+incompatible // indirect
|
|
github.com/google/pprof v0.0.0-20211214055906-6f57359322fd // indirect
|
|
github.com/gorilla/websocket v1.5.0 // indirect
|
|
github.com/grpc-ecosystem/go-grpc-middleware v1.3.0 // indirect
|
|
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 // indirect
|
|
github.com/grpc-ecosystem/grpc-gateway v1.16.0 // indirect
|
|
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
|
|
github.com/hashicorp/hcl v1.0.0 // indirect
|
|
github.com/inconshreveable/mousetrap v1.0.0 // indirect
|
|
github.com/jmespath/go-jmespath v0.4.0 // indirect
|
|
github.com/jonboulle/clockwork v0.2.2 // indirect
|
|
github.com/json-iterator/go v1.1.12 // indirect
|
|
github.com/klauspost/asmfmt v1.3.2 // indirect
|
|
github.com/klauspost/compress v1.15.9 // indirect
|
|
github.com/klauspost/cpuid/v2 v2.0.12 // indirect
|
|
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
|
|
github.com/magiconair/properties v1.8.5 // indirect
|
|
github.com/mattetti/filebuffer v1.0.1 // indirect
|
|
github.com/mattn/go-runewidth v0.0.2 // indirect
|
|
github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
|
|
github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8 // indirect
|
|
github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3 // indirect
|
|
github.com/mitchellh/mapstructure v1.5.0 // indirect
|
|
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
|
|
github.com/modern-go/reflect2 v1.0.2 // indirect
|
|
github.com/oklog/ulid v1.3.1 // indirect
|
|
github.com/pierrec/lz4/v4 v4.1.15 // indirect
|
|
github.com/pmezard/go-difflib v1.0.0 // indirect
|
|
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
|
|
github.com/prometheus/common v0.33.0 // indirect
|
|
github.com/prometheus/procfs v0.7.3 // indirect
|
|
github.com/sirupsen/logrus v1.7.0 // indirect
|
|
github.com/soheilhy/cmux v0.1.5 // indirect
|
|
github.com/spf13/afero v1.6.0 // indirect
|
|
github.com/spf13/cast v1.3.1 // indirect
|
|
github.com/spf13/jwalterweatherman v1.1.0 // indirect
|
|
github.com/stretchr/objx v0.4.0 // indirect
|
|
github.com/subosito/gotenv v1.2.0 // indirect
|
|
github.com/tklauser/go-sysconf v0.3.10 // indirect
|
|
github.com/tklauser/numcpus v0.4.0 // indirect
|
|
github.com/tmc/grpc-websocket-proxy v0.0.0-20201229170055-e5319fda7802 // indirect
|
|
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2 // indirect
|
|
github.com/yusufpapurcu/wmi v1.2.2 // indirect
|
|
github.com/zeebo/xxh3 v1.0.2 // indirect
|
|
go.etcd.io/etcd/client/v2 v2.305.5 // indirect
|
|
go.etcd.io/etcd/pkg/v3 v3.5.5 // indirect
|
|
go.etcd.io/etcd/raft/v3 v3.5.5 // indirect
|
|
go.mongodb.org/mongo-driver v1.7.5 // indirect
|
|
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.25.0 // indirect
|
|
go.opentelemetry.io/otel v1.0.1 // indirect
|
|
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.0.1 // indirect
|
|
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.0.1 // indirect
|
|
go.opentelemetry.io/otel/sdk v1.0.1 // indirect
|
|
go.opentelemetry.io/otel/trace v1.0.1 // indirect
|
|
go.opentelemetry.io/proto/otlp v0.9.0 // indirect
|
|
go.uber.org/atomic v1.7.0 // indirect
|
|
go.uber.org/multierr v1.6.0 // indirect
|
|
go.uber.org/zap v1.17.0 // indirect
|
|
golang.org/x/crypto v0.1.0 // indirect
|
|
golang.org/x/net v0.2.0 // indirect
|
|
golang.org/x/sys v0.2.0 // indirect
|
|
golang.org/x/text v0.4.0 // indirect
|
|
golang.org/x/tools v0.3.0 // indirect
|
|
golang.org/x/xerrors v0.0.0-20220609144429-65e65417b02f // indirect
|
|
google.golang.org/appengine v1.6.7 // indirect
|
|
google.golang.org/genproto v0.0.0-20220503193339-ba3ae3f07e29 // indirect
|
|
gopkg.in/ini.v1 v1.62.0 // indirect
|
|
gopkg.in/natefinch/lumberjack.v2 v2.0.0 // indirect
|
|
gopkg.in/yaml.v3 v3.0.1 // indirect
|
|
nhooyr.io/websocket v1.8.6 // indirect
|
|
)
|
|
|
|
go 1.18
|