scriptoriumadmin/featurebase
1
0
Fork 0
mirror of https://github.com/featurebasedb/featurebase.git synced 2026-08-28 10:54:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
featurebase/idk/docker-compose.yml
Travis Turner a44b622aa0 Introduce ServiceManager and Refactor DAX Integration tests (#2320) 
* Introduce ServiceManager and Refactor DAX Integration tests

The ServiceManager provides an interface with which to manage
featurebase (dax) services (mds, queryer, computer). It replaces the
confusing interface implementations in /dax/server/server.go (which
optionally used pointers to in-process objects to satisfy an interface)
with (for now) http implementations. The thought is that even if we're
running all services in-process, we should communicate between services
over http in order to mirror what we would do in a production
environment where the services are running on different nodes.

This batch of commits does quit a lot, most of which is captured here:

- Added `path` support to `dax.Address`. Address is now a string of the form [scheme]://[host]:[port]/[path].
- Added `Holder.directiveApplied` to determine (in tests) if the computer has completed applying the latest directive. This is somewhat temporary until we improve the mds-to-computer logic.
- Removed the "service prefix" code which was prepending client URL paths with the prefix. Instead, the serviceType (mds, queryer, computer[n] is now part of `dax.Address`).
- Removed, from the dax config, the top level `StorageMethod` and `StorageDSN` and now just have `MDS.Config.DataDir`.
- Added `Computer.Config.N` to specify the number of computers to run in-process.
- Moved the `pilosa.MDS` interface to `computer.Registrar`. This is an example of getting the interfaces defined in the right packages.
- Added `SnapshotTable()` method to the mds client (to align with its API).
- Changed `Balancer.AddJob()` to `Balancer.AddJobs()` to support, for example, adding 256 partitions in a single call. Refactored some of the naive Balancer to account for this.
- Added a `Seed` to the top-level config. It's not really useful because of package `crypto/rand`.
- Added an in-memory implementation of the DisCo interface and disabled etcd in a computer service.
- Create sepearte data-dirs for each in-process computer.
- Disabled grpc in dax.
- Modified the sql3 test definition format to support multiple insert steps and separate query results (to align with those steps).

* Changes necessary to get multiple computer instance running in-process

For now the config looks like this:

```
[computer]
run = true
n = 4
```

but we can probably just change that to be something like:

```
[computer]
run = 4
```

*Issues found running multiple "computers" in-process*
- grpc was trying to bind on the same port
  - changed GRPCListener from `*net.TCPListener` to `net.Listener`
  - created a nopListener and set to that for now (i.e. disabled grpc)
- etcd was starting more than once
  - changed dax to use in-memory implementations of the disco interfaces (i.e. stop using etcd)
- IDAllocator (which uses boltdb) was trying to open the `idalloc.db` file more than once
  - realized we have to set separate data-dirs for each holder. that fixed it.

* Port dax integration tests to ManagedCommand

* Modify Balancer-related methods like AddJob to AddJobs

There were (and still are) a lot of places where we were adding on job
at a time, even when we had a long list of jobs to add. This resulted in
every job add (for example adding 1 of 256 shards) taking ~40ms, or over
10s to create a keyed table. One reason was because each job add was
making multiple boltdb transactions.

* Port over more dax integration test stuff

* Add DirectiveApplied to signify that snapshot/writes have loaded.

We use this in tests to avoid using sleeps.
This should be considered temporary; we're going to need a more robust
solution for determining when a computer node is ready to serve complete
data.

* Finish porting dax integration tests

* Improve godocs

* Remove docker-based DAX integration tests.

* go mod tidy

* Move test/managed.go to avoid package conflicts

* Modify IDK integration tests to work with ServiceManager changes

This is really just computer -> computer0
And the MDS DataDir config change.

* cleanup found during review

* echo $CI_COMMIT_REF_SLUG in CI

* remove docker image arg, use build instead

(cherry picked from commit 2843f218bc)
2022-12-12 09:01:20 -08:00

153 lines
5 KiB
YAML
Raw Blame History

version: '3'
services:
postgres:
image: postgres
environment:
POSTGRES_DB: postgres
POSTGRES_USER: postgres
POSTGRES_PASSWORD: password
zookeeper:
image: confluentinc/cp-zookeeper:${CONFLUENT_VERSION:-5.5.3}
environment:
ZOOKEEPER_CLIENT_PORT: 2181
KAFKA_OPTS: "-Djava.security.auth.login.config=/etc/kafka/secrets/jaas_kafka_broker.conf"
volumes:
- ./docker-sasl/ssl_keys:/etc/kafka/secrets
kafka:
image: confluentinc/cp-kafka:${CONFLUENT_VERSION:-5.5.3}
environment:
KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:19092,LISTENER_DOCKER_EXTERNAL://kafka:9092,LISTENER_DOCKER_EXTERNAL_SSL://kafka:9094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL_SSL:SASL_SSL
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
KAFKA_BROKER_ID: 1
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_OPTS: "-Djava.security.auth.login.config=/etc/kafka/secrets/jaas_kafka_broker.conf"
KAFKA_SASL_ENABLED_MECHANISMS: PLAIN
KAFKA_SSL_KEYSTORE_FILENAME: broker_kafkaBroker_server.keystore.jks
KAFKA_SSL_KEYSTORE_CREDENTIALS: credentials
KAFKA_SSL_KEY_CREDENTIALS: credentials
KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: PLAIN
volumes:
- ./docker-sasl/ssl_keys:/etc/kafka/secrets
depends_on:
- zookeeper
# UI can be nice for debugging stuff and messing with Kafka. Not used by tests.
kafka-ui:
container_name: kafka-ui
image: provectuslabs/kafka-ui:latest
ports:
- 8080:8080
depends_on:
- zookeeper
- kafka
- schema-registry
environment:
KAFKA_CLUSTERS_0_NAME: local
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092
KAFKA_CLUSTERS_0_ZOOKEEPER: zookeeper:2181
KAFKA_CLUSTERS_0_JMXPORT: 9997
KAFKA_CLUSTERS_0_SCHEMAREGISTRY: http://schema-registry:8081
schema-registry:
image: confluentinc/cp-schema-registry:${CONFLUENT_VERSION:-5.5.3}
environment:
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:19092
SCHEMA_REGISTRY_HOST_NAME: schema-registry
SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081
depends_on:
- zookeeper
- kafka
pilosa:
#image: registry.gitlab.com/molecula/featurebase/featurebase:linux-amd64-${BRANCH_NAME} # (jaffee) I think we should build directly from this tree rather than pulling the image... makes the local dev flow easier
build: ..
environment:
PILOSA_DATA_DIR: /data
PILOSA_BIND: 0.0.0.0:10101
PILOSA_BIND_GRPC: 0.0.0.0:20101
PILOSA_ADVERTISE: pilosa:10101
PILOSA_LOOKUP_DB_DSN: "postgresql://postgres:password@postgres:5432/postgres?sslmode=disable"
depends_on:
- postgres
volumes:
- ./testenv/certs:/certs
pilosa-tls:
#image: registry.gitlab.com/molecula/featurebase/featurebase:linux-amd64-${BRANCH_NAME}
build: ..
environment:
PILOSA_DATA_DIR: /data
PILOSA_BIND: https://0.0.0.0:10111
PILOSA_BIND_GRPC: 0.0.0.0:20111
PILOSA_ADVERTISE: https://pilosa-tls:10111
PILOSA_TLS_CA_CERTIFICATE: /certs/ca.crt
PILOSA_TLS_CERTIFICATE: /certs/pilosa-tls.crt
PILOSA_TLS_KEY: /certs/pilosa-tls.key
PILOSA_TLS_ENABLE_CLIENT_VERIFICATION: 1
volumes:
- ./testenv/certs:/certs
pilosa-auth:
#image: registry.gitlab.com/molecula/featurebase/featurebase:linux-amd64-${BRANCH_NAME}
build: ..
environment:
PILOSA_DATA_DIR: /data
PILOSA_BIND: 0.0.0.0:10105
PILOSA_BIND_GRPC: 0.0.0.0:20105
PILOSA_ADVERTISE: pilosa-auth:10105
PILOSA_CONFIG: /testdata/featurebase.conf
volumes:
- ./testdata:/testdata
depends_on:
- fakeidp
fakeidp:
build:
context: .
dockerfile: Dockerfile-fakeIDP
idk-test:
build:
context: ../.
dockerfile: ./idk/Dockerfile-test
environment:
IDK_DEFAULT_SHARD_TRANSACTIONAL: ${IDK_DEFAULT_SHARD_TRANSACTIONAL}
volumes:
- ./testenv/certs:/certs
- ./docker-sasl/ssl_keys:/ssl_keys
- ./testdata:/testdata
depends_on:
#- kafka
#- postgres
- fakeidp
wait:
build:
context: .
dockerfile: Dockerfile-wait
volumes:
- ./testenv/certs:/certs
dax:
build:
context: ..
dockerfile: Dockerfile-dax
environment:
FEATUREBASE_BIND: 0.0.0.0:8080
FEATUREBASE_VERBOSE: "true"
FEATUREBASE_QUERYER_RUN: "true"
FEATUREBASE_MDS_RUN: "true"
FEATUREBASE_MDS_CONFIG_DATA_DIR: /dax-data/mds
FEATUREBASE_WRITELOGGER_RUN: "true"
FEATUREBASE_WRITELOGGER_CONFIG_DATA_DIR: /dax-data/wl
FEATUREBASE_SNAPSHOTTER_RUN: "true"
FEATUREBASE_SNAPSHOTTER_CONFIG_DATA_DIR: /dax-data/snaps
FEATUREBASE_COMPUTER_RUN: "true"
FEATUREBASE_COMPUTER_CONFIG_DATA_DIR: /dax-data/computer
Reference in a new issue View git blame Copy permalink