mirror of
https://github.com/featurebasedb/featurebase.git
synced 2026-09-05 08:10:50 +00:00
25 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
tgruben
|
e8505d8a53
|
[FB-1831] distribute bulk insert to owning node (#2391)
* distribute bulk insert to owning node |
||
|
Travis Turner
|
2843f218bc
|
Introduce ServiceManager and Refactor DAX Integration tests (#2320)
* Introduce ServiceManager and Refactor DAX Integration tests The ServiceManager provides an interface with which to manage featurebase (dax) services (mds, queryer, computer). It replaces the confusing interface implementations in /dax/server/server.go (which optionally used pointers to in-process objects to satisfy an interface) with (for now) http implementations. The thought is that even if we're running all services in-process, we should communicate between services over http in order to mirror what we would do in a production environment where the services are running on different nodes. This batch of commits does quit a lot, most of which is captured here: - Added `path` support to `dax.Address`. Address is now a string of the form [scheme]://[host]:[port]/[path]. - Added `Holder.directiveApplied` to determine (in tests) if the computer has completed applying the latest directive. This is somewhat temporary until we improve the mds-to-computer logic. - Removed the "service prefix" code which was prepending client URL paths with the prefix. Instead, the serviceType (mds, queryer, computer[n] is now part of `dax.Address`). - Removed, from the dax config, the top level `StorageMethod` and `StorageDSN` and now just have `MDS.Config.DataDir`. - Added `Computer.Config.N` to specify the number of computers to run in-process. - Moved the `pilosa.MDS` interface to `computer.Registrar`. This is an example of getting the interfaces defined in the right packages. - Added `SnapshotTable()` method to the mds client (to align with its API). - Changed `Balancer.AddJob()` to `Balancer.AddJobs()` to support, for example, adding 256 partitions in a single call. Refactored some of the naive Balancer to account for this. - Added a `Seed` to the top-level config. It's not really useful because of package `crypto/rand`. - Added an in-memory implementation of the DisCo interface and disabled etcd in a computer service. - Create sepearte data-dirs for each in-process computer. - Disabled grpc in dax. - Modified the sql3 test definition format to support multiple insert steps and separate query results (to align with those steps). * Changes necessary to get multiple computer instance running in-process For now the config looks like this: ``` [computer] run = true n = 4 ``` but we can probably just change that to be something like: ``` [computer] run = 4 ``` *Issues found running multiple "computers" in-process* - grpc was trying to bind on the same port - changed GRPCListener from `*net.TCPListener` to `net.Listener` - created a nopListener and set to that for now (i.e. disabled grpc) - etcd was starting more than once - changed dax to use in-memory implementations of the disco interfaces (i.e. stop using etcd) - IDAllocator (which uses boltdb) was trying to open the `idalloc.db` file more than once - realized we have to set separate data-dirs for each holder. that fixed it. * Port dax integration tests to ManagedCommand * Modify Balancer-related methods like AddJob to AddJobs There were (and still are) a lot of places where we were adding on job at a time, even when we had a long list of jobs to add. This resulted in every job add (for example adding 1 of 256 shards) taking ~40ms, or over 10s to create a keyed table. One reason was because each job add was making multiple boltdb transactions. * Port over more dax integration test stuff * Add DirectiveApplied to signify that snapshot/writes have loaded. We use this in tests to avoid using sleeps. This should be considered temporary; we're going to need a more robust solution for determining when a computer node is ready to serve complete data. * Finish porting dax integration tests * Improve godocs * Remove docker-based DAX integration tests. * go mod tidy * Move test/managed.go to avoid package conflicts * Modify IDK integration tests to work with ServiceManager changes This is really just computer -> computer0 And the MDS DataDir config change. * cleanup found during review * echo $CI_COMMIT_REF_SLUG in CI * remove docker image arg, use build instead |
||
|
pokeeffe-molecula
|
47d8be26f5
|
implement fb_exec_requests system table (#2327)
implements an fb_exec_requests system table. The purpose of this table is to allow access to internal state to see what queries are running and have been run. Co-authored-by: Travis Turner <travis@molecula.com> |
||
|
tgruben
|
2f1beaf119
|
Dataframe (#2241)
* Dataframe |
||
|
Travis
|
20a8b5713a |
Add DAX - full list of squashed commits below
In this commit, the Directive is mocked; it doesn't actually reach out to a controller. Limits key translation to only those partitions (per index) specified in the Directive. Attempting to create or find a key (or ID) for a partition which is not handled by this node will result in an error; translation requests are no longer forwarded to other nodes. Limits import into only those shards specified, per index, by the Directive. Attempting to import into a shard which is not handled by this node will result in an error. Stub out /directive endpoint The `applyDirective()` method still needs to be implemented. Update mds references to use the new /mds/types structure In mds, we moved the shared types to mds/types. FeatureBase needs to reference those instead. This also bumps the mds version in go.mod. Implement the Add/Remove Index part of Holder.ApplyDirective() This adds functionality to `Holder.ApplyDirective()` which adds or removes indexes (tables) based on those provided in the Directive. Still to be implemented here: shards and partitions. WIP: remove client from Batch Move Batch into its own package: batch Also, in order to avoid import loops, this introduces packages: /batch/types /client/types Reorganize the Importer-related code Moved the Importer interface to package: batch Move the "pilosa client" implementation of the Importer interface to package: client Modify batch.NewBatch to take an Importer (not client) This commit modifies the batch.NewBatch() function to use a functinal option on Batch to inject an Importer into the Batch. Prior to this, NewBatch() took a pointer to a client, which was a little too restrictive. Now, MDS can implement an Importer which uses information from MDS to determine to which node(s) the import calls should be directed. Add client.SetAuthToken() method to satisfy SchemaManager interface Update ApplyDirective logic to include fields. This needs more work, but it was enough to get a basic test passing. Move Transaction type into /types package. Add interface check on batch.Importer no-op implementation Updated ApplyDirective to create all currently support Field types There are still the following TODOs: - [ ] impolement field options (ex: decimal scale, int min/max, etc). - [ ] `time` fields Added support for Decimal.Scale in ApplyDirective Update mds dependency Add /health endpoint Update to use dax (dax/mds) instead of mds. After moving the mds repository into the dax repository as a sub-package, this commit changes everything in FeatureBase to use the dax repo instead of the now abandoned mds repo. Introduce and implment the WriteLogger interfaces. This adds both a `WriteLogReader` and `WriteLogWriter` interface. They are both implemented by the implementation: `fileWriteLogger`. The `fileWriteLogger` uses the dax/writelogger API to append log messages to files on disk. Add WriteLogWriter.ImportRoaring method to interface This commit adds the `ImportRoaring` method to the `WriteLogWriter` interface. Still to implement are the `Import` and `ImportValue` methods. Reorganize the ApplyDirective code The primary goal was to cache the incoming Directive on the Holder prior to applying all of the changes in the directive (i.e. loading data from the WriteLogger) because applying those changes often validated against the accepted state of the node. Implement all of the WriteLogger read/write methods Implement the HTTP WriteLogger implementation WIP: Introduce shard.Version. Implement snapshotter. Add HTTP Snapshotter implementation This also recofigures server to use the HTTPSnapshotter instead of the FileSnapshotter. Implement snapshotter: TableKeys Implement snapshotter: FieldKeys Dependency dance last of the dependency dance Add support for prototype This adds the Makefile targets to build the docker container and push it to ECR. SQL3 changes which break with dax changes Missed TODO: implement FieldVersion version to WriteLogger Address bug causing missing TranslateStores to error Originally, we tried to limit the TranslateStores which get allocated to only those for which the node is responsible. This works when adding a new table. But if a table already exists, there's no logic to start missing TranslateStores. This reverts back to the old FeatureBase logic which brutishly allocates a TranslateStore for every partition, even if one is not needed. We need to address this by allowing the ApplyDirective logic to initialize TranslateStores when they don't yet exist. Move the ImportRoaringShardRequest type to the types package Since the ImportRoaringShardRequest object is part of the Importer interface, we need to move it to a non-root (i.e. pilosa) package. All the other interface types are either concrete types or part of a sub-package (such as roaring). We do this to prevent an implementer of the interface from having to import the entire pilosa package and risk circular imports. buncha changes to support latest dax stuff Move dax related types to /dax sub-package This commit moves all the common "dax" types into the /dax sub-package. The idea is to ensure that featurebase does not import dax at all. It's ok if dax imports featurebase. In the future, we might need to split the dax sub-package (common data types used by muliple molecula data-plan services) into it's own repo. Add type: dax.Schema This isn't currently being used; I started to use is as a replacement for pilosa_client.Schema, but then deferred that. But we'll need to do it eventually, so it doesn't hurt to have this type in place. Export RowIDs.Merge() method for use in orchestrator. Add CreateSQL method to dax.Table type The CreateSQL() method will return the "CREATE TABLE" statement required to create the dax.Table. Comment out confusing writelogger log message. We need to revisit this, but for now, this log message is confusing. Also, rename daxSharder to versionStore. Remove hard-coded AWS account Implement more FieldOptions such as Epoch Some of the FieldOption logic was stubbed out in the dax package. This commit fills that out more; specifically, it adds the dax.Field.Options.Epoch parameter. export stuff needed for TopK in orchestrator export ValCount stuff to implement Percentile in orchestrator export more stuff to support less code in orchestrator, shared objs Port dax repo over to featurebase/dax (run all as sub-services) This commit does ALOT. Sorry. It introduces a `featurebase dax` sub-command which can be configured to run the various dax services as sub-services within the same process, or individually as the lone service in process. It also changes all the URL paths to be prefixed with the service name. So for example, instead of calling localhost:8080/status, you would now call localhost:8080/featurebase/status. Also, note that all services provide a /health endpoint to confirm they are running in process. Clean up integration tests. Remove PILOSA_ config prefix. Remove duplicate clients (mistake from porting dax to featurebase) Rename sub-service "featurebase" to "computer" In the places where we have hard-coded the sub-service name into a URI path, I've tried to tag the line with a comment containing: `// #SERVICEPATHPREFIX` Update copilot manifest files to reference "computer" Port dax/README.md from dax repository Separate (toml) Queryer Config from Injections We needed to separate the toml config from the configuration required to inject sub-services into the Queryer. I'm not sure this is the best solution, but it's *a* solution. So here we are. Clean up (i.e. remove) the queryer "implementations" package Remove old test file Run WriteLogger and Snapshotter as local sub-services. Prior to this commit, the writelogger and snapshotter services only worked when run as separate services. This allows them to be run in the same process as all the other dax services. There is still some naming issues that we should address, but it's functional for now. Clean up (i.e. organize) the intra-service interfaces. Implement alpha Director for local messages from MDS to Computer Prior to this commit, messages from MDS to the computer service were still going over http. This commit introduces an interface implementation which registers the local computer command, and use that command's API to directly reference methods used by the Director. Clean up a few more interface names Add Queryer OpenAPI document. Update copilot manifests to reflect latest changes Add OpenAPI documents for WriteLogger and Snapshotter Add OpenAPI document for MDS service Add OpenAPI document for Computer service Consolidate errors to use fb/errors package. This commit is a first pass at trying to ensure that all of the DAX code uses: "github.com/molecula/featurebase/v3/errors" This package is a wrapper for "github.com/pkg/errors", so going forward we want to avoid importing that package. The only method which isn't backward-compatible is `New()`; the New() method in the featurebase/errors package takes an errors.Code. If this becomes a problem, we could change this by reverting New() and then introducing something like NewCoded(). But for now I think it might actually discourage someone from just creating a New() error without thinking about how it should be coded. Introduce VersionStore interface Move the existing VersionStore code to the `inmem` package as the in-memory implementation of the new dax.VersionStore interface. Introduce NodeService interface With this, the Controller can maintain a registry of nodes by using this NodeService interface as opposed to an in-memory map of nodes on the Controller struct. This also adds an inmem implementation of the NodeService interface. Introduce controller.Balancer interface This moves the existing balancer package to controller/naive package. The idea is to allow us to add a different Balancer implementation in the future. Introduce DirectiveVersion interface This commit also includes *A LOT* of refactoring to use dax.Worker and dax.Job types everywhere instead of strings. Introduce Schemar interface The previous `Schemar` struct was moved to the `schemar/inmem` package, and `Schemar` is now an interface implemented by that inmem package. Remove unused type `nUnit` Add boltdb implementation of VersionStore interface. This removed the previous sqlite implementation; we decided not to use sqlite for now (as a basic, local disk implementation) because it requires CGO. -------------------------------------------- No longer applicable: Add sqlite implementation of VersionStore interface. This commit implements the VersionStore interface using sqlite. Sqlite requires CGO, so this may not be something we want to include, but it's implemented here to get a feel for how an external implementation might be used; the next step will be to determine how the user configured FeatureBase to run using sqlite as a backing store for services like MDS. Add boltdb implementation of NodeService and DirectiveVersion interfaces. Add boltdb implementation of naive Balancer interfaces. This includes the two interfaces defined in `naive/balancer.go`: - WorkerJobService - FreeJobService Add boltdb implementation of Schemar interface. clean up a linter issue Thread context.Context through all the interfaces. Some of the interface implementations are going to use context, so we need to make that part of the interface. The boltdb implementations, for example, take a context. This is probably so we can do things like cancel or timeout operations. Update interfaces to return error; remove `panic(err)` everywhere. Down-rev grpc version to 1.38.0 Later versions (after 1.42.0?) cause MustRunCluster.Close() in tests to deadlock. This commit also adds an `isComputeNode` feature flag around some of the write log and shard/partition check functionality so that it doesn't run under normal conditions (this is excercised by running the sql3 tests for example). Add MDS_Persistence test to cover meta data persistence This adds a basic test which configures the MDS container to use boltdb as its persistence storage, saved on a docker volume. Then, the mds container is stopped/replaced, and we confirm that the data stored on the volume is availble to the new MDS container. Fix a few things after rebase with sql-experiment branch The lastest version of sql-experiment contains a fairly significan refactor of the way query iteration works. This commit adjusts for those changes. pull dax IDK changes in to FB IDK (#2177) * pull dax IDK changes in to FB IDK * Move docker-related IDK build stuff to featurebase root Building the docker image required the root level go.mod and vendor directory. This change moves the make targets to the root level Makefile, and the Dockerfiles now copy the root level vendor directory (and everything else in the root for that matter). * Fix batch- and client-related tests * InitializePoller on MDS restart/replacement Prior to this change, if MDS was restarted, its internal poller (which maintains an in-memory list of nodes to poll) is empty. This is bad, because it doesn't know about nodes that it should be polling. This change fixes that. Upon MDS startup, it intializes the poller with the list of nodes that MDS keeps in persistent storage (currently: boltdb). * Add EFS volume to MDS Copilot manifest This allows us to use MDS's persistent storage (via boltdb) in the Copilot demo by saving metadata in a boltdb file on EFS. * Thread logger.Logger through all dax components * Revert some of the breaking changes from DAX development. When we first started prototyping DAX, we made changes to the featurebase core code which would have broken the existing featurebase functionality. This commit reverts some of those changes. Anywhere that we need to modify core featurebase functionilty, we put it behind some kind of feature flag. This flag is typically determined by whether the running node is a "compute" node (i.e. DAX.COMPUTER.RUN = true). Co-authored-by: Travis Turner <travis@molecula.com> add packaging for DAX need cgo for datagen build bind to 0.0.0.0, pass GOOS and GOARCH explicitly not sure if the explicit GOOS/GOARCH is actually necessary... Get INSERT INTO (aka ingest) working through SQL3 This commit does a few things which I'll try do describe here. - Introduces a Qctx interface. The existing Qcx is an implementation of this interface, and can be used exactly how it has been. But this allows us to abstract away the notion of Qcx in the Queryer (which is handling SQL3) until we're ready to address that. As an example, the Qcx has a notion of a featurebase Holder, but that doesn't make sense when we're at the Queryer layer. For now, the Qctx used in the Queryer is a no-op. - Adds a ComputeAPI interface implementation for the Queryer. This is effectively the Import() and ImportValues() methods used for ingest. The logic here handles the incoming ImportRequest by first doing any necessary column and row translation for the entire request, then it splits the records by shard, and generates a new ImportRequest per shard with only the shard-appropriate records. - Changes the mds.Importer to take an MDS interface implementation (which can be an mds client) instead of an mdsAddress. This allows us to use a localy MDS implementation rather than assuming we need a client to make calls over a network. Add queryer.Importer interface to handle ingest via SQL (#2203) * Add queryer.Importer interface to handle ingest via SQL This is meant to support ingest through SQL when the queryer and the compute services are running in the same process, or when they are on seperate processes and need to talk via http client. * remove datagen from RPM was originally added as a convenience to generate test data, but is unused and annoying because datagen doesn't easily cross-compile due to cgo * add marshalUnmarshal to controller to avoid passing pointers passing pointers across API boundaries can cause unpredictable things in local vs remote configurations. Co-authored-by: Matthew Jaffee <jaffee@pilosa.com> "fix" a few issues with wrong default partition numbers these still need to be properly fixed and actually get the correct data from MDS go mod tidy Introduce TableQualifier (OrganizationID/DatabaseID) (#2220) * add check in ApplyDirective that version is increasing fix TestAPIDirective to make version always increasing * fix docker image build and break out dax test in CI We have to run the DAX integration tests separately as they call out to Docker, and so it isn't easy to run them in a Docker container as the other tests do. So we run them directly on the CI runner which has Docker and Go installed. We also explicitly exclude these tests from running during the other tests. Also my editor was automatically reformatting some comments badly which is why I added the "data" thing in those two places * add timeout to poller * give Poller a default Logger apparently we can NPE sometimes, seen in CI: https://gitlab.com/molecula/featurebase/-/jobs/3028286364 * bunch of testing fixes, mostly IDK/DAX related make MDS error if sendDirectives errors, don't just log. sendDirectives can error if computer nodes disagree about the validity of a schema (for example), in which case it might need to get deleted and user notified somehow. very messy, needs more thought. re-introduce old env prefix to maintain compatibility with master branch make self-contained dax container for IDK testing build IDK images from source (now that all the source is available since it's in the same repo) catch errors in DoExtractQuery in idktest.go fix IDK bug where prefix path was hardcoded in all cases rather than only when useMDS was true fix TestBatchTargetMDS... needed to add field options and catch error when creating table. also needed an _id field * fix env prefix in tests * WIP getting tests to pass, wanna see CI * don't error if we get a zero version directive and we don't have a directive yet * cleanup debugging junk * "fix" future.rename thing, run IDK tests * Introduce TableQualifier (OrganizationID/DatabaseID) This commit introduces a lot of new types (in dax/table.go) related to TableQualifer (which is made up of OrganizationID and DatabaseID), as well as things like TableID and TableKey. For the most part, we try to thread a QualifiedTableID through the entirety of DAX. There are some places (for example in the Balancers, which are just aware of string keys) which use a string TableKey (tbl__org__db__tableid). * Remove some debugging comments * Add Org/DB support to CLI. This commit adds support for special commands: SET SET ORG acme SET DB db1 USE db1 * remove ".pulled" from IDK Makefile I don't think we need it any more as most things can be built locally. I think it was only there to refresh the FeatureBase images that were tagged as master, but we don't need to do that any more. * Change DAX json tags to kebab-case (i.e. hyphenated) This commit also renames some struct arguments to more accurately reflect their type: for example, renaming `Table` to `TableKey` when the type is TableKey. * Return DAX TableName in SHOW TABLES (instead of Index.Name) There are cases where SchemaAPI is used to return DAX friendly table names (as opposed to featurebase index names, which are DAX TableKey). This is an attempt to do that. With that said, it's not ideal because anything could call those API methods and expect the other type. * Fix a bug which wasn't completely dropping a table. When using boltdb as a backend, DROP TABLE wasn't removing the reverse-lookup key for the table in boltdb. * Remove idk/testenv/certs which got accidentally committed. also update .gitignore to include those. * Fix IDK ingest tests to be TableQualifier aware. * Add example Table types to dax/table.com godoc. * ignore idk.Main fields for flags, upgrade commandeer * go mod tidy * Fix DAX integration tests: ingester using wrong ENV VARs We change from ORGANIZATION_ID to ORG_ID and from DATABASE_ID to DB_ID * Clarify things around idk (docker) tests * Stop running TestKafkaSourceIntegration with t.Parallel() This test can't be run in parallel as it's currently written. Doing so allows for interleaving of messages to the same kafka topic between tests. I didn't attempt to modify the test so it could be run in parallel. That could be done, but left for someone more ambitious. Co-authored-by: Matthew Jaffee <jaffee@pilosa.com> Require Directive.Version be a non-zero value. (#2227) Because the directive cached on the holder is not a pointer, its default version is 0. In order to avoid having to compare against that, we just require that Directive.Version start at 1. General, non-invasive code cleanup and comment adjustment. Move ImportRoaringShardRequest out of the types package Early on in the DAX development, I moved ImportRoaringShardRequest into a types package. There must have been some import loop going on, but since that is not longer the case, it's safe to move this back into the core featurebase (er... pilosa) package. Move Transaction struct back into the pilosa package (from types) Revert some name changes (cli -> client) Add DAX Handler CloseTimeout This was implemented in htt_handler.go, but it had been commented out in the DAX handler. This just uncomments that and finishes the implementation. Remove Qcx from queryer.Importer interface This sets us up to revert the Qctx interface that was initially introduced to allow us to abstract away the need for a Qcx when calling the ComputeAPI from a remote service (i.e. the queryer). Add some go-doc comments and remove unused code. Move SchemaManager setup from datagen to idk.Main (#2233) The set for idk.SchemaManager (for dax implementations) was previously in datagen. This may have been because of some import loop problem during development, but that's no longer an issue. The setup for this should be in idk.Main so anything using that can leverage the MDS-specific SchemaManager setup. Fix issues around nil TxFactory First, don't return a nil. Rather return a new *TxFactory (with no holder). Second, don't call `f.holder` in the testhook outside of checking if `f.holder` is nil. Wrap all bare errors Make service prefixes constants Instead of having `"computer"` throughout the code, use instead a constant: `dax.ServicePrefixComputer`. MDS skip errors when sending empty directives also add in the docker-login and ecr-push changes for serverless DAX Fix the logic in Directive.IsEmpty() (#2236) Update the cached value for Index.translatePartitions In the case where a node already knows about an index, but its assignment of partitions for that index changes (for example, when another node goes down and the node in question is now responsible for more partitions than it previously was), then we need to update the cached value of Index.translatePartitions because that's used in translation checks. minor fixes for IDK-related bugs WIP: tokenize CLI to access cloud FB CLI cloud support with automatic token refresh Also adds support for a GET command which allows making HTTP GET queries to cloud CP which can be handy for debugging stuff. E.g. GET /v2/databases buncha little fixes working on writelogger stuff fix writelogger/snapshotter setup bugs implement writelogging for importRoaringShard add debug endpoint to MDS use shard transactional endpoint in MDS datagen add debugging to API related to writelogger revert handleroption change clean up big PR remove "GET" command from CLI for making arbitrary HTTP request to cloud control plane (was a messy hack and not that useful) remove json tags from FB objects where we had to duplicate the object elsewhere due to import loops and weren't actually json encoding it unexport handlerOption which was exported to try to avoid doing certain things if we're in DAX mode, but I didn't end up merging that code. remove (hopefully) unecessary extra call to api.indexField fix some formatting, unexport some vars, godoc, etc oops, fix build failure Update FeatureBase CLI to support a standard deployment The standard deployment uses a different endpoint and request payload. This commit tries to detect is the standard deployment is being used, and if so, it uses a standard-specific FBQueryer. It also modifies the auto-detection logic to try standard featurebase and dax ports in the case where a port was not provided. MDS API refactor (#2259) * MDS API refactor table IDs are exposed but only created server side also cleaned up dax Makefile * clean up review feedback Co-authored-by: Travis Turner <travis@pilosa.com> * remove TablesByName * rip out inmem implementations and use boltdb everywhere * remove inmem balancer, create bolt tempfile by default on startup * WIP on snapshot table impl and test * Minor comment and code layout adjustments. This commit also adds the `Equals` method to `QualifiedTableID` for equality comparisons. It's no longer safe to compare struct (two structs might still be equal even if one of the structs doesn't have a `Name` value. * Use a unique docker network for each dax test Ocassionally we would see some test failures due to a network already existing. This shouldn't happen, but to avoid that, this commit generates a unique name for each sub test (which gets deleted at the end of every test). * Fix one instance of NewQualifiedTableID losing Name We should probably check the other instances and see if Name is getting lost. * simplify unique network stuff and fix api directive tests * Remove TableIDRequest and TableIDResponse types for /table-id (#2267) For the mds/table-id http requests, just use dax.QualifiedTableID as both the request and response types. * remove lattice from dax, no error on node re-reg, dax docker-compose * various updates * WIP: mds-refactor branch review * no-op on SnapshotTableKeys if table is not keyed * Makefile helpers * add doWeCare so controller doesn't fail unnecessarily * clean up table creation (#2272) * Strip underscores from TableID stub name * fix boltdb versionstore tests: generate unique, sorted tables * fix controller test related to reregistering a node * JobSet -> generic Set Co-authored-by: Travis Turner <travis@pilosa.com> Co-authored-by: Travis Turner <travis@molecula.com> Cleanup after rebase on master The latest rebase on master entailed all the client/batch changes as well as some of the qcx refactoring. It made for a hairy rebase. This commit fixes some of the tests that were failing after that rebase. Fix batch/client import loop missed during rebase (#2280) It's not surprising that `batch` can't import `client`. It was doing that here (importing an error type from the `client` package). What is surprising is that it's okay for `batch_test.go` to import `client` even though `batch_test.go` is an internal test and therefore part of the `batch` package. different boltDB's for schemar/controller, explicit balancers nice helpers for dax docker-compose, make build really fast build FB binary outside of docker, then create Docker image with its working dir in an empty subdirectory so it doesn't send a GB of context to the daemon. error on unassigned jobs and use client with timeout fix CR feedback deregister batch of nodes also make removal faster via director dial timeout implement WorkersForJobPrefix so orchestrator doesn't make up shards also fix some godocs and remove unused method Run sub-tasks of a Directive concurrently in a worker pool. (#2275) * Run sub-tasks of a Directive concurrently in a worker pool. This allows the compute node to concurrently load shapshot and writelog data concurrently, instead of one keyset/partition/shard at a time. It introduces a config parameter called `DirectiveWorkerPoolSize`. * code review cleanup * Use unique container names in DAX integration tests We were seeing "container already exists" errors in CI, so just to be safe, this commit constructs a unique container name for every container in the DAX integration test run. Stub in SystemAPI to Queryer (note: will not work if used) This just makes is so that dax can compile. Actually implementing system-table functionality for dax will take some planning. Tlt/dax merge prep (#2282) * Remove copilot directory * Remove Dockerfile-datagen-long * Remove orphaned RegisterNodeRequest This type is not defined in the dax/mds/http package. * implement TIMEQUANTUM and TTL in Table.Field type * Remove the "service" misdirection in queryer/writelogger/snapshotter. We had originally used an additional layer, er.. package, for a "service". The main distinction was that the Config differed in that it was internal, unlike the Config that we need to provide for the top-level server config (i.e. toml). Having that additional layer just to support a different Config seemed premature at best. So I'm removing it. * Remove dax docker containers no longer used in tests Since we run everything as "featurebase", we don't have multiple container types anymore. * Some minor comment updates * Remove nfpm stuff related to dax * Fix linter issues Fix "duplicate" issues raised by sonarcloud. run docker components of dax integration tests with coverage trying to get dax integration coverage add coverate volume mounts throughout dax integration tests add a lock, tweak dax Makefile, remote flag on query handler remove some unused code convert batch tests to use clustertests to get coverage maybe fix clustertests more authclustertests fixes, test is failing locally but also seems to have been silently failing in CI prior to these changes... let's see if it's still silent fix some lint to kick CI just re-running the job wasn't working... strange behavior remove RetryLogic test and pipe which don't work RetryLogic test removed due to etcd changes. Seebs thinks we shouldn't test this here. Pipe was being ignored since we're no longer using "bash -c" to execute the command. If we need to generate that output file we'll either have to reintroduce bash -c and set -o pipefail so that it actually fails properly, or figure out some other solution. shooting into the dark... first cut at bulk node registration remove unused stuff from batch tests, set coverpkg to ../... batch registration timeout and fix tests disable most tests and don't run fb background batch test debuggin!!!!!!!!! and then he tried this.... Implement importer (for INSERT INTO) in the Queryer Prior to this, we we passing a nil value in for the importer to the planner.NewExecutionPlanner in the Queryer. This meant that INSERT INTO statements didn't work. Now they should. It uses the importer that we build for IDK in /idk/mds/importer.go, and wrapps that with a type that can determine if the provided string "index" is of the form indexName or TableKey. turn off debug mode, fix log saving Run sql3 test definitions in a dax integration test There are currently 22 tests which are not passing. They are skipped in the "skips" slice. WIP, not working, pql queries to tests Add TableQualifier to PQL query logic in the Queryer Add more PQL tests to the keyed table Allow instant node registration if registration-batch-timeout=0 When running dax services in process, we don't want to wait 3s for the compute node to register; we know it's there because it's in the same process. Fixes related to IncludesColumn PQL test. Tests for ConstRow and FieldValue cleanup add UnionRows and Options, better error reporting on bad queries delete unused schemar client.go, clean up unused in batch test CI move test timeouts into more reasonable territory apparently this had already been done, but got merge-stommped at some point move dax bolt test helpers into dax package Add computer CheckIn routine (#2296) * Add computer CheckIn routine This adds a background routine which sends a "check-in" request to MDS every <interval>. This is to address the case where the poller has removed a computer node from the node list (due to a network fault, for example), but the node is still healthy and becomes available again. In that case, the node needs to "check-in" to tell MDS it is still there. MDS will likely send the node a new directive with Method=reset telling the node to delete all of its data an apply the latest directive. * Don't send directives to Deregistered (i.e. removed) nodes We have an issue where we're locking on sendDirective in the controller, and when the node is unavailable, the send hangs and never releases the lock. This is a temporary fix for that until we address the real problem. Fix .gitlab-ci.yml after rebase fix some indentation shenanigans |
||
|
Garrison Davis
|
0f5a56c958 |
Stop using string keys in contexts
This fixes the OriginalIP and RequestUserID in the main featurebase package, and the Access and Refresh tokens, the UserInfo, and the []string of Indexes passed with context.Context(s) in the authn package. An empty struct was used for all of these keys (and relevant helper functions we added) to avoid allocations where possible while still using the context functionality. Some of the logic in the server.GetIndexes function was fixed. |
||
|
Seebs
|
fff9ddc1f5 |
drop anti-entropy feature, since it doesn't work
The anti-entropy feature has never actually worked. We've been talking about removing it or replacing it for ages, but haven't had a concrete motivation. But the anti-entropy interface is the sole user of several components of the Tx interface, and now that we're trying to replace that interface, being able to drop those components has some appeal, so let's remove the one thing that used them, in the hopes that this will simplify life. This also lets us drop ForEach and ForEachRange, which were barely used at all. The one surviving usage (CSV export) can be handled by using the container iterator we already have, and making ContainerCallback exported so we can use it to just call things for every bit. |
||
|
Seebs
|
16ccbc461a |
Drop the ingest subpackage and related endpoints.
The internal/ingest and internal/schema endpoints were developed with intent that they'd be the primary interface new users would work with, because they were Easy To Use, and did not require any kind of setup, the counterpoint being that ingest done this way had performance issues because it ended up with huge amounts of JSON parsing to reformat things into our native format. But this was understood to be the price of providing a new-user-friendly JSON ingest experience. A year later, we have no evidence that it's ever been used. We never even moved it out of the `/internal` path. It's a lot of very complex fiddly code and we don't seem to be using it, and at this point, our anticipation is that if we really need something, we'll use CSV, which we already have working, or something in the new SQL code. Either way, we don't seem to be using this. |
||
|
Seebs
|
c3b032d5cb |
drop ioutil
The ioutil package is deprecated, with all of its functions having moved into os or io. Do the replacements so we stop having this impending. |
||
|
Seebs
|
d710f83e2d |
reduce verbosity of retryablehttpclient
The default client appears to be pretty spammy and flood us with debug messages about POST and GET requests, and honestly we don't really need these or benefit from them, I don't think, so let's not. |
||
|
Seebs
|
3c9a338d09 |
log errors we retry from, reuse retryablehttp client more consistently
We make clients pass in a context which has an associated data structure which can be used to annotate errors we've encountered inside retryablehttp. You might ask why we do this instead of just giving retryablehttp a logger that isn't nopLogger; the answer is that retryablehttp's logging is too spammy to use. So we create a context, and then pass that in for requests, so that when we check to see whether we should be retrying, we can log the errors we encounter, so if we encounter errors we can log something specific instead of "gave up after 5 retries". We also make a second retryable client that would forward authentication, and reuse that, rather than making a new client for every call that needs to forward authentication. This allows us to avoid the problem where that inner client wasn't picking up the timeout settings we'd applied to the parent. We log errors with Errorf if we actually failed entirely to do the request, we log them with Infof if there's errors but the request ultimately succeeds. We also try to log part of the response body when there is one, but there wasn't an error. If there's no response, we produce a spurious 500 and a message saying that we produced it. |
||
|
Samir Patel
|
6f1514933b |
verify available space before backup
compares free space in output directory to the usage of either the data directory or index depending on what is being backed up. - adds an http_handler endpoint to get usage of a particular index - adds InternalClient methods to get DiskUsage and IndexUsage |
||
|
Seebs
|
b3a4e52a13 |
simplify, streamline, and possibly debug embedded etcd
The root problem this is attempting to address is sporadic
weird cases in which etcd mistakenly thinks it's down even when
it's up. I am not confident that this is addressed, but there's
a reasonable chance that it is, and I can't trigger it at the
moment, but it was always sporadic, so that doesn't prove much.
There's a lot going on here, and it comes into roughly three
categories.
First: Dropping unused/unneeded code. There's a lot of leftover
bits from the initial development and refactoring of this.
Second: Unifying and shuffling some of the design. We had
multiple interfaces which are functionally impossible to
usefully implement separately, so they're combined together,
and in some cases, moved.
Third: Streamlining logic and simplifying design choices.
This is combined into one commit because the changes are
thoroughly entertwined with each other and you can't usefully
break most of them out.
Also, a bunch of test coverage for most of these changes.
Big changes:
We merge the topology and disco packages. The topology and disco
packages being separate creates a complicated tangle of problems
and dependencies. The fundamental problem, approximately, is that
topology.Node has to track disco.NodeState.
There's three core interfaces interacting here:
topology.Noder (maintains list of nodes)
disco.Stator (maintains the state of a node)
disco.Metadator (stores, possibly retrieves, node metadata)
But the node state mantained by the Noder *is* the set of node
metadata, plus state updates produced by Stators. The only actual
non-trivial and usable implementation of these interfaces is a single
thing which implements all three, and in which the implementations
share a single backend data source which they are all modifying.
But you can't move Noder into disco, because Noder has to refer
to topology.Node, but topology.Node refers to disco.
Solution: First, merge these two packages. Second, merge these
three interfaces, to provide a single interface which is more
clear about the fact that (metadator.)SetMetadata() and
(stator.)Started() are both changing the output we'll get from
(noder.)Nodes().
We rework the node state tracking.
We have this nodeStates map which is almost unused. Really, we
don't need it at all. Every node's state is either its last heartbeat
state or "Unknown", so we simplify this a bit. Also, we ensure that
the populateNodeStates function itself is yielding the sorted nodes
list, so we don't have to be as worried about possible later lookups
of sortedNodes happening outside a lock. We also add diagnostics
for deleting nodes from the metadata list (this should never happen),
and try to track heartbeat state more closely.
This is *probably* what fixes the underlying reported problem,
if anything did.
Still an open issue: Make heartbeat state changes aware of when
they're talking about *this* node and possibly not try to
mark it down? Except this may have a flaw: That would result in
each node disagreeing with other nodes in etcd about the state
of that node in the failure cases, and undermine the point of
using etcd to keep these states consistent.
We reduce the number of contexts and cancelfuncs in the etcd wrapper.
We create a shared context for the non-etcd.embed children of our
etcd wrapper, the heartbeat/keepalive and the node watcher, so we
can cancel that one context and cancel all of those at once, so
we don't need to separately track a function to call to cancel
the watch, AND be closing another channel. Also, our shutdown
now propagates automatically to the various etcd API calls we've
made for things like the node watcher and keepalive calls.
We still need to watch that channel in watchNodesOnce, though,
because apparently the watch doesn't yield an error even if the
context calling it is canceled. Whee.
This should reduce the risk of ending up in an inconsistent state,
and also the Close() function is probably idempotent now.
Smaller changes:
* Remove config-generators that existed to generate etcd
configs but were used only for tests that no longer exist
or make sense.
* Move the logic to generate etcd configs into the etcd
package, instead of the "testing" subpackage. This allows
us to write a self-contained config generator for
clusters where the nodes know about each other, but do
this just with etcd, not with full featurebase servers.
* Move the thing generating `fake:%d` socket names into
the etcd package, which is the only place we use it.
Also simplify it slightly.
* Don't panic on invalid URLs, report errors from them.
* At least try to use etcd's config.Validate functionality.
It's underdocumented, so we're not sure what it will report,
but at least if it does we'll get reports from it and
know what they are?
* Try to handle CompactRevision errors from watches more
correctly -- after a CompactRevision, any future attempt
to watch from a lower revision will necessarily fail, so
we adjust our target revision up. We don't have good
testing for this.
* Drop the Metadata() method (that used to be in Metadator)
because nothing ever used it and it didn't make much sense
to try.
* Convert SetMetadata from taking an arbitrary json blob
to taking the only data that would ever be valid since
we always use it to extract node information anyway.
* Drop several unused functions, unexport things only used
internally.
* Replace Started() with SetState("STARTED"), allowing us
to write tests that mess with states. We weren't really thinking
carefully about state transitions sometimes and now it's much
easier to do that thinking.
* Stop leaving stray localhost:2380 and localhost:2379 in
our embed config. We still sometimes see peer requests from
those and I honestly don't know why, but at least it should
be rarer.
|
||
|
Seebs
|
fd9d4de31d |
Remove most of the resize-related logic
We had two different, incompatible-with-each-other, and both individually broken, partial implementations of resizing logic. There's the original pre-etcd resize, and then the etcd resize, and neither works, but there's conflicts between the ways they don't work. No attempt to fix this is likely to yield decent results, so instead, we yank them both out entirely, so if we decide to implement resizing (which we will) we won't be confused by stray code pertaining to resizing that's not really hooked up to anything. We're leaving the resize messages in protobuf to avoid renumbering protobuf messages. We rename some of our message types to UNUSED0, etcetera, so that any code still using the old names won't compile, to make sure we get rid of it, but we can't just drop the numbers without breaking rolling restart. The Resize_AddNode tests are removed not just because we don't have resizing, but because they were completely broken anyway and never worked at all. But there's no reason to fix them because they exist to fix the functionality we didn't have and are now removing the vestigial remains of. We also drop the one usage of the AddNode function of Noder, because it was used only by one test code fragment that was creatincg clusters, and that can be done more correctly. There were no other call sites at all. We mark the monitorAntiEntropy function to be ignored by code coverage because it's not actually being covered. There's a separate ticket for removing that entirely. |
||
|
reesporte
|
8ba81643d2
|
[FB-1379] Create a featurebase subcommand to obtain an auth token (#2079)
* Add CleanOAuthConfig endpoint We will use this to get the OAuthConfig information, without the client secret, from FeatureBase without having to have access to the config file. This will be useful for the auth-token subcommand. * Add string manipulation utility functions Go doesn't have native support for these kind of things, so I added this to make it easier to do string reversal, and replacing the first string encountered from the end of the string to the front. * Add auth-token subcommand This is for work on [FB-1379](https://molecula.atlassian.net/browse/FB-1379). We need this new auth-token subcommand to allow users to get access and refresh tokens without having to login to featurebase via the UI. This commit adds that functionality. * error on oauth endpoint if auth isn't on * https as default scheme in cmd, not internalclient |
||
|
reesporte
|
60e6900c2e
|
Add refresh token header/cookie (#2071)
* Add refresh token header/cookie As part of work on automatic refreshing of access tokens in the grafana plugin (FB-1377), we will now accept a refresh token in the "X-Molecula-Refresh-Token" header or the "refresh-molecula-chip" cookie. This refresh token will be used if the access token is expired. To achieve this, there was a lot of plumbing that had to be done. Here is a list of some of it: * Added lots of constants for the new values. * Removed token cache, since we will be keeping state on the clients. * We now only refresh tokens when they are expired, which is more inline with the OAuth spec. * Refactored SetGRPCMetadata to be simpler to read. * Refactored AddAuthToken. * Update failing tests. * We now don't split GRPC cookies on ";". Not sure why we did that before tbh. I also added TODOs to add the refresh token to other subcommands. This is out of scope for my current ticket, but it would be nice to have in the future. * remove unnecessary context from Authenticate * Add comments on why we check both cases for headers It's because some GRPC clients lowercase metadata names. I've run into issues with this enough that I think it's worth the extra checks. We prefer lowercase though, because that's "standard". * Fix test that broke during rebase |
||
|
souhailanoor
|
3986e202bf
|
FB-1378: Use IP whitelisting for ingest authentication and authorization (#2070)
* Use IP whitelisting for ingest For ingest, use configured IPs to authenticate the requests. Auth-token will no longer be used for requests from ingest consumers. If IP in request is in configured IPs, authenticate and authorize as an admin. If IP in request is not in configured IPs, proceed with the standard authentication/authorization using ADD. * need to remove port from client IP * addressed review comments |
||
|
Travis Turner
|
7ccc845aac
|
Change Ttl to TTL (#2038)
* Change Ttl to TTL Following go convention, acronyms should have a consistent case. See [Initialisms](https://github.com/golang/go/wiki/CodeReviewComments#initialisms) This commit changes some public-facing methods, so any code importing this package and using these methods will need to be updated. * rewrite Ttl -> TTL Co-authored-by: reesporte <reesedporter@gmail.com> |
||
|
reesporte
|
f529e723e7 |
refactor AddAuthToken to fallback to userinfo
This commit addresses a bug in https://molecula.atlassian.net/browse/SUP-200 where the Authorization header was not being set correctly when the token was passed via "userinfo" in the context and not "token". Now, we prefix the token with "Bearer " when the token comes from userinfo. This commit also adds a unit test for this function, and simplifies logic. It also fixes a test that didn't quite test the behavior we wanted. |
||
|
hphamMolecula
|
a7a9722722
|
Merge branch 'master' into fb-1188-ttl | ||
|
reesporte
|
248dc4fe85 |
rip out ui/usage
addresses concerns in [fb-1127](https://molecula.atlassian.net/browse/FB-1127) TLDR; /ui/usage was a hotbed for issues and SEs have been turning it off anyway for ages |
||
|
Hoang Pham
|
e16171cb4b | FB-1188 - Added TTL field option | ||
|
Matthew Jaffee
|
6cc5d198ee |
remove unused stuff and fix a bunch of random staticcheck issues
sorry... once I saw, I couldn't unsee |
||
|
Matthew Jaffee
|
c2ed9ecdba | remove InternalQueryClient | ||
|
Matthew Jaffee
|
254bacc40c |
remove http subpackage and bring implementations into core
remove interfaces as necessary |
Renamed from http/client.go (Browse further)