The attempt to set the TrackExistence option for fields
happened before checking whether the field was created
successfully or not. Credit to Rachith for spotting this.
Bug was introduced with the TrackExistence stuff, but
we apparently never had a test case for invalid min/max
values.
* FB-1739: Add ability to add a description to a table on creation
- Added CommentOption to handle text after COMMENT option.
- added description field in the createtable plan.
- The description is stored in the existing index metadata.
(cherry picked from commit ad350c2d49)
* Introduce ServiceManager and Refactor DAX Integration tests
The ServiceManager provides an interface with which to manage
featurebase (dax) services (mds, queryer, computer). It replaces the
confusing interface implementations in /dax/server/server.go (which
optionally used pointers to in-process objects to satisfy an interface)
with (for now) http implementations. The thought is that even if we're
running all services in-process, we should communicate between services
over http in order to mirror what we would do in a production
environment where the services are running on different nodes.
This batch of commits does quit a lot, most of which is captured here:
- Added `path` support to `dax.Address`. Address is now a string of the form [scheme]://[host]:[port]/[path].
- Added `Holder.directiveApplied` to determine (in tests) if the computer has completed applying the latest directive. This is somewhat temporary until we improve the mds-to-computer logic.
- Removed the "service prefix" code which was prepending client URL paths with the prefix. Instead, the serviceType (mds, queryer, computer[n] is now part of `dax.Address`).
- Removed, from the dax config, the top level `StorageMethod` and `StorageDSN` and now just have `MDS.Config.DataDir`.
- Added `Computer.Config.N` to specify the number of computers to run in-process.
- Moved the `pilosa.MDS` interface to `computer.Registrar`. This is an example of getting the interfaces defined in the right packages.
- Added `SnapshotTable()` method to the mds client (to align with its API).
- Changed `Balancer.AddJob()` to `Balancer.AddJobs()` to support, for example, adding 256 partitions in a single call. Refactored some of the naive Balancer to account for this.
- Added a `Seed` to the top-level config. It's not really useful because of package `crypto/rand`.
- Added an in-memory implementation of the DisCo interface and disabled etcd in a computer service.
- Create sepearte data-dirs for each in-process computer.
- Disabled grpc in dax.
- Modified the sql3 test definition format to support multiple insert steps and separate query results (to align with those steps).
* Changes necessary to get multiple computer instance running in-process
For now the config looks like this:
```
[computer]
run = true
n = 4
```
but we can probably just change that to be something like:
```
[computer]
run = 4
```
*Issues found running multiple "computers" in-process*
- grpc was trying to bind on the same port
- changed GRPCListener from `*net.TCPListener` to `net.Listener`
- created a nopListener and set to that for now (i.e. disabled grpc)
- etcd was starting more than once
- changed dax to use in-memory implementations of the disco interfaces (i.e. stop using etcd)
- IDAllocator (which uses boltdb) was trying to open the `idalloc.db` file more than once
- realized we have to set separate data-dirs for each holder. that fixed it.
* Port dax integration tests to ManagedCommand
* Modify Balancer-related methods like AddJob to AddJobs
There were (and still are) a lot of places where we were adding on job
at a time, even when we had a long list of jobs to add. This resulted in
every job add (for example adding 1 of 256 shards) taking ~40ms, or over
10s to create a keyed table. One reason was because each job add was
making multiple boltdb transactions.
* Port over more dax integration test stuff
* Add DirectiveApplied to signify that snapshot/writes have loaded.
We use this in tests to avoid using sleeps.
This should be considered temporary; we're going to need a more robust
solution for determining when a computer node is ready to serve complete
data.
* Finish porting dax integration tests
* Improve godocs
* Remove docker-based DAX integration tests.
* go mod tidy
* Move test/managed.go to avoid package conflicts
* Modify IDK integration tests to work with ServiceManager changes
This is really just computer -> computer0
And the MDS DataDir config change.
* cleanup found during review
* echo $CI_COMMIT_REF_SLUG in CI
* remove docker image arg, use build instead
(cherry picked from commit 2843f218bc)
* fixed error messages for alter table add and drop; added test coverage
* Removed two CI tests that are failing intermittently for no known reason.
(cherry picked from commit 3c2c8c6011)
