This commit introduces an interface called `TableKeyer` which anything that means to represent a "table"
can implement. Examples are `dax.QualifiedTable`, `dax.Table`, and `string` (for legacy pilosa calls
where Execute simply took `index string`).
In the case of `orchestrator.Execute()` and `qualifiedOrchestrator.Execute()`, we are intentionally strict
about which type of `TableKeyer` the respective method accepts. If we find, in the future, this is too
restrictive, we can loosen that; but for now it helps us understand what is expected.
(cherry picked from commit a61d1a9571)
* Move batch.Importer interface to pilosa.Importer
In addition to moving the interface, it updates all the methods to use
dax.TableID (for example) intead of a string pilosa index name.
* Change unused onPremImporter methods to no-op.
onPremImporter is a wrapper around API which implements the Importer
interface. This is currently only used by sql3 running locally in standard
(i.e not "serverless") mode. Because sql3 always sets
`useShardTransactionalEndpoint = true`, There are several methods which this
implemtation of the Importer interface does not use, and therefore they
intentionally no-op.
(cherry picked from commit 12d608c80d)
* Thread Table.Description through SchemaAPI
* Thread Table.CreatedAt through SchemaAPI
* Thread Field.CreatedAt through SchemaAPI
(cherry picked from commit 734477aaee)
* This adds in support to the lattice UI application to use the SQL3
endpoint. If the `/sql` endpoint returns 404, it will use the SQL1
endpoint. If the `/sql` endpoint is available, it will send SQL queries
to that. It does not try the SQL1 endpoint if the SQL3 endpoint returns
an error processing the query. That is, it is all SQL3 or SQL1.
- Below are the specific changes:
- Adds a file that contains functions for interacting with http services as opposed to just grpc/event-based services. As of this commit, it is only the SQL3 endpoint.
- This adds a variable to track if we are using the SQL3 endpoint or not
- This adds a function to handle the response from the SQL3 endpoint
- Adds a function to eventServices to query the sql3 HTTP endpoint
- Fixed a missing semicolon in grpcServices
Co-authored-by: Fletcher Haynes <fletcher.haynes@generalassemb.ly>
(cherry picked from commit 7ab453e289)
* WIP: Convert SchemaAPI to be DAX-centric
* Tables(), CreateField()
* CreateTable(), DeleteTable(), DeleteField()
* More cleanup
* Remove the old SchemaAPI
(cherry picked from commit a15783cb49)
* Move batch.Importer interface to pilosa.Importer
In addition to moving the interface, it updates all the methods to use
dax.TableID (for example) intead of a string pilosa index name.
* Change unused onPremImporter methods to no-op.
onPremImporter is a wrapper around API which implements the Importer
interface. This is currently only used by sql3 running locally in standard
(i.e not "serverless") mode. Because sql3 always sets
`useShardTransactionalEndpoint = true`, There are several methods which this
implemtation of the Importer interface does not use, and therefore they
intentionally no-op.
(cherry picked from commit 12d608c80d)
* Thread Table.Description through SchemaAPI
* Thread Table.CreatedAt through SchemaAPI
* Thread Field.CreatedAt through SchemaAPI
(cherry picked from commit 734477aaee)
* This adds in support to the lattice UI application to use the SQL3
endpoint. If the `/sql` endpoint returns 404, it will use the SQL1
endpoint. If the `/sql` endpoint is available, it will send SQL queries
to that. It does not try the SQL1 endpoint if the SQL3 endpoint returns
an error processing the query. That is, it is all SQL3 or SQL1.
- Below are the specific changes:
- Adds a file that contains functions for interacting with http services as opposed to just grpc/event-based services. As of this commit, it is only the SQL3 endpoint.
- This adds a variable to track if we are using the SQL3 endpoint or not
- This adds a function to handle the response from the SQL3 endpoint
- Adds a function to eventServices to query the sql3 HTTP endpoint
- Fixed a missing semicolon in grpcServices
Co-authored-by: Fletcher Haynes <fletcher.haynes@generalassemb.ly>
(cherry picked from commit 7ab453e289)
* WIP: Convert SchemaAPI to be DAX-centric
* Tables(), CreateField()
* CreateTable(), DeleteTable(), DeleteField()
* More cleanup
* Remove the old SchemaAPI
(cherry picked from commit a15783cb49)
* FB-1739: Add ability to add a description to a table on creation
- Added CommentOption to handle text after COMMENT option.
- added description field in the createtable plan.
- The description is stored in the existing index metadata.
(cherry picked from commit ad350c2d49)
* Clean up dax service interfaces
Rename some of the `computer` interfaces and organize them in the
appropriate files.
Remove `dax/computer/alpha` package
* Remove ComputeAPI (it was replaced by batch.Importer)
* add nss-tools dependecy to smoke test
(cherry picked from commit 969bf055b2)
* moved the debug code to the right spot
* enforce int min/max constraints on inserts
* add a check for decimal min and max
* fixed borked tests
* fix the decimal to int conversion in constraint check
Co-authored-by: Travis Turner <travis@molecula.com>
(cherry picked from commit e392ce3460)
* Introduce ServiceManager and Refactor DAX Integration tests
The ServiceManager provides an interface with which to manage
featurebase (dax) services (mds, queryer, computer). It replaces the
confusing interface implementations in /dax/server/server.go (which
optionally used pointers to in-process objects to satisfy an interface)
with (for now) http implementations. The thought is that even if we're
running all services in-process, we should communicate between services
over http in order to mirror what we would do in a production
environment where the services are running on different nodes.
This batch of commits does quit a lot, most of which is captured here:
- Added `path` support to `dax.Address`. Address is now a string of the form [scheme]://[host]:[port]/[path].
- Added `Holder.directiveApplied` to determine (in tests) if the computer has completed applying the latest directive. This is somewhat temporary until we improve the mds-to-computer logic.
- Removed the "service prefix" code which was prepending client URL paths with the prefix. Instead, the serviceType (mds, queryer, computer[n] is now part of `dax.Address`).
- Removed, from the dax config, the top level `StorageMethod` and `StorageDSN` and now just have `MDS.Config.DataDir`.
- Added `Computer.Config.N` to specify the number of computers to run in-process.
- Moved the `pilosa.MDS` interface to `computer.Registrar`. This is an example of getting the interfaces defined in the right packages.
- Added `SnapshotTable()` method to the mds client (to align with its API).
- Changed `Balancer.AddJob()` to `Balancer.AddJobs()` to support, for example, adding 256 partitions in a single call. Refactored some of the naive Balancer to account for this.
- Added a `Seed` to the top-level config. It's not really useful because of package `crypto/rand`.
- Added an in-memory implementation of the DisCo interface and disabled etcd in a computer service.
- Create sepearte data-dirs for each in-process computer.
- Disabled grpc in dax.
- Modified the sql3 test definition format to support multiple insert steps and separate query results (to align with those steps).
* Changes necessary to get multiple computer instance running in-process
For now the config looks like this:
```
[computer]
run = true
n = 4
```
but we can probably just change that to be something like:
```
[computer]
run = 4
```
*Issues found running multiple "computers" in-process*
- grpc was trying to bind on the same port
- changed GRPCListener from `*net.TCPListener` to `net.Listener`
- created a nopListener and set to that for now (i.e. disabled grpc)
- etcd was starting more than once
- changed dax to use in-memory implementations of the disco interfaces (i.e. stop using etcd)
- IDAllocator (which uses boltdb) was trying to open the `idalloc.db` file more than once
- realized we have to set separate data-dirs for each holder. that fixed it.
* Port dax integration tests to ManagedCommand
* Modify Balancer-related methods like AddJob to AddJobs
There were (and still are) a lot of places where we were adding on job
at a time, even when we had a long list of jobs to add. This resulted in
every job add (for example adding 1 of 256 shards) taking ~40ms, or over
10s to create a keyed table. One reason was because each job add was
making multiple boltdb transactions.
* Port over more dax integration test stuff
* Add DirectiveApplied to signify that snapshot/writes have loaded.
We use this in tests to avoid using sleeps.
This should be considered temporary; we're going to need a more robust
solution for determining when a computer node is ready to serve complete
data.
* Finish porting dax integration tests
* Improve godocs
* Remove docker-based DAX integration tests.
* go mod tidy
* Move test/managed.go to avoid package conflicts
* Modify IDK integration tests to work with ServiceManager changes
This is really just computer -> computer0
And the MDS DataDir config change.
* cleanup found during review
* echo $CI_COMMIT_REF_SLUG in CI
* remove docker image arg, use build instead
(cherry picked from commit 2843f218bc)
* fixed a bunch of issues with non-pql aggregation; moved some decimal related functionality; made top actually top (for the non-pql case); experimental create function
* drive up test coverage
(cherry picked from commit 0be0c42b66)
implements an fb_exec_requests system table. The purpose of this table is to allow access to internal state to see what queries are running and have been run.
Co-authored-by: Travis Turner <travis@molecula.com>
(cherry picked from commit 47d8be26f5)
Reproducible builds are something we should be doing, and we are there
as far as making them in CI is concerned with this change.
The changes to the Dockerfile/Makefile do nothing if the
SOURCE_DATE_EPOCH environment variable is not set before `make build`
happens, or if the build arg is not passed in to docker.
(cherry picked from commit fd93fc99c9)
* updated optimizer to be smarter when trying to push a top operator down; added test coverage
* skip a dax sql test that keeps failing
(cherry picked from commit e0d6b292bc)
* fixed error messages for alter table add and drop; added test coverage
* Removed two CI tests that are failing intermittently for no known reason.
(cherry picked from commit 3c2c8c6011)
FB-1771
In api_test.go, it was being used to make sure that incorrect input
produced the right errors; this is now handled by making sure the
error isn't nil and then checking its string against the expected
error string.
In executor_test.go and internal_client_test.go, it was being used
to compare QueryResponse structures, which contain an error.
handler.go now has a function specifically for comparing them,
which can provide additional detail if necessary.
Added a test for SameAs to handler_test.go.
(cherry picked from commit 775fd0b08c)
* tighten up checks for order by expressions fixed ordering by expressions
* added testing to cover order by cases
* Add DecimalAgg member to proto GroupCount definition
In DAX, where we have split the orchestrator from the executor, and the
orchestrator can run on a different host, there are cases where
`GroupCount`s can travel over the wire via the Internal Client. In these
cases, when the group count contains a decimal aggregate, we need to
send that value as the appropriate type.
* fixed missing cases in order by and case block eval
Co-authored-by: Travis Turner <travis@molecula.com>
(cherry picked from commit 158cc669d9)
stdout/stderr around
A lot of functions in the cmd and ctl packages were passing these
around and barely using them. Replaced them with a logger for most
functions. Some functions get an io.Writer instead so that their
tests can find the output they're looking for.
More cleanup on fb-1766: reworked the tests that were using io.Pipe
or os.Pipe to check their results so they now use a bytes.Buffer.
Unexported some variables that didn't need to be exported.
Fixed NewConfigCommand to use the provided stderr, not os.Stderr.
Added tests for rbf_dump, rbf_page, and keygen, since those weren't
being tested at all.
Added chksum_test, final cleanup.
(cherry picked from commit f627199acb)
There are now only four tests remaining which do not pass.
One is related to error format mismatch.
Two require orchestrator work.
One won't pass until table name conversion is supported for multiple
tables.
(cherry picked from commit 5bf5b5364d)
MarshalLogMessage serializes the log message and prepends additional encoding
information to each message. Currently, we prepend three bytes to each log
message:
byte[0]: encodeVersion - this is currently a constant within the code. If we
modify structs such that they encode differently, we'll have to change the
constant and keep previous versions of structs for deserialization.
byte[1]: encodeType (e.g. "json", etc.)
byte[2]: logMessageType
If we get into a situation where we want more flexibility in these message
header bytes—for example, if we want to use more than three bytes—we could do
something with the first bit of the encodeVersion: if it's 1, that could
indicate that there are additional header bytes, and the following seven bits
could indicate how many.
(cherry picked from commit 6740bc250e)
* Fix formatting in CLI results with custom SQLResonse.UnmarshalJSON
When I started this, it was meant to be a quick fix to address the confusing
result formats we were seeing in the CLI. For example, all large integer values
were displayed in scientifc notation. This is because we were passing the result
types from JSON (in this case, float64) into pretty print. Similarly, `IDSets`
and `StringSets` where being printed using the default go Stringer for the types
[]int64 and []string respectively.
I started by writing a customer UnmarshalJSON() method for the `SQLResponse`
type. Part of this (the part which converts data types based on header types)
was already being used in dax tests, so this just formalizes that logic as part
of the `SQLResponse` type.
Then I realized that the sql3 tests (run against the `sql3` package) were
failing because sql3 is not actually returning the `IDSets` and `StringSets`
types. A future task is to formalize return types, define them, and modify sql3
to return them. Once that is done, we can remove the "typed" switch in the
`SQLResponse` json unmarshaller.
Another significant change is the modification to the `ExprDataType` interface:
```
type ExprDataType interface {
exprDataType()
TypeName() string
TypeDescription() string
TypeInfo() map[string]interface{}
}
```
I added two more methods in order to distinguish between a type (`DECIMAL`), its
description (`DECIMAL(2)`), and its type info (`"scale": int64(2)`). Currently,
the description can be used as the field definition in a CREATE TABLE statement,
but we may want to re-think that. Also, Decimal is the only type currently using
TypeInfo.
Finally, I tried to consilidate things around `dax.FieldType` instead of
comparing against parser types outside of sql3. We still have some sql3 parser
and planner types lurking about, but we can address those in future commits.
* Add some test coverage
* smoke test expected INT, now int
* minor fixes
* Introduce WireQueryResponse and related types
This also changes dax.FieldType to dax.BaseType.
* Populate WireQueryResponse correctly
Currently this is in the http handler, and in the queryer.
* Convert sql3 and dax tests to expect pilosa.WireQueryField in results
* fix PQL tests in the SQL defs
* Address a few of the skipped sql tests in dax
(cherry picked from commit f4385df2cf)
this commit changes the way the plan is retrieved; implements Stringer on types.PlanExpression in preparation for HAVING support; removes last vestiges internal float64 arithmetic; implements a filter on PlanOpFilter; fixes various bugs in the PlanOptimizer when rewriting qualified references
* fixed selects with unqualified identifiers
* handle bad and non-existent query param inputs more appropriately
* added test coverage for PlanExpression Stringer
Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
(cherry picked from commit 5f662d2bce)
We use part of the dax.TableName in the pilosa.Index.Name.
This just ensure that we don't let invalid characters get through.
(cherry picked from commit 10e8aa5c45)
