enriched metadata for tables
added support for the concept of a table and field owners in metadata; mechanism to derive owner from http request metadata; metadata for table description
We centralize the creation paths for test indexes, fields,
etcetera so they all have a common path, all using standard
test holders. There's still two versions, one for test.* functions
and one for internal. They do share a TestHolderConfig though.
Large hunks of the related APIs are simplified/streamlined.
* Fragments are always created with a Field and don't need
a workaround in case they don't have it.
* Creation of test fragments, etc., use optional FieldOptions
but don't specify names because they're all using new holders
for each thing created anyway. This dramatically reduces
the complexity of the calls.
* test fragments are created inside test views which are created
inside test fields, etcetera, so everything is using the same
logic; test views aren't bypassing the other layers, they're
creating themselves normally within a field.
* Quite a few things now use the standard runtime/production
logic instead of being custom workarounds; for instance, instead
of `mustOpenMutexFragment` creating a fragment and then creating
a mutex vector for it, we just create a mutex-typed field and
have the normal runtime code do this.
* Similarly, we now use the same field creation logic that production
does, instead of having our own test-only thing that validates
field names directly, so our test that we're validating field names
is actually testing the runtime code. Yay.
* fragSpec goes away. it was a replacement for fragProxy which existed
to solve memory allocation problems but replaced them with interface
overhead problems. Now we just have pointers to things and maintain
valid data structures.
* Many panics are now Fatal or Fatalf calls.
* Some specific bugs fixed, like a cluster which was requested and
then had its first node directly overwritten, which isn't valid with
shared clusters.
* Drop the temp-dir test flag and TempDir variable, we can just use
$TMPDIR.
* Drop a benchmark of "write file to disk" that was purely a benchmark
of file write speed, not a benchmark of rendering the data that needs
to be written.
* Drop the unused "flags" parameter to fragment creation, which was
only used back when we changed the BSI format.
* Use holder.Txf() rather than index.Txf(). The TxFactory has to be
holder-level anyway, referring to it via the index is misleading.
* Test holders automatically close themselves and delete themselves,
we remove various other things that thought they were responsible
for deleting themselves.
* squashed 45 commits into one :)
* tlt/sql experiment (#2035)
* Move PlanOperator to sql3/planner/types package
includes:
type PlanOperatorColumn struct
type PlanOperator interface
* Remove planner dependencies from pilosa package
The goal after this is to prevent the planner package (which doesn't exist yet)
from being imported by the pilosa package; we just want it injected into the server
in server/server.go. This is because the planner package uses pilosa types, so we need
to avoid circular dependencies.
Added ExecutionPlannerFn
Make public: pilosa.ExecOptions
Added a pilosa.Executor interface
Added a planner.types.CompilePlanner interface
Isolated the planner calls to:
- Executor.Execute()
- *API.[method]()
* Move executionplanner files into the sql3/planner package.
This required a bit of gymnastics, and there are some things around
FieldOptions which need to be addressed soon.
* Remove the hacky FieldOptions stuff I added earlier
This implementation just uses the pilosa.FieldOption functional options
provided by the API (as opposed to trying to build a FieldOptions
object.
It also changes field types to constants. These are private for now, but
if we need to make them public, we should put them in the planner/types
package.
* Implement the "scale" value from Decimal(scale)
Also, precision and scale were currently reversed in the parser. This
fixes that.
* Modify the parser to handle CACHETYPE <type> SIZE <size>
It's a little odd to me that the cache type values are Tokens, but I
guess it's ok. One thing to keep in mind is that FeatureBase expects
lowercase values, so this commit changes the parser to set the value to
the lowercase version of the type.
* Fix the /sql2 tests
This entailed a combination of commenting out or t.Skip()-ing tests
which covered code in the parser that has been commented out or removed
as not currently supported in sql3.
It also adds some coverage for the sql.Contraint stringers.
* Prevent JSON sql results from containing closing commas
This commit just re-works the existing output code to avoid inserting
closing commas (which results in invalid JSON).
* Enhance the CREATE TABLE test coverage.
In particular, ensure that the fields which get created in FeatureBase
are what we expect based on the fields defined in the CREATE TABLE
statement.
This also ensures that the TIMEQUANTUM and CACHETYPE contraints are not
provided for the same field (since those constraints are not supported
together).
* Adjust the EBNF file to indicate SIZE contraint is optional
A CACHETYPE can be provided without a SIZE. This change indicates that
SIZE is optional.
* Remove `executionplanner_` from file names (#2040)
* implementation of ALTER TABLE (sans column RENAME)
* refactored expression analysis; added more robust type checking; all unary and bin ops function on ints
* added type support for expressions; full bin/unary op support; added cast; more literal support
* cast int to all other types
* all literals (except idset, stringset & timestamp) make it thru; cast to all types with int as source now works
* implemented LIKE/NOT LIKE
* Implemented IS [NOT] NULL
* Move sql2 files into sql3/parser package (#2045)
* Move sql2 files into sql3/parser package
This also removes the sql2 package.
* Fix tests which were typing _id fields as INT intead of ID
* implemented BETWEEN, NOT BETWEEN
* Add featurebase/error package (#2046)
* Add featurebase/error package
I copied the `dax/errors` package which I am starting to use in the DAX
prototype into `featurebase/errors` in order to start using it with the
sql3 package. It's basically a wrapper around `github.com/pkg/errors`,
but it uses a customer coded error.
The sql package can define its own errors based on the
`featurebase/errors` types. Then do things like `Wrap()` and `Is()`.
* Address the linter complaints: shadowed variables, unreachable code
* implemented IN & NOT IN with expression lists
* first cut of CASE
* Fixed some errors from rebase
* updated bnf; removed unused code; tightened up error handling
* first crack at basic CLI for SQL3
Use: `featurebase cli`
Still lots to do here, but for example:
> select count(*) from tremor
+--------------+
| COUNT |
+--------------+
| 1.158321e+06 |
+--------------+
* Iterate on the CLI (#2057)
Handle the errors.
Add an "exit" command.
Add some general formatting and white space.
Add termination character: ";" (semicolon)
This commit allows a user to provide multiple or partial SQL statements.
Example of multiple statements:
```
show tables; select * from foo;
```
Example of partial (multi-line) statements:
```
select *
from foo;
```
Don't uppercase the header values
* error refactoring; first cut of TOP; remove unused code; use log.Printf instead of fmt.Printf
* fixed a bug with QualifiedRef from refactoring; added bones of INSERT; removal of unused code; tightened up errors more; fixed failing tests
* single value list for INSERT
* Update bnf per discussion with Travis; INSERT now doing the requisite stuff
* Pat's eyes went square - nothing wrong with TOP, Pat needed to learn arrays again.
* improved some errors; fixed tests to suit
* send warnings back in the api; update CLI to display warnings
* start warning on stuff not implemented so we don't get bugged about it
* Tlt/sql experiment (#2063)
* Expresssion -> Expression
* Add SQL planner test
- adds a test to which it is easier to add tables and SQL statments
- un-exports all of the expression types
- removes the planner pointer from the expression types (it can be added
back later if need be)
* Fix where clause on a string field
Prior to this commit, the binary expression for a where clause on a
string field was building the call by providing a range operator which
is typically used for BSI fields. This changes it to use the call.Args
for string values.
* Update planner tests to handle multiple sql for the same results
* Reorganize SQL tests
Introduce a test/helpers package and move shared MustQueryRows into that
package.
* Add a compatibility map for field types. (#2064)
This is primarily to address the fact that ID fields were previously
incompatible with INT literals.
We should probably consider introducing a custom type for FieldType
which can be used to define compatibilities.
* significantly refactored type checking
* Handle nil (NULL) values in the sql CLI. (#2067)
go-pretty panics if the interface{} field value is nil. This replaces
nil values with a "NULL" string.
* Squash some commits
fixed a still failing test
added line, col to all error messages
refactored source handling to enable table aliases
fixed some copypasta per review
warnings for order by & topn; implemented select as a source
starting to handle in (select...); added stub for optimizer
JSON-encode the sql error and warning strings (#2069)
Error strings with unencoded characters (like double quotes) were
resulting in invalid json.
got insert working; added symbol table; added concrete optimizer; added nascent NestedLoopsOperator; rewrite "where foo in (select..." as inner join
* all about the sets (#2085)
* implemented setcontains()
* implemented set literal; insert set column values; setcontains/all/any both in expr eval and pql filters
* Convert test to use latest framework. (#2086)
* fixed some comments
* removed refactored tests
Co-authored-by: Travis Turner <travis@pilosa.com>
* Add support for Decimal fields to the sql test. (#2090)
* dates (#2094)
* return dates as strings in output; tightened up decimal type checking
* return dates as strings in output; tightened up decimal type checking
* fixed failing tests after decimal changes
* can now insert decimal values
* implemented insert for timestamp data type; implemented current_date, current_timestamp constants
* fixed some failing tests
* handle date literals from strings in insert statements
* changes from feedback
* Fix pointer method error
* sql3 API interface (#2110)
* Introduce API-related interfaces: SchemaAPI, ComputeAPI
The sql3 code was relying on the pointer: *pilosa.API in order to call
API methods directly on the local node. If we want to import and use the
sql3 package in another service (the DAX queryer, for example), we need
to be able to use an implementation of an interface for those API method
calls.
This commit introduces two interfaces, both automatically implemented by
pilosa.API:
- SchemaAPI
- ComputeAPI
* Convert sql3 code to use IndexInfo instead of Index
The sql3 code was relying on a *pilosa.Index and its methods to get
general information like index and field name, type, etc. This commit
converts everything to use a *pilosa.IndexInfo instead.
This allows us to modify the SchemaAPI interface to also return
IndexInfo instead of Index, which will be a lot easier to implement in a
non-pilosa package (like DAX); creating a *pilosa.Index requires
providing things like data directory paths and holders, which are not
necessary for these use cases.
* Unary and Binary Ops R US plus CAST (#2111)
* implemented string literal for timestamp epoch
* fixed failing test
* fixed the failing test again
* refactored tests; implemented unary op tests for all datatypes; implemented binop tests for int/int, int/id, int/decimal & ID/int
* implemented all binary ops for INT & all other types, ID & all other types
* implemented binary ops for DECIMAL types & all other types
* added STRING & BOOL to various tests; implemented all remaining binOp tests
* fix up some stuff after rebasing
* refactored test defs into multiple files; implemented CAST for every datatype
* added tests for like/not like
* addressed review feedback
* addressed type review feedback
* tightened up IS [NOT] NULL behavior plus tests (#2118)
* tightened up IS [NOT] NULL behavior plus tests
* BETWEEN/NOT BETWEEN with all data types
* addressed review feedback
* Handle negative integers in column min/max constraints (#2120)
This commit parses the min/max contraint as an expression, as opposed to
an int literal, so that negative values are treated as Unary
expressions.
There currently isn't support for min/max constraints on `decimal`
fiels, so for now this change only expects +/- integer values.
* Implement the CREATE TABLE keypartitions logic (#2123)
* Execution time, IN/NOT IN & multiple aggregates (#2124)
* added display of execution time
* IN/NOT IN tests for all data types
* fixed date parsing
* removed duplicative tests
* refactoring aggregates
* suport multiple aggregates
* Address review feedback
* final round of feedback
* Add method SchemaAPI.CreateIndexAndFields() (#2127)
In order to support a CREATE TABLE statement as a single command, this
commit alters the SchemaAPI interface to contain a single method which
handles both the index and its fields. It also updates the sql3 code to
use this interface instead of CreateIndex() and CreateField()
indepedently.
* Symbol Handling (Again) (#2129)
* Refactored symbol handling in the planner; re-instated the select as source tests
* removed commented out code
* addressing review feedback
* Move hard-coded _id field out of planner and into interface implementation (#2130)
This commit moves the hard-coded addition of the `_id` field from the
planner to the SchemaAPI.IndexInfo() implementation method.
NOTE: If anything was expecting SchemaAPI.Schema() to also return the
`_id` field as part of its field list in each table, then it would not
be there because the `_id` field is only added in the IndexInfo() method
for now. Currently that's not a problem because nothing is expecting the
`_id` field for `Schema()`.
* Multiple aggregates, all aggregates stand alone and in GROUP BY (#2132)
* handle multiple aggregates in group by queries
* added handling for avg() aggregate both stand alone and in group by
* tightened up sum & avg outside of group by
* added min, max & percentile
* added warnings
* Make MaterializedRowSet implement the PlanOperator interface. (#2133)
This commit refactors the PQLMultiGroupByOperator to have a PlanOperator
as its output. Then, when it initializes, it sets up a
MaterializedRowSet and populates that with the values from the multiple
group by operations.
* added explicit min/max pql operators
* saved a file I forgot to save
* per review
* Un-indent some if/else nesting (#2136)
Co-authored-by: Travis Turner <travis@pilosa.com>
* Add optional `name` argument to test structs.
This commit adds the `name` argument to `tableTest` and `sqlTest` so
that a test can be optionally named. This allows a developer to more
easily run/identify a particular test by name.
* Inbuilt functions (redux) (#2141)
* set functions type parameter type checking
* implemented datepart
* include SQL3 type in SHOW COLUMNS output
* fixed select as source; failing SHOW COLUMNS test
* select in select list
* dump output columns; handle optimization for select list subqueries
* make it an error to return multiple rows for a select list subquery
* added description
* contants and test coverage for datepart function
* SQL3 Refactor-palooza (#2182)
* removed unneeded IsAggregate()
* first cut of working nested loops operator aka INNER JOIN
* remove selectListItemPlanExpression
* added some warnings
* all the tests are passing again!
* addressed some linter complaints
* added basic order by
* bug fixes; added 'or replace'/'replace' to insert
* for insert references should return appropriately
* added back ability to use subquery singleton expressions
* removed dead code; fixed test
* json-able plan, Schema() plus refactoring
* fixed dumb code
* add some tests for time quantum behavior
* Code cleanup during review. Also fixed INSERT to keyed table bug.
This commit contains a lot of minor adjustments made during code review.
It also contains a bug fix that was preventing INSERT into a keyed table
(i.e. _id type STRING) from working.
Co-authored-by: Travis Turner <travis@molecula.com>
* Fix expected min/max on timestamp column test (decimal field)
I don't know why this changed, but presumably something to do with
decimal related work that happened on master.
* Fix compile problem after rebase
* review feedback
Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
Co-authored-by: Travis Turner <travis@pilosa.com>
Co-authored-by: Travis Turner <travis@molecula.com>
Co-authored-by: Fletcher Haynes <fletcher@capitalprawn.com>
This way we can't get into an inconsistent state.
Consider this example:
You have a 3-node cluster, nodes A, B and C.
You create an index "blah" while all three nodes are up.
Nodes B and C go down.
You attempt to delete the index. It is removed from node A's holder, but is not removed from nodes B and C.
When nodes B and C are restarted, the schema still shows this "blah" index.
If you attempt to delete the index from node A, you receive an index not found error, but the schema indicates the index exists.
With this change however, when you first attempt to delete the index, it is not removed from the holder until there is enough nodes up to achieve consensus.
The same situation applies to fields and views.
* Revert "make pql.Decimal.Value a private big.Int field"
This reverts commit eab6174388.
* Revert "pql.Decimal for DecimalVal in ValCount&GroupCount"
This reverts commit a0c9eec410.
* Revert "Add AddDecimal support to pql"
This reverts commit 50787fd37a.
This ensures that we can't overflow when adding `pql.Decimal`s together. The
only place we can possibly overflow is when converting pql.Decimal to an Int64,
but that is a risk we have to take. Also, the only place we do this is in our
ToRowser. We could maybe change that to strings, so the presentation of data
doesn't indicate an overflow, but that is a later decision to make. It will
also involve fixing the generate-proto-grpc make command, because that's broken
rn.
the ranked cache must be fully rebuilt as the BitmapRewriter doesn't
have an easy way to track which rows had how many bits changed which
we would need to update the cache.
We also needed to add a Clear method to topn cache to completely
remove old values before the rebuild as otherwise they'd sit there and
pollute the cache after it was rebuilt.
This also includes fixing a strange idiosyncrasy where the _exists
field was a set field, but didn't have its type explicitly set. This
was causing it to have a ranked cache even though that option was
turned off. Hoping this doesn't have any weird follow-on effects... or
if it does the tests catch them.
* FB-1403 - fixed TTL to not allowed negative values
* FB-1403 - TTL - fixed comment
* FB-1403 - TTL - added comments for unit test, removed extra if statement, removed extra unit test
Prior to this, we locked the index-wide i.mu around writes to i.fields.
But now the entire open process is holding that lock, so we couldn't
lock it here (that'd deadlock) and didn't think we needed to (because
it was held). But in fact that means that multiple fields being opened
at once can concurrently write to the map. Conveniently, we *already*
pass a shared mutex into openField() to prevent exactly this sort of
problem; we just need to actually use it when doing the write.
This is a lot more complex than it sounds like it will be.
We shut down the cache flush when a holder is closed, but if you're
deleting an index, we don't check for that, and can have a cache flush
still creating cache files in an index which could conceivably result
in os.RemoteAll() failing. This shouldn't happen often, but it's happened
at least once.
To address this, first, we make sure that every tier of this operation
bails as quickly as it can after the thing it's working on closes. Second,
we retry RemoveAll.
Unfortunately, some things get reopened, so we have to handle that,
have mutexes covering the access to the channel, and so on. Also, some
things were getting double-closed, which was previously harmless but
could now fail. So, first, catch all the existing double-closes and
remove them, second, make the double-close fail with an error. Note
that virtually none of the tests check for errors on close.
This passes tests and should be unable to hit the original problem.
Unfortunately, it's unreasonably hard to check that, because it
requires an incredible coincidence of timing on the delete aligning
with a cache flush.
Two CreateField messages reaching different nodes in a cluster at the same
time could cause a deadlock because each CreateField runs with a write lock
held, then issues requests to other nodes which, at a minimum, need
a read lock and which may require a write lock. Reorder things a bit to
make the broadcast to other nodes happen outside the lock. We may also
need to do something to have nodes handle the case where something's been
created in etcd but they haven't gotten the message about it yet.
i used this script, a little clunky but it got the job done
```bash
for file in `find . -type f -print | grep '\.go'`; do
sed '1,/^\/\/ limitations under the License.$/d' $file > $file.tmp;
result=`cat $file.tmp`
if [[ result != "" ]]; then
gofmt $file.tmp &> /dev/null;
if [[ $? == 0 ]]; then
mv $file.tmp $file && gofmt -w $file;
else
rm $file.tmp;
fi
else
rm $file.tmp;
fi
done
```
These commits are hard to disentagle, and doing them separately means
re-modifying the same chunks of code several times before removing it,
and similar things.
Basically:
(1) Drop the bolt backend storage.
(2) Drop the blue-green wrapper that compares two backends.
(3) Drop unused or barely-used Tx API components from all the
remaining backends.
(4) Minor related cleanup to simplify things related to these.
The boltdb backend existed only to verify RBF. The blue-green wrapper
was mostly used to verify RBF, but in practice we had to do a lot
of working around that, and it introduced a lot of special cases.
Types removed:
IteratorFinder: Used only to implement the roaring iterator
on top of boltdb, and to complicate the way it worked in roaring.
Reverted the complications. Also unexport NewSliceContainers
which is used only for that outside of roaring's internals.
PortMapper from cluster_internal_test.go: Used only for a test
we removed early this year. Never used for anything else.
RawRoaringData: Totally unused.
TxStore: Totally unused.
Functions removed from Tx API, and sometimes corresponding
members were removed from structs:
* Dump: debugging code, I don't think I found any actually reachable
paths to it.
* Group: only used for debugging TxGroup stuff
* IncrementOpN: only used by fragment, fragment can increment its
own opN.
* Options: unused?
* Pointer: debugging only
* Readonly: used only to decide how to handle Tx in a TxGrp,
but we never add a non-readonly Tx to a TxGrp. Removed also all
the corresponding write-aware stuff.
* RoaringBitmapReader: Used exactly once, can just be a bm.WriteTo.
* Sn (and OpenSnList): Unused
* UnionInPlace: unused and conceptually-invalid; it didn't write
to storage and shouldn't have, and was just "create a bitmap
then call union-in-place", which we can do directly.
* UseRowCache: just checked storage.UseRowCache.
Other things removed:
The SetRequiredForAtomicWriteTx and ClearRequiredForAtomicWriteTx
functions go away, since nothing now seems to be using them? Same
for holder_internal_test's `testHasBit` and `testMustNotHaveBit`,
which were unused.
The DBPerShard "DeleteDBPath" and "HasData" functions and related
parts were mostly unused; took out the parts that were never
actually being reached.
Changed the API of one function to simplify special cases and
remove things:
* ImportRoaringBits had a special "data" argument which gave it
subtly different semantics for RBF and roaring (for roaring, it
could produce a roaring bitmap *with ops log*), didn't seem to
be adding much. Removed corresponding "readStorageFromArchive"
which is not otherwise used.
Also took out various debugging/dumping functions that were unused
and may have bitrotted.
Dropped a test from txfactory_internal_test, and the "pjobs"
code, because those two were the only things that needed Barrier
and thus idem, which lets us drop two more dependencies. We already
have errgroup for grouping things which want to terminate as
soon as one of them errors, approximately. To do better we'd have
to have context-threading, really.
Unbroke the WriteFragment test for non-roaring tests and made it
not roaring-only.
Performance of tests on MacOS has been atrocious for a while, and
a lot of that is fsync, so we're trying to make that optional.
To test all of this, I modified RBF to panic if anything tried to
open an RBF database without disabling fsync, and ran the tests that
way, and tracked down the various places this could still happen.
There's a lot of places in our tree where we were creating
test holders which were not getting created with fsync disabled, which
results in a surprisingly large number of points at which we end
up calling fsync in tests, which makes tests much slower than they
need to be. There's also a bunch of places where the flags don't get
propagated correctly; for instance, storage.fsync didn't propagate
to the RBFConfig.
We add an "fsync enabled" flag to OpenTranslateStoreFunc, so we can
tell translation stores that we don't need syncing, so the server's
config can be passed on appropriately.
More of the test code that sets things up is correctly configuring
that flag by default.
We also change the barely-used bolt storage backend to support this as
well.
With this done, the only calls to fsync left in a run of `go test -short`
in the top-level directory are from the zap logger in etcd, and consumed
around 0.03 seconds. The overall impact is that `go test -short`
went from "takes enough more than 10 minutes that i don't know how long
it takes" to about 2.5 minutes.
In fact, we have a number of things assuming that values passed to Import
always fit within a single known shard, so, drop all the extra complexity
around this, drop the computation of fancy view/shard keys, and so on.
There's a lot of room left to improve this probably but it's at least
better, I think.
Unfortunately, there's a handful of things, basically all of which are
test cases, which were relying on this, so, we also add functionality
for splitting import requests by shards. But this allows us to stop
duplicating each shard's inputs one at a time... which turns out to
mean that we now care that the import operation can write back to the
import request. This only affects test cases, so we adopt a crufty
hack involving cloning import requests in those rare cases, and also
when reusing the same column IDs to write to the existence field that
we'd be using later to write to another field.
Note that even if we weren't overwriting the column IDs with positions,
we'd be sorting the column/row ID lists by row-then-column, which means
we'd still be corrupting the column ID lists. This may want to change
at some point.
We also reuse a single Tx for all the views, because DB-per-shard
means that should work fine, and reduces the cost of doing these
updates, probably.
We're reading timestamps as []int64, instead of allocating a time.Time
for each timestamp, just use the same logic to determine whether to use the
int64 timestamp that we would have used to decide whether to allocate it.
We still have to check the whole run, though, because we're providing a large
list of 0s instead of "no timestamps", for Reasons.
There's only ever one view in importValue, but there's also only ever
one shard, because importValue is only called by things called from
the API after it has split everything up by shard.
Attributes are unmaintained and unused.
They have become more of a liability than a benefit.
This change eliminates them from the codebase.
The only user-visible change (assuming that attrs are not used) is that the attrs field will no longer appear in row JSON.
Before this change, we were only caching the BitDepth on the
field.options. This was ok as long as applyOptions() was called after
that. But unfortunately, during startup, applyOptions() was called prior
to that being set. So with this commit, we explicitly set the value in
bsiGroup.BitDepth as well.
