mirror of
https://github.com/featurebasedb/featurebase.git
synced 2026-08-28 02:44:59 +00:00
14 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Pat Okeeffe
|
284f62dcb9
|
create model, create function... all the goodies (#2264)
* create function, create/drop model; re-introduced limit; added COPY; var(); corr() * review feedback |
||
|
Seebs
|
54dbeec1af |
support null/non-null tests for non-BSI fields
There's a lot going on here. First, we were treating "the test is a Condition" as implying BSI, which it doesn't anymore. Second, the behavior of conditions was weird and BSI-specific. Third, we had to propagate these changes and features throughout a bunch of code, including both the core featurebase code and the DAX replacements/copies of it, plus the SQL3 layer. We refactor this so that tests for equality and inequality work for non-BSI fields, so now if you accidentally use `==` in a Row call on a non-BSI field, it still works; that's not specific to BSI fields anymore. We add a TrackExistence flag to fields, and propagate it through things like our protobuf code, etcetera, so that we can successfully create fields. Newly-created fields get this by default, because we add it unconditionally to them, but the paths that are being called with existing fields don't add it. So, when we "create" (really, just load the definition of) a field from something stored in the schema, we don't add TrackExistence to it, but any path to creating a new field should. A time quantum field with NoStandardView will *effectively* lack TrackExistence. For sets, mutexes, and time quantums with a standard view, anything that sets bits will also set a corresponding bit for the record in a new "existence" view. This allows us to distinguish between an empty set and a null, and also allows null checks to be constant-time. When clearing bits, we don't clear existence bits EXCEPT that if you clear a bit in a mutex, *and the bit actually existed*, we clear the existence bit. For sets and time quantums, clearing bits never clears the existence bit. Deleting records clears the existence bit. We also add code to the `batch` subpackage to generate suitable existence field bitmaps and import them. This logic correctly handles empty sets and nils. The `batch` package does not allow specification of anything equivalent to clearing a single bit from an existing record, so we don't have to deal with the mutex complexity in that case, which is good because it would be impossible. This requires a number of other subtle changes, such as allowing new fields to have more than one FieldOption specified for them. We also drop the handful of implementation bits relating to the "fullySorted" internal-use-only import flag, which existed only to support the JSON ingest API, which we've removed. The most dangerous part of this is that the mutex semantics are impossible to implement on top of our existing API, because they require us to know, not how *many* bits we cleared, but which *specific* bits we cleared. I've implemented this as a new Tx method, which is almost certainly going to be tech debt one day; if we some day drop the Import API, we should remove that. The testing for this is only currently covering the Set/Clear behavior of PQL, and the Import API. The batch tests haven't been written yet. Fields that don't have existence tracking enabled refuse to perform null/not-null tests. They should also report themselves as having no null values -- if a record exists, sets in it are considered empty rather than null. The SQL3 support requires a number of subtle modifications to both featurebase and some addon tooling. The essential thing is dropping the unconditional translation of nil slices to non-nil empty slices in translateResult, both in the executor and the orchestrator. We also modify the logic that handles generating results from Extract calls, to ensure that non-null sets get an empty slice created for them even if they never have any values assigned. The expected results for some tests are different now; we expect to get nil slices, rather than 0-length non-nil slices, for fields which were never written for a given record. Most tests were not changed. (In every case, if a test was failing, I actually checked the logic before changing expected results. This required a lot of tracking down of edge cases.) The batch package now rejects as an error attempts to clear single bits from mutex fields, because so far as I can tell it's simply impossible to have a roaring import that specifies the correct semantics there; you can't tell whether to clear an existence bit without access to the currently-set bits, which the batch API doesn't have. We already supported the special case of specifying a clear value of nil for clearing a mutex field; now that is the only allowed value for a mutex field to have in row.Clears. We change the logic for fixing up incoming view names (in two places) to stop assuming that any view in a time field other than "" that does not have viewStandard as a prefix is a partial time quantum name that should have "standard_" prepended to it. This allows us to submit bitmaps for "existence" to time quantum fields and not have them silently transformed into "standard_existence" because that's what we'd do with "202203". We drop the field ClearBits method, which was totally unused. We drop the sliceDifference function, which was used in a previous mutex implementation and hasn't been used in ages, and the test case for it, and the helper function used only by that test case. |
||
|
tgruben
|
bc07fb4a96
|
SQL3 Test Coverage: complete coerceValue coverage (#2343)
* complete coereceVal test coverage * sql between test * optimized between operator * basic operator test |
||
|
Pat Okeeffe
|
9386fc75b2
|
implement select from time quantum columns (fb-1654) (#2282)
* first time quantum queries working * implement select from timequantum columns * skip a dax test * make linter happy * addressed review feedback * reverted over eager test elimination |
||
|
Pat Okeeffe
|
74885c9718
|
make count(*) not depend on _id (#2258) | ||
|
Pat Okeeffe
|
5c74b64722
|
A bug fix roundup (#2242)
* fb-1940 re-implemented some changes that got missed private-public * fb-1939 fixes to between + decimals * fb-1935 - avg() on and id type + fixed some tests * fb-1953 add min/max for string types * fb-1938 - remove internal_type column from show columns * fb-1964 - fix space_used in fb_cluster_nodes to be int * fb-1996 - make sure all Idents that are being used as object references to schema objects are lowercased * fixed failing test * added some missed changes * fb-1969 found another case issue with identifier used for column idents |
||
|
pokeeffe-molecula
|
a6c165dd57 |
Implement DELETE (fb 1557) (#2382)
* delete implementation with test coverage
* optimize IN expressions; stop linter complaining
* fixed some uncovered query cases
* skip test in DAX for now
(cherry picked from commit
|
||
|
pokeeffe-molecula
|
c304991e9f |
implemented extract ddl; tightened up type related stuff (#2329)
* implemented extract ddl; tightened up type related stuff
* added some test coverage
* review feedback
(cherry picked from commit
|
||
|
pokeeffe-molecula
|
41e231504e |
handle filters on _id columns (fb-1765) (#2313)
* handle filters on _id columns using ConstRow
* handle keyed and un-keyed _id columns
* tests!
(cherry picked from commit
|
||
|
pokeeffe-molecula
|
e14cde7341 |
SQL BULK INSERT (fb-1749) (#2291)
SQL BULK INSERT
This change is to support a BULK INSERT/REPLACE statement that adds the ability to 1) take its input from a file, url or in-line blob 2) to map from the input source to the target columns
3) to transform data (using sql expressions) before inserting
4) support csv and ndjson formats
* improving test coverage
* increase test coverage again
* refactoring for handling transformation with types other than id and int
(cherry picked from commit
|
||
|
CLoZengineer
|
daceee2ab6
|
merge: featurebase merge updates for 2022-10-28 (#2188)
* use t.Fatal(f) to abort tests, not panic * make perf_able run at all, make it debug a bit better switch perf-able to using same node type we use for other spot instances, because otherwise it never finds any available capacity. we switch the perf-able script to use the standard get_value function instead of direct jq calls. we try to grab server logs if the restore fails in the hopes of finding out why the restore very occasionally fails. * Fix some issues with running IDK tests in docker. (#2248) *Stop running TestKafkaSourceIntegration with t.Parallel() This test can't be run in parallel as it's currently written. Doing so allows for interleaving of messages to the same kafka topic between tests. I didn't attempt to modify the test so it could be run in parallel. That could be done, but left for someone more ambitious. * Remove idk/testenv/certs which got accidentally committed. also update .gitignore to include those. * changes to add bool support in idk (#2240) * initial changes to add bool support in idk * modifying some default parameters for testing, will revert them later * adding support for bool in making fragments function * boolean values implementation without supporting empty or null values at this point * Implement bool support in batch using a map (and a slice for nulls) (#2247) * Implement bool support in batch using a map (and a slice for nulls) * Keep the PackBools default for now But set it explicity in the ingest tests which rely on it. * Modify batch to construct bool update like mutex The code in API.ImportRoaringShard has a switch statement which causes bool fields to be handled like mutex fields. This means, that the viewUpdate.Clear value should only contain data in the first "row" of the fragment, which it will treat as records to clear for *all* rows. This makes more sense for mutex fields; for bool fields, there's only one other row to clear. But since the code is currently handling them the same, we need to construct viewUpdate.Clear such that it conforms to that pattern. This commit also adds a test which covers this logic. * Remove commented code; revert config for testing This commit also removes the DELETE_SENTINEL case for non-packed bools, since that isn't supported anyway. * Revert default setting * remove inconsistent type scope * correcting the logic of string converstion to bool * resolving an error in a test * adding tests to cover code related to bool support in batch.go file and interface.go files * modifying interfaces test * added one more test case Co-authored-by: Travis Turner <travis@pilosa.com> Co-authored-by: Travis Turner <travis@molecula.com> * resolving bool null field ingestion error (#2254) * resolving bool null field ingestion error * testing issues * adding null support for bools * updating the null bool field ingestion * trying to resolve issue when ingesting null value for bool type * adding a clearing support for bool type * resolving issues with bool null value ingestion * updating the jwt go package version and removing changes made in docker compose file * reverting jwt go version * removing v4 of jwt * adding a comment in test file to see if sonar cloud accepts this file * don't obtain stack traces on rbf.Tx creation We thought stack traces were mildly expensive. We were very wrong. Due to a complicated issue in the Go runtime, simultaneous requests for stack traces end up contending on a lock even when they're not actually contending on any resources. I've filed a ticket in the Go issue tracker for this: https://github.com/golang/go/issues/56400 In the mean time: Under some workloads, we were seeing 85% of all CPU time go into the stack backtraces, of which 81% went into the contention on those locks. But even if you take away the contention, that leaves us with 4/19 of all CPU time in our code going into building those stack backtraces. That's a lot of overhead for a feature we virtually never use. We might consider adding a backtrace functionality here, possibly using `runtime.Callers` which is much lower overhead, and allows us to generate a backtrace on demand (no argument values available, but then, we never read those because they're unformatted hex values), but I don't think it's actually very informative to know what the stack traces were of the Tx; they don't necessarily reflect the current state of any ongoing use of the Tx, so we can't necessarily correlate them to goroutine stack dumps, and so on. * fb-1729 Enriched Table Metadata (#2255) enriched metadata for tables added support for the concept of a table and field owners in metadata; mechanism to derive owner from http request metadata; metadata for table description * tightened up is/is not null filter expressions (FB-1741) (#2260) Covers tightening up handling filter expressions that contain is/is not null ops. These filters may have to be translated into PQL calls to be passed to the executor and even though sql3 language supports nullability for any data type, currently only BSI fields are nullable at the storage engine level (there is a ticket to add support for non-BSI field here FB-1689: IS SQL Argument returns incorrect error) so when these fields are used in filter conditions we need to handle BSI and non-BSI fields differently. * added a test to cover the keyword replace as being synonymous with insert (#2261) * update molecula references to featurebase (#2262) Co-authored-by: Seebs <seebs@molecula.com> Co-authored-by: Travis Turner <travis@pilosa.com> Co-authored-by: Pranitha-malae <56414132+Pranitha-malae@users.noreply.github.com> Co-authored-by: Travis Turner <travis@molecula.com> Co-authored-by: pokeeffe-molecula <85502298+pokeeffe-molecula@users.noreply.github.com> Co-authored-by: Stephanie Yang <stephanie@pilosa.com> |
||
|
pokeeffe-molecula
|
15382a2863 |
sql3 changes (#2211)
* first cut of working (slowly) bulk insert; table valued functions and a tuple data type to support time quantums * oversight * filter pushdown implementation; bulk insert * addressed some linter issues |
||
|
Seebs
|
e47bdb7889 | cleaned up sync from private repo | ||
|
pokeeffe-molecula
|
204558b46f |
added /sql endpoint; implemented SHOW TABLES (#1935)
* squashed 45 commits into one :) * tlt/sql experiment (#2035) * Move PlanOperator to sql3/planner/types package includes: type PlanOperatorColumn struct type PlanOperator interface * Remove planner dependencies from pilosa package The goal after this is to prevent the planner package (which doesn't exist yet) from being imported by the pilosa package; we just want it injected into the server in server/server.go. This is because the planner package uses pilosa types, so we need to avoid circular dependencies. Added ExecutionPlannerFn Make public: pilosa.ExecOptions Added a pilosa.Executor interface Added a planner.types.CompilePlanner interface Isolated the planner calls to: - Executor.Execute() - *API.[method]() * Move executionplanner files into the sql3/planner package. This required a bit of gymnastics, and there are some things around FieldOptions which need to be addressed soon. * Remove the hacky FieldOptions stuff I added earlier This implementation just uses the pilosa.FieldOption functional options provided by the API (as opposed to trying to build a FieldOptions object. It also changes field types to constants. These are private for now, but if we need to make them public, we should put them in the planner/types package. * Implement the "scale" value from Decimal(scale) Also, precision and scale were currently reversed in the parser. This fixes that. * Modify the parser to handle CACHETYPE <type> SIZE <size> It's a little odd to me that the cache type values are Tokens, but I guess it's ok. One thing to keep in mind is that FeatureBase expects lowercase values, so this commit changes the parser to set the value to the lowercase version of the type. * Fix the /sql2 tests This entailed a combination of commenting out or t.Skip()-ing tests which covered code in the parser that has been commented out or removed as not currently supported in sql3. It also adds some coverage for the sql.Contraint stringers. * Prevent JSON sql results from containing closing commas This commit just re-works the existing output code to avoid inserting closing commas (which results in invalid JSON). * Enhance the CREATE TABLE test coverage. In particular, ensure that the fields which get created in FeatureBase are what we expect based on the fields defined in the CREATE TABLE statement. This also ensures that the TIMEQUANTUM and CACHETYPE contraints are not provided for the same field (since those constraints are not supported together). * Adjust the EBNF file to indicate SIZE contraint is optional A CACHETYPE can be provided without a SIZE. This change indicates that SIZE is optional. * Remove `executionplanner_` from file names (#2040) * implementation of ALTER TABLE (sans column RENAME) * refactored expression analysis; added more robust type checking; all unary and bin ops function on ints * added type support for expressions; full bin/unary op support; added cast; more literal support * cast int to all other types * all literals (except idset, stringset & timestamp) make it thru; cast to all types with int as source now works * implemented LIKE/NOT LIKE * Implemented IS [NOT] NULL * Move sql2 files into sql3/parser package (#2045) * Move sql2 files into sql3/parser package This also removes the sql2 package. * Fix tests which were typing _id fields as INT intead of ID * implemented BETWEEN, NOT BETWEEN * Add featurebase/error package (#2046) * Add featurebase/error package I copied the `dax/errors` package which I am starting to use in the DAX prototype into `featurebase/errors` in order to start using it with the sql3 package. It's basically a wrapper around `github.com/pkg/errors`, but it uses a customer coded error. The sql package can define its own errors based on the `featurebase/errors` types. Then do things like `Wrap()` and `Is()`. * Address the linter complaints: shadowed variables, unreachable code * implemented IN & NOT IN with expression lists * first cut of CASE * Fixed some errors from rebase * updated bnf; removed unused code; tightened up error handling * first crack at basic CLI for SQL3 Use: `featurebase cli` Still lots to do here, but for example: > select count(*) from tremor +--------------+ | COUNT | +--------------+ | 1.158321e+06 | +--------------+ * Iterate on the CLI (#2057) Handle the errors. Add an "exit" command. Add some general formatting and white space. Add termination character: ";" (semicolon) This commit allows a user to provide multiple or partial SQL statements. Example of multiple statements: ``` show tables; select * from foo; ``` Example of partial (multi-line) statements: ``` select * from foo; ``` Don't uppercase the header values * error refactoring; first cut of TOP; remove unused code; use log.Printf instead of fmt.Printf * fixed a bug with QualifiedRef from refactoring; added bones of INSERT; removal of unused code; tightened up errors more; fixed failing tests * single value list for INSERT * Update bnf per discussion with Travis; INSERT now doing the requisite stuff * Pat's eyes went square - nothing wrong with TOP, Pat needed to learn arrays again. * improved some errors; fixed tests to suit * send warnings back in the api; update CLI to display warnings * start warning on stuff not implemented so we don't get bugged about it * Tlt/sql experiment (#2063) * Expresssion -> Expression * Add SQL planner test - adds a test to which it is easier to add tables and SQL statments - un-exports all of the expression types - removes the planner pointer from the expression types (it can be added back later if need be) * Fix where clause on a string field Prior to this commit, the binary expression for a where clause on a string field was building the call by providing a range operator which is typically used for BSI fields. This changes it to use the call.Args for string values. * Update planner tests to handle multiple sql for the same results * Reorganize SQL tests Introduce a test/helpers package and move shared MustQueryRows into that package. * Add a compatibility map for field types. (#2064) This is primarily to address the fact that ID fields were previously incompatible with INT literals. We should probably consider introducing a custom type for FieldType which can be used to define compatibilities. * significantly refactored type checking * Handle nil (NULL) values in the sql CLI. (#2067) go-pretty panics if the interface{} field value is nil. This replaces nil values with a "NULL" string. * Squash some commits fixed a still failing test added line, col to all error messages refactored source handling to enable table aliases fixed some copypasta per review warnings for order by & topn; implemented select as a source starting to handle in (select...); added stub for optimizer JSON-encode the sql error and warning strings (#2069) Error strings with unencoded characters (like double quotes) were resulting in invalid json. got insert working; added symbol table; added concrete optimizer; added nascent NestedLoopsOperator; rewrite "where foo in (select..." as inner join * all about the sets (#2085) * implemented setcontains() * implemented set literal; insert set column values; setcontains/all/any both in expr eval and pql filters * Convert test to use latest framework. (#2086) * fixed some comments * removed refactored tests Co-authored-by: Travis Turner <travis@pilosa.com> * Add support for Decimal fields to the sql test. (#2090) * dates (#2094) * return dates as strings in output; tightened up decimal type checking * return dates as strings in output; tightened up decimal type checking * fixed failing tests after decimal changes * can now insert decimal values * implemented insert for timestamp data type; implemented current_date, current_timestamp constants * fixed some failing tests * handle date literals from strings in insert statements * changes from feedback * Fix pointer method error * sql3 API interface (#2110) * Introduce API-related interfaces: SchemaAPI, ComputeAPI The sql3 code was relying on the pointer: *pilosa.API in order to call API methods directly on the local node. If we want to import and use the sql3 package in another service (the DAX queryer, for example), we need to be able to use an implementation of an interface for those API method calls. This commit introduces two interfaces, both automatically implemented by pilosa.API: - SchemaAPI - ComputeAPI * Convert sql3 code to use IndexInfo instead of Index The sql3 code was relying on a *pilosa.Index and its methods to get general information like index and field name, type, etc. This commit converts everything to use a *pilosa.IndexInfo instead. This allows us to modify the SchemaAPI interface to also return IndexInfo instead of Index, which will be a lot easier to implement in a non-pilosa package (like DAX); creating a *pilosa.Index requires providing things like data directory paths and holders, which are not necessary for these use cases. * Unary and Binary Ops R US plus CAST (#2111) * implemented string literal for timestamp epoch * fixed failing test * fixed the failing test again * refactored tests; implemented unary op tests for all datatypes; implemented binop tests for int/int, int/id, int/decimal & ID/int * implemented all binary ops for INT & all other types, ID & all other types * implemented binary ops for DECIMAL types & all other types * added STRING & BOOL to various tests; implemented all remaining binOp tests * fix up some stuff after rebasing * refactored test defs into multiple files; implemented CAST for every datatype * added tests for like/not like * addressed review feedback * addressed type review feedback * tightened up IS [NOT] NULL behavior plus tests (#2118) * tightened up IS [NOT] NULL behavior plus tests * BETWEEN/NOT BETWEEN with all data types * addressed review feedback * Handle negative integers in column min/max constraints (#2120) This commit parses the min/max contraint as an expression, as opposed to an int literal, so that negative values are treated as Unary expressions. There currently isn't support for min/max constraints on `decimal` fiels, so for now this change only expects +/- integer values. * Implement the CREATE TABLE keypartitions logic (#2123) * Execution time, IN/NOT IN & multiple aggregates (#2124) * added display of execution time * IN/NOT IN tests for all data types * fixed date parsing * removed duplicative tests * refactoring aggregates * suport multiple aggregates * Address review feedback * final round of feedback * Add method SchemaAPI.CreateIndexAndFields() (#2127) In order to support a CREATE TABLE statement as a single command, this commit alters the SchemaAPI interface to contain a single method which handles both the index and its fields. It also updates the sql3 code to use this interface instead of CreateIndex() and CreateField() indepedently. * Symbol Handling (Again) (#2129) * Refactored symbol handling in the planner; re-instated the select as source tests * removed commented out code * addressing review feedback * Move hard-coded _id field out of planner and into interface implementation (#2130) This commit moves the hard-coded addition of the `_id` field from the planner to the SchemaAPI.IndexInfo() implementation method. NOTE: If anything was expecting SchemaAPI.Schema() to also return the `_id` field as part of its field list in each table, then it would not be there because the `_id` field is only added in the IndexInfo() method for now. Currently that's not a problem because nothing is expecting the `_id` field for `Schema()`. * Multiple aggregates, all aggregates stand alone and in GROUP BY (#2132) * handle multiple aggregates in group by queries * added handling for avg() aggregate both stand alone and in group by * tightened up sum & avg outside of group by * added min, max & percentile * added warnings * Make MaterializedRowSet implement the PlanOperator interface. (#2133) This commit refactors the PQLMultiGroupByOperator to have a PlanOperator as its output. Then, when it initializes, it sets up a MaterializedRowSet and populates that with the values from the multiple group by operations. * added explicit min/max pql operators * saved a file I forgot to save * per review * Un-indent some if/else nesting (#2136) Co-authored-by: Travis Turner <travis@pilosa.com> * Add optional `name` argument to test structs. This commit adds the `name` argument to `tableTest` and `sqlTest` so that a test can be optionally named. This allows a developer to more easily run/identify a particular test by name. * Inbuilt functions (redux) (#2141) * set functions type parameter type checking * implemented datepart * include SQL3 type in SHOW COLUMNS output * fixed select as source; failing SHOW COLUMNS test * select in select list * dump output columns; handle optimization for select list subqueries * make it an error to return multiple rows for a select list subquery * added description * contants and test coverage for datepart function * SQL3 Refactor-palooza (#2182) * removed unneeded IsAggregate() * first cut of working nested loops operator aka INNER JOIN * remove selectListItemPlanExpression * added some warnings * all the tests are passing again! * addressed some linter complaints * added basic order by * bug fixes; added 'or replace'/'replace' to insert * for insert references should return appropriately * added back ability to use subquery singleton expressions * removed dead code; fixed test * json-able plan, Schema() plus refactoring * fixed dumb code * add some tests for time quantum behavior * Code cleanup during review. Also fixed INSERT to keyed table bug. This commit contains a lot of minor adjustments made during code review. It also contains a bug fix that was preventing INSERT into a keyed table (i.e. _id type STRING) from working. Co-authored-by: Travis Turner <travis@molecula.com> * Fix expected min/max on timestamp column test (decimal field) I don't know why this changed, but presumably something to do with decimal related work that happened on master. * Fix compile problem after rebase * review feedback Co-authored-by: Matthew Jaffee <jaffee@pilosa.com> Co-authored-by: Travis Turner <travis@pilosa.com> Co-authored-by: Travis Turner <travis@molecula.com> Co-authored-by: Fletcher Haynes <fletcher@capitalprawn.com> |