Commit graph

34 commits

Author SHA1 Message Date
Seebs
f4905891d4 unbreak nested joins
It turns out that the problem with nested joins was that we were
trying to cleverly invert them, but that seems to be incorrect and
resulted in incorrect nesting.

The test case for this is
	SELECT * FROM X INNER JOIN Y ON true INNER JOIN Z ON false
this is now parsed as
	(X inner join Y on true) inner join z on false

Which, as it turns out, is the structure that stringizes back to the
original statement.

We were previously parsing it as
	X inner join (y inner join z on false) on true
which stringizes out to a different form, and is also, I think,
just straightforwardly not what we want.

So basically, we had special case code to recognize that we
were doing a join on top of another join, and invert them in
some way, and I have no idea why because that seems not to be
correct, or at least, it produces nonsensical stringizing that
we can't then parse.
2023-03-27 16:56:29 -05:00
Seebs
3f7ae75e17 increase parser test coverage significantly
We now test the tuple-assignment at all, although it's
perhaps confusing because we expect a ()-list of columns
to go with a {}-list of values. We also test a lot more
errors and some more successes, and additional literal types
in mustParseLiteral.
2023-03-27 16:56:29 -05:00
Seebs
d4fb807664 drop unused isHex and IsInteger functions 2023-03-27 16:56:29 -05:00
Vengata Krishnan
2c3be9d1e8
Fix failing selects on views defined with date literals (#2313)
* Fix failing selects on views defined with date literals
* System variables implementation.
2023-03-17 12:24:00 -04:00
Travis Turner
aa17b8d725
Enable linter: stylecheck (#2317)
* Enable linter: stylecheck

This enabled the stylecheck linter, but excludes some staticchecks for
now. The following are ignored because they will take a bit of time to
address, but the intention is to address them and remove them from the
exclusion list.

ST1000: at least one file in a package should have a package comment
ST1003: golang naming standards
ST1008: error should be returned as the last argument
ST1016: methods on the same type should have the same receiver name
ST1020: comment on exported function

* Address ST1015

For some reason this failed in CI but not locally. I can't figure out
why that check isn't happening locally. This just moves the switch
statements around so that the `default` is the first (or last) item.

* Adjust error string in test to match case-adjusted error

* Remove TestCloseTimeout
2023-03-14 08:45:18 -05:00
Pat Okeeffe
9386fc75b2
implement select from time quantum columns (fb-1654) (#2282)
* first time quantum queries working

* implement select from timequantum columns

* skip a dax test

* make linter happy

* addressed review feedback

* reverted over eager test elimination
2023-03-02 00:06:58 -06:00
seebs
b7e9879526
forward-port tests from SQL1 tree (#2261)
This forward-ports a number of tests from the previous SQL
implementation. The porting is approximate in a number of ways,
and not all tests are implemented/tested yet.

In particular, several tests are currently disabled because
we don't support `limit n` constructs.

The tests that were primarily tests of the parser have been
brought forward as parser tests. One of them has been altered
to add parentheses, because our parser interprets
	fld1 between 1 and 3 and fld2 = 2
as:
	fld1 between (1 and 3) and (fld2 = 2)
which is invalid, while the old parser apparently interpreted it
as:
	(fld1 between 1 and 3) and (fld2 = 2)

We have not yet verified the SQL spec's requirements here, but
sqlite agrees with our old parser, not our new parser, so this
may be a regression.

The old tests expected an INNER JOIN to suppress duplicate
values. Our new code does not, which is consistent with other
SQL implementations. This is a change, but the old behavior
appears to have been wrong. (You can still suppress duplicate
values by specifying DISTINCT.)

In the previous implementations, a value like `count(*)` had
`count(*)` as its column name. In the new implementation,
it has an empty string as its column name.

Related to this, the prior implementation allowed you to
write
	select age, count(*) from grouper group by age having count > 1
but the new implementationt requires that to be spelled as
	having count(*) > 1

This is consistent with other SQL implementations, so I think
the new behavior is correct.

The behavior of SHOW COLUMNS and SHOW TABLES has changed, in
that the specific results returned are significantly different.
Perhaps more significantly, the old system spelled the former
query as SHOW FIELDS, rather than SHOW COLUMNS. This may be
considered a regression, in that `SHOW FIELDS` no longer works,
and we should consider whether any hypothetical users might
have been relying on the output of either of these. (I hope
not, the new output is much better.)

Some of the old tests (the ones in handler_test) were accommodated
by adding a couple of specific test cases to existing tests,
specifically:
	* handling timestamp values with `Z` rather than `+00:00`
	* a join with a WHERE clause referring to fields in both
	  source tables

We introduce a new "partial" comparison type, because there's
no way for a test of `SHOW TABLES` to contain a correct table
row, because `SHOW TABLES` includes timestamps from when tables
were created. I'm not sure this is the right way to do this.

We add corresponding changes to dax_test, because the DAX tree
tests against the SQL tests.

We change the returned types of field names and field types to
plain strings, ironically because DAX needs this -- the test code
in the DAX tree is getting them back as plain strings, rather
than as dax.FieldName and dax.BaseType.

The tests using `having` are commented out because they don't
seem to be working, a ticket has been filed for this.

Two of the tests that should return strings are instead returning
untranslated integer IDs, but only for DAX, not for the regular
SQL tests, and the `delete` test has been commented out for
DAX-specific errors. If we merge this, the next step is to
ticket those and address them separately.
2023-03-01 13:40:50 -06:00
Vengata Krishnan
6777e3dc07
FB-1968 timestamp data type related fixes and enhancements (#2256)
* Removed support for EPOCH column constraint from TIMESTAMP SQL data type.
* Implicit conversion of integers to timestamp will treat the integer value as seconds since unix epoch.
* Add new ToTimeStamp(num, timeunit) SQL scalar function to help convert integer values to timestamp.
2023-02-24 14:57:17 -05:00
Seebs
5dffbccdff add test coverage, fix bugs caught by added test coverage
Note: We now skip a test because we can't pass it but fixing
it is presently beyond my understanding. In parser_test.go,
we skip
	SELECT * FROM X INNER JOIN Y ON true INNER JOIN Z ON false
because if we stringify it, we put the "ON true" in the wrong
place, and end up with something we can't parse.

The main change here is modifying AssertParseStatement and
AssertStatementStringer (and the corresponding Expression
functions) to also verify that they can clone and round-trip,
and that we can walk expressions. This gives us a ton
more coverage of Clone and conversions to string, and caught
a number of subtle typos and missing type switch cases.

Some of the changes are mostly cosmetic, such as only
including optional words when stringifying expressions
or statements if those optional words were present originally,
as shown by the Pos value stored for those words.

We also drop a lot of trailing apostrophes from some of the
parse test cases, which appear to be harmless but won't be
reproduced when converting back to strings.

We also add a number of additional test cases, or add
clauses to existing test cases, to improve coverage of a
lot of error testing. For instance, we added a decimal
field to the tests of show table, and added cases
using KEYPARTITIONS. (Although it doesn't *do* anything.)

Similarly, whenever we create a statement, we check
the behavior of requesting a list of sources from it, to
verify that source finding code at least runs.
2023-02-15 14:34:51 -06:00
Travis Turner
126be915a9
Support for setting individual DatabaseOptions (#2231)
* Implement Schemar.SetDatabaseOption(option, value string)

This replaces the temporary `SetDatabaseOptions()` method, which
replaced the entire DatabaseOptions struct, with `SetDatabaseOption`
which takes an option/value pair of strings to set.

* Add SetDatabaseOption to controller http handler and client

This commit also:
- renames some `writeLog` to `writelog`
- updates ApplyDirective to call resource.Unlock() on any resources
  being removed from the local worker

* Add Database related methods to SchemaAPI interface

Currently all implementations of this interface are implemented with
"unimplemented" errors on those methods. Next will be to implement the
necessary methods.

* SQL: CREATE DATABASE and SHOW DATABASES

* SQL: DROP DATABASE

* SQL: Add UNITS option to CREATE DATABASE

* SQL: ALTER DATABASE

* User serverlessStorage.Remove[*]Resource instead of resource.Unlock()

* Add WITH keyword to CREATE/ALTER DATABASE

* fix some WITH logic

* linter fixes

* WITH on CREATE DATABASE is not required
2023-02-06 08:50:28 -06:00
Pat Okeeffe
d92ea8babf
hand comma version of inner join (#2221) 2023-01-25 13:37:05 -06:00
Pat Okeeffe
b3a552e0d3 implement left join (fb-1888) (#2420)
* implement left join

* fixed failing test

(cherry picked from commit 7386f13159)
2023-01-19 22:10:15 +00:00
pokeeffe-molecula
5d025e399b implement CREATE/ALTER/DROP VIEW (fb-1592) (#2408)
* implement CREATE/ALTER/DROP VIEW

* fixed failing test

* another failing test

* fixed some broken serverless tests

(cherry picked from commit c620aae350)
2023-01-19 21:51:16 +00:00
pokeeffe-molecula
0ddf69a322 fix sum aggregate (fb-1874) (#2404)
* handle sum aggregates with ints; handle escaped quotes in blob literals

* added test ceoverage

* skip subquery test for dax

(cherry picked from commit af475a27f2)
2023-01-12 00:49:58 +00:00
pokeeffe-molecula
4e43b767df Bug fix round up (fb-1841, fb-1819, fb-1867) (#2396)
* check root operator after optimize

* round of bug fixes

(cherry picked from commit 9f042216a7)
2023-01-10 23:28:09 +00:00
pokeeffe-molecula
f8120d4833 Consistency in error handling (fb-1799) (#2383)
* return 200 once plan compilation starts; if error, return error in response.

* removed some commented out code that is definitely not needed.

(cherry picked from commit 9dda3ff215)
2023-01-10 23:26:29 +00:00
pokeeffe-molecula
a6c165dd57 Implement DELETE (fb 1557) (#2382)
* delete implementation with test coverage

* optimize IN expressions; stop linter complaining

* fixed some uncovered query cases

* skip test in DAX for now

(cherry picked from commit 021219935f)
2023-01-10 23:25:08 +00:00
pokeeffe-molecula
211c3b759d add allow_missing_values option to bulk insert (fb-1823) (#2372)
* add allow_missing_values option to bulk insert

* test coverage

* review feedback

(cherry picked from commit 15d2ee8b07)
2023-01-10 23:22:58 +00:00
pokeeffe-molecula
4ef70c19da SHOW CREATE TABLE issues (fb-1810) (#2365)
* fixed ddl issues with cache type/size; removed shardwidth option; improved error message

(cherry picked from commit c88d60c9ab)
2023-01-10 23:22:58 +00:00
pokeeffe-molecula
75999414a7 implement having; create view experiment (#2357)
(cherry picked from commit eca3168d63)
2023-01-10 23:20:15 +00:00
Fletcher Haynes
5c39a49285 Sync from private repo to commit 12d608c80d 2022-12-12 09:01:20 -08:00
rachithrr
891a42f9fc FB-1739: Add ability to add a description to a table on creation (#2332)
* FB-1739: Add ability to add a description to a table on creation

- Added CommentOption to handle text after COMMENT option.
- added description field in the createtable plan.
- The description is stored in the existing index metadata.

(cherry picked from commit ad350c2d49)
2022-12-12 09:01:20 -08:00
pokeeffe-molecula
c304991e9f implemented extract ddl; tightened up type related stuff (#2329)
* implemented extract ddl; tightened up type related stuff

* added some test coverage

* review feedback

(cherry picked from commit f62313762c)
2022-12-12 09:01:20 -08:00
pokeeffe-molecula
d6d5ddb501 non-sql aggregation, top, decimal and sundries (#2328)
* fixed a bunch of issues with non-pql aggregation; moved some decimal related functionality; made top actually top (for the non-pql case); experimental create function

* drive up test coverage

(cherry picked from commit 0be0c42b66)
2022-12-12 09:01:20 -08:00
pokeeffe-molecula
c210aaba48 produce a better error when a user tries to sort something unsortable (#2324)
(cherry picked from commit 1ce3a1c103)
2022-12-12 09:01:20 -08:00
Travis Turner
444d4804ec Fix formatting in CLI results with custom SQLResonse.UnmarshalJSON (#2305)
* Fix formatting in CLI results with custom SQLResonse.UnmarshalJSON

When I started this, it was meant to be a quick fix to address the confusing
result formats we were seeing in the CLI. For example, all large integer values
were displayed in scientifc notation. This is because we were passing the result
types from JSON (in this case, float64) into pretty print. Similarly, `IDSets`
and `StringSets` where being printed using the default go Stringer for the types
[]int64 and []string respectively.

I started by writing a customer UnmarshalJSON() method for the `SQLResponse`
type. Part of this (the part which converts data types based on header types)
was already being used in dax tests, so this just formalizes that logic as part
of the `SQLResponse` type.

Then I realized that the sql3 tests (run against the `sql3` package) were
failing because sql3 is not actually returning the `IDSets` and `StringSets`
types. A future task is to formalize return types, define them, and modify sql3
to return them. Once that is done, we can remove the "typed" switch in the
`SQLResponse` json unmarshaller.

Another significant change is the modification to the `ExprDataType` interface:
```
type ExprDataType interface {
	exprDataType()
	TypeName() string
	TypeDescription() string
	TypeInfo() map[string]interface{}
}
```
I added two more methods in order to distinguish between a type (`DECIMAL`), its
description (`DECIMAL(2)`), and its type info (`"scale": int64(2)`). Currently,
the description can be used as the field definition in a CREATE TABLE statement,
but we may want to re-think that. Also, Decimal is the only type currently using
TypeInfo.

Finally, I tried to consilidate things around `dax.FieldType` instead of
comparing against parser types outside of sql3. We still have some sql3 parser
and planner types lurking about, but we can address those in future commits.

* Add some test coverage

* smoke test expected INT, now int

* minor fixes

* Introduce WireQueryResponse and related types

This also changes dax.FieldType to dax.BaseType.

* Populate WireQueryResponse correctly

Currently this is in the http handler, and in the queryer.

* Convert sql3 and dax tests to expect pilosa.WireQueryField in results

* fix PQL tests in the SQL defs

* Address a few of the skipped sql tests in dax

(cherry picked from commit f4385df2cf)
2022-12-12 09:01:20 -08:00
pokeeffe-molecula
e2500fcee8 Fb 1767 (#2308)
* identifiers can now have the '-' character
* skip some integration tests

(cherry picked from commit cca319fbe5)
2022-12-12 09:01:20 -08:00
pokeeffe-molecula
b17ee6a203 fixed join bugs by fixing query optimizer (fb-1699, fb-1700) (#2307)
this commit changes the way the plan is retrieved; implements Stringer on types.PlanExpression in preparation for HAVING support; removes last vestiges internal float64 arithmetic; implements a filter on PlanOpFilter; fixes various bugs in the PlanOptimizer when rewriting qualified references

* fixed selects with unqualified identifiers

* handle bad and non-existent query param inputs more appropriately

* added test coverage for PlanExpression Stringer

Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
(cherry picked from commit 5f662d2bce)
2022-12-12 09:01:20 -08:00
pokeeffe-molecula
e14cde7341 SQL BULK INSERT (fb-1749) (#2291)
SQL BULK INSERT

This change is to support a BULK INSERT/REPLACE statement that adds the ability to 1) take its input from a file, url or in-line blob 2) to map from the input source to the target columns
3) to transform data (using sql expressions) before inserting
4) support csv and ndjson formats

* improving test coverage

* increase test coverage again

* refactoring for handling transformation with types other than id and int

(cherry picked from commit 8f660a5033)
2022-11-15 11:33:38 -08:00
tgruben
751b7a74fe staticcheck fixes (#2278)
(cherry picked from commit 0aa5efcc51)
2022-11-15 11:33:10 -08:00
Travis Turner
bcb32addaf Batch insert via SQL (multiple tuples) (#2243)
* Formatting adjustments made during code review.

While reviewing the BULK INSERT logic (in order to decide how best to
approach "ingest via sql" in the cloud), I made a few formatting and
comment changes. I'm just adding them here as a separate commit so they
don't muddy up my actual work.

* Parser modifications to support mulitple tuples in INSERT INTO

This commit doesn't include all of the changes required in the
planner. Fow now, the planner is simply modified to continue supporting
a single tuple (the first tuple in the list).

* Update the planner to handle multiple INSERT INTO tuples

This is part 1. It's still using the existing logic which builds an
ImportRequest for every record (and every field!).

The next step will involve using a client.Batch to handle the records.

* Introduce client.Importer interface (used by client.Batch)

Instead of the Batch having a pointer to a client, this puts an
interface there instead (which the client implements). It also allows us
to inject a different importer (i.e. other than a featurebase.client)
into the Batch.

* Decouple batch from client

This commit pulls batch-specific code out of the client package and into
a new batch package. It introduces the batch.Importer interface, the
methods of which replace all the calls that batch was previously making
directly to client methods.

Finally, it contains two implementations of the batch.Importer
interface: one is a wrapper around client, and the other is a wrapper
around featurebase.API.

* Use docker (instead of MustRunCluster) for internal batch tests

Because the `batch` package tests are internal, using
test.MustRunCluster() resulted in an import loop (because it eventually
imports `server`, and we can't have that). So this commit replaces the
use of `test.MustRunCluster()` with docker. The setup is basically the
same as that used in the idk docker tests.

Here we also remove all client-side references to `UseIngestAPI`, which
is an experimental (json) ingest api. It's still suppored on the server,
but here we remove the external usage of it.

* cherry-pick fix

* Use batch.Import() for sql3 INSERT INTO statements

* Thread logger into sql3

* fix batch test

* Fix some shadowing complaint by linter

* Address some test issues related to stringsets

* Exclude batch integration tests from CI

* Address PR feedback

- Added description to batch.README
- Consolidated grep commands in .gitlab-ci.yml
- Removed some debugging comments
- Replaces some inadvertantly removed license headers

* Add batch package to gitlab CI

* Updated CI for batch package

Updated CI include path

Update gitlab ci

Update CI

Update CI

Trying new include path for ci

Updated gitlab ci include path

Made idk race job optional for sonarcloud upload

add testdata directory

remove testenv from dockercompose file

use GIT_STRATEGY clone in batch CI

add testdata volume to dockercompose

Co-authored-by: Fletcher Haynes <fletcher.haynes@generalassemb.ly>
(cherry picked from commit 00ef2380e5)
2022-11-15 11:25:36 -08:00
pokeeffe-molecula
15382a2863 sql3 changes (#2211)
* first cut of working (slowly) bulk insert; table valued functions and a tuple data type to support time quantums

* oversight

* filter pushdown implementation; bulk insert

* addressed some linter issues
2022-09-30 11:21:11 -07:00
Seebs
e47bdb7889 cleaned up sync from private repo 2022-09-30 11:20:58 -07:00
pokeeffe-molecula
204558b46f added /sql endpoint; implemented SHOW TABLES (#1935)
* squashed 45 commits into one :)

* tlt/sql experiment (#2035)

* Move PlanOperator to sql3/planner/types package

includes:
type PlanOperatorColumn struct
type PlanOperator interface

* Remove planner dependencies from pilosa package

The goal after this is to prevent the planner package (which doesn't exist yet)
from being imported by the pilosa package; we just want it injected into the server
in server/server.go. This is because the planner package uses pilosa types, so we need
to avoid circular dependencies.

Added ExecutionPlannerFn
Make public: pilosa.ExecOptions
Added a pilosa.Executor interface
Added a planner.types.CompilePlanner interface
Isolated the planner calls to:
- Executor.Execute()
- *API.[method]()

* Move executionplanner files into the sql3/planner package.

This required a bit of gymnastics, and there are some things around
FieldOptions which need to be addressed soon.

* Remove the hacky FieldOptions stuff I added earlier

This implementation just uses the pilosa.FieldOption functional options
provided by the API (as opposed to trying to build a FieldOptions
object.

It also changes field types to constants. These are private for now, but
if we need to make them public, we should put them in the planner/types
package.

* Implement the "scale" value from Decimal(scale)

Also, precision and scale were currently reversed in the parser. This
fixes that.

* Modify the parser to handle CACHETYPE <type> SIZE <size>

It's a little odd to me that the cache type values are Tokens, but I
guess it's ok. One thing to keep in mind is that FeatureBase expects
lowercase values, so this commit changes the parser to set the value to
the lowercase version of the type.

* Fix the /sql2 tests

This entailed a combination of commenting out or t.Skip()-ing tests
which covered code in the parser that has been commented out or removed
as not currently supported in sql3.

It also adds some coverage for the sql.Contraint stringers.

* Prevent JSON sql results from containing closing commas

This commit just re-works the existing output code to avoid inserting
closing commas (which results in invalid JSON).

* Enhance the CREATE TABLE test coverage.

In particular, ensure that the fields which get created in FeatureBase
are what we expect based on the fields defined in the CREATE TABLE
statement.

This also ensures that the TIMEQUANTUM and CACHETYPE contraints are not
provided for the same field (since those constraints are not supported
together).

* Adjust the EBNF file to indicate SIZE contraint is optional

A CACHETYPE can be provided without a SIZE. This change indicates that
SIZE is optional.

* Remove `executionplanner_` from file names (#2040)

* implementation of ALTER TABLE (sans column RENAME)

* refactored expression analysis; added more robust type checking; all unary and bin ops function on ints

* added type support for expressions; full bin/unary op support; added cast; more literal support

* cast int to all other types

* all literals (except idset, stringset & timestamp) make it thru; cast to all types with int as source now works

* implemented LIKE/NOT LIKE

* Implemented IS [NOT] NULL

* Move sql2 files into sql3/parser package (#2045)

* Move sql2 files into sql3/parser package

This also removes the sql2 package.

* Fix tests which were typing _id fields as INT intead of ID

* implemented BETWEEN, NOT BETWEEN

* Add featurebase/error package (#2046)

* Add featurebase/error package

I copied the `dax/errors` package which I am starting to use in the DAX
prototype into `featurebase/errors` in order to start using it with the
sql3 package. It's basically a wrapper around `github.com/pkg/errors`,
but it uses a customer coded error.

The sql package can define its own errors based on the
`featurebase/errors` types. Then do things like `Wrap()` and `Is()`.

* Address the linter complaints: shadowed variables, unreachable code

* implemented IN & NOT IN with expression lists

* first cut of CASE

* Fixed some errors from rebase

* updated bnf; removed unused code; tightened up error handling

* first crack at basic CLI for SQL3

Use: `featurebase cli`

Still lots to do here, but for example:

> select count(*) from tremor
+--------------+
|        COUNT |
+--------------+
| 1.158321e+06 |
+--------------+

* Iterate on the CLI (#2057)

Handle the errors.
Add an "exit" command.
Add some general formatting and white space.

Add termination character: ";" (semicolon)

This commit allows a user to provide multiple or partial SQL statements.

Example of multiple statements:
```
show tables; select * from foo;
```

Example of partial (multi-line) statements:
```
select *
from foo;
```

Don't uppercase the header values

* error refactoring; first cut of TOP; remove unused code; use log.Printf instead of fmt.Printf

* fixed a bug with QualifiedRef from refactoring; added bones of INSERT; removal of unused code; tightened up errors more; fixed failing tests

* single value list for INSERT

* Update bnf per discussion with Travis; INSERT now doing the requisite stuff

* Pat's eyes went square - nothing wrong with TOP, Pat needed to learn arrays again.

* improved some errors; fixed tests to suit

* send warnings back in the api; update CLI to display warnings

* start warning on stuff not implemented so we don't get bugged about it

* Tlt/sql experiment (#2063)

* Expresssion -> Expression

* Add SQL planner test

- adds a test to which it is easier to add tables and SQL statments
- un-exports all of the expression types
- removes the planner pointer from the expression types (it can be added
  back later if need be)

* Fix where clause on a string field

Prior to this commit, the binary expression for a where clause on a
string field was building the call by providing a range operator which
is typically used for BSI fields. This changes it to use the call.Args
for string values.

* Update planner tests to handle multiple sql for the same results

* Reorganize SQL tests

Introduce a test/helpers package and move shared MustQueryRows into that
package.

* Add a compatibility map for field types. (#2064)

This is primarily to address the fact that ID fields were previously
incompatible with INT literals.

We should probably consider introducing a custom type for FieldType
which can be used to define compatibilities.

* significantly refactored type checking

* Handle nil (NULL) values in the sql CLI. (#2067)

go-pretty panics if the interface{} field value is nil. This replaces
nil values with a "NULL" string.

* Squash some commits

fixed a still failing test

added line, col to all error messages

refactored source handling to enable table aliases

fixed some copypasta per review

warnings for order by & topn; implemented select as a source

starting to handle in (select...); added stub for optimizer

JSON-encode the sql error and warning strings (#2069)

Error strings with unencoded characters (like double quotes) were
resulting in invalid json.

got insert working; added symbol table; added concrete optimizer; added nascent NestedLoopsOperator; rewrite "where foo in (select..." as inner join

* all about the sets (#2085)

* implemented setcontains()

* implemented set literal; insert set column values; setcontains/all/any both in expr eval and pql filters

* Convert test to use latest framework. (#2086)

* fixed some comments

* removed refactored tests

Co-authored-by: Travis Turner <travis@pilosa.com>

* Add support for Decimal fields to the sql test. (#2090)

* dates (#2094)

* return dates as strings in output; tightened up decimal type checking

* return dates as strings in output; tightened up decimal type checking

* fixed failing tests after decimal changes

* can now insert decimal values

* implemented insert for timestamp data type; implemented current_date, current_timestamp constants

* fixed some failing tests

* handle date literals from strings in insert statements

* changes from feedback

* Fix pointer method error

* sql3 API interface (#2110)

* Introduce API-related interfaces: SchemaAPI, ComputeAPI

The sql3 code was relying on the pointer: *pilosa.API in order to call
API methods directly on the local node. If we want to import and use the
sql3 package in another service (the DAX queryer, for example), we need
to be able to use an implementation of an interface for those API method
calls.

This commit introduces two interfaces, both automatically implemented by
pilosa.API:
- SchemaAPI
- ComputeAPI

* Convert sql3 code to use IndexInfo instead of Index

The sql3 code was relying on a *pilosa.Index and its methods to get
general information like index and field name, type, etc. This commit
converts everything to use a *pilosa.IndexInfo instead.

This allows us to modify the SchemaAPI interface to also return
IndexInfo instead of Index, which will be a lot easier to implement in a
non-pilosa package (like DAX); creating a *pilosa.Index requires
providing things like data directory paths and holders, which are not
necessary for these use cases.

* Unary and Binary Ops R US plus CAST (#2111)

* implemented string literal for timestamp epoch

* fixed failing test

* fixed the failing test again

* refactored tests; implemented unary op tests for all datatypes; implemented binop tests for int/int, int/id, int/decimal & ID/int

* implemented all binary ops for INT & all other types, ID & all other types

* implemented binary ops for DECIMAL types & all other types

* added STRING & BOOL to various tests; implemented all remaining binOp tests

* fix up some stuff after rebasing

* refactored test defs into multiple files; implemented CAST for every datatype

* added tests for like/not like

* addressed review feedback

* addressed type review feedback

* tightened up IS [NOT] NULL behavior plus tests (#2118)

* tightened up IS [NOT] NULL behavior plus tests

* BETWEEN/NOT BETWEEN with all data types

* addressed review feedback

* Handle negative integers in column min/max constraints (#2120)

This commit parses the min/max contraint as an expression, as opposed to
an int literal, so that negative values are treated as Unary
expressions.

There currently isn't support for min/max constraints on `decimal`
fiels, so for now this change only expects +/- integer values.

* Implement the CREATE TABLE keypartitions logic (#2123)

* Execution time, IN/NOT IN & multiple aggregates (#2124)

* added display of execution time

* IN/NOT IN tests for all data types

* fixed date parsing

* removed duplicative tests

* refactoring aggregates

* suport multiple aggregates

* Address review feedback

* final round of feedback

* Add method SchemaAPI.CreateIndexAndFields() (#2127)

In order to support a CREATE TABLE statement as a single command, this
commit alters the SchemaAPI interface to contain a single method which
handles both the index and its fields. It also updates the sql3 code to
use this interface instead of CreateIndex() and CreateField()
indepedently.

* Symbol Handling (Again) (#2129)

* Refactored symbol handling in the planner; re-instated the select as source tests

* removed commented out code

* addressing review feedback

* Move hard-coded _id field out of planner and into interface implementation (#2130)

This commit moves the hard-coded addition of the `_id` field from the
planner to the SchemaAPI.IndexInfo() implementation method.

NOTE: If anything was expecting SchemaAPI.Schema() to also return the
`_id` field as part of its field list in each table, then it would not
be there because the `_id` field is only added in the IndexInfo() method
for now. Currently that's not a problem because nothing is expecting the
`_id` field for `Schema()`.

* Multiple aggregates, all aggregates stand alone and in GROUP BY (#2132)

* handle multiple aggregates in group by queries

* added handling for avg() aggregate both stand alone and in group by

* tightened up sum & avg outside of group by

* added min, max & percentile

* added warnings

* Make MaterializedRowSet implement the PlanOperator interface. (#2133)

This commit refactors the PQLMultiGroupByOperator to have a PlanOperator
as its output. Then, when it initializes, it sets up a
MaterializedRowSet and populates that with the values from the multiple
group by operations.

* added explicit min/max pql operators

* saved a file I forgot to save

* per review

* Un-indent some if/else nesting (#2136)

Co-authored-by: Travis Turner <travis@pilosa.com>

* Add optional `name` argument to test structs.

This commit adds the `name` argument to `tableTest` and `sqlTest` so
that a test can be optionally named. This allows a developer to more
easily run/identify a particular test by name.

* Inbuilt functions (redux) (#2141)

* set functions type parameter type checking

* implemented datepart

* include SQL3 type in SHOW COLUMNS output

* fixed select as source; failing SHOW COLUMNS test

* select in select list

* dump output columns; handle optimization for select list subqueries

* make it an error to return multiple rows for a select list subquery

* added description

* contants and test coverage for datepart function

* SQL3 Refactor-palooza (#2182)

* removed unneeded IsAggregate()

* first cut of working nested loops operator aka INNER JOIN

* remove selectListItemPlanExpression

* added some warnings

* all the tests are passing again!

* addressed some linter complaints

* added basic order by

* bug fixes; added 'or replace'/'replace' to insert

* for insert references should return appropriately

* added back ability to use subquery singleton expressions

* removed dead code; fixed test

* json-able plan, Schema() plus refactoring

* fixed dumb code

* add some tests for time quantum behavior

* Code cleanup during review. Also fixed INSERT to keyed table bug.

This commit contains a lot of minor adjustments made during code review.

It also contains a bug fix that was preventing INSERT into a keyed table
(i.e. _id type STRING) from working.

Co-authored-by: Travis Turner <travis@molecula.com>

* Fix expected min/max on timestamp column test (decimal field)

I don't know why this changed, but presumably something to do with
decimal related work that happened on master.

* Fix compile problem after rebase

* review feedback

Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
Co-authored-by: Travis Turner <travis@pilosa.com>
Co-authored-by: Travis Turner <travis@molecula.com>
Co-authored-by: Fletcher Haynes <fletcher@capitalprawn.com>
2022-09-30 11:10:24 -07:00