Until we have API token support, connecting to cloud requires the
cognito configuration. This commit adds the production cognito settings
for defaults, and we have intentinally omitted these from the documentation.
FB-1896
Added the datetimename function, which returns parts of a timestamp
as strings. Month and day of the week are named ("January", "Monday")
while others are returned as a string of digits ("2023").
Added tests to the test definitions.
* Set fbsql prompt based on the connected database
This also changes the prompt to align with psql, where it begins with:
db=#
and the mid looks like:
db-#
* Require organizationID in on-prem, serverless queries
* Support meta-commands in `--file` command
I'm not sure why this was restricted before. Just an oversight.
* Change default history file name to fbsql_history
* Add port short flag: p
* Support meta command \list (for \l) and \out (for \o)
* cleanup while writing docs
* Have \cd with no arguments change to home directory
* Avoid shadowing `action`
* Add kafka support to CLI (fbsql)
This commit adds the ability to provide a `--kafka-config` command line
argument referncing a toml file to configure kafka.
* Move "Molecula Consumer" message to the logger; hide it in basic mode
* Fold decimal(scale) into kafka.source-type
* Build fbsql with cgo in docker for CI
* Re-organize the fbsql kafka config and setup.
Allow field config to use the table schema if no fields provided.
* Display timestamp fields with format RFC3339Nano
* remove kafkaRunner (no longer used)
* Fix cli/batch test (and make sure it's not excluded from CI)
The logic in our Makefile was exluding from tests any package with
`/batch` in the package name. This excluded `/cli/batch`, which is not
good.
This commit changes the exclusion logic to include the `/v3` portion of
the package name, so `/v3/batch`.
* Rename Basic() to SetBasic()
Buckle in, this one's a ride.
This is attached to the same PR as a fix for exiting abruptly
during some tests because I ran into that issue, and comprehended
it, while trying to track down weird and sporadic test failures
that were actually this issue.
The actual, underlying, problem: `make test`, by running all the
tests at once, was hitting a bug that was mostly effectively
triggered by running the `dax/test/dax` tests, and the top-level
`featurebase/v3` tests, at the same time. However, the interaction
was nothing as obvious as temporary files, etcd configuration,
or whatever.
We were running out of port numbers.
The tests were using a bit over 30k simultaneous established TCP
connections, each to different ports, because we were creating
new clients for basically every single operation. For instance,
in a single SQL test that did an import and then a read, we
were creating a new client for each field written to, and then
also creating a new client for each field in results that needed
key translation. And none of these clients were closed or
timed out in any way. In fact, Go doesn't really *do* "closing"
of clients; the closest is that an http.Client can be told to
close idle connections that it has been keeping open.
The worst offenders were both named `fbClient`, and were nigh-identical,
except one of them was implemented as a method on `importer` in the
IDK tree, and one was a standalone function.
It may seem surprising that the method on `importer` is using a shared
client pool for all importers, rather than a new pool for each
importer. This is because we potentially make quite a few importers
during tests.
Before this, running either of the dax tests or the top-level
tests would show well over ten thousand simultaneous ESTABLISHED
connections. After this, the dax tests used nearly twenty.
The problem with port consumption like this, while more noticeable
on MacOS, is also something we could hit on the CI runners, especially
if a single runner ended up with more than one test suite running
at the same time. This probably manifests as sporadic very strange
failures of CI, with messages about "cannot assign requested address".
(Note that an outgoing connection to a successfully-created port
requires *another* port to be assigned for the outbound socket.)
This was complicated dramatically by the fact that, for some
utterly cursed reason, it was *especially* common for the point
at which we hit this, in the top-level featurebase tests, to be
running one of the backup tests in TestVariousQueries, and
specifically, to be hitting it on the dataframe part of the
backup... Which is to say, on the *one* path in the backup function
that called log.Fatal, and thus terminated the featurebase process
abruptly without further commentary.
The testing package is full of subtle magic, and one of the most
subtle is this: t.Logf, etcetera, all write to a buffer which is
then displayed after the test is run. Which means that, if you
exit, the buffer is never displayed. This means that, if a test
case can fail in a way that causes an instant exit, you don't
hit defers, you don't get your log messages, you just get a mysterious
exit of the process.
We have two cases where backup commands were calling log.Fatal
instead of returning an error. The error in question is displayed
correctly and informatively if returned, so we return it.
We also have one case where we were using os.Exit to avoid a
deadlock. Instead, we make the thing that would deadlock
conditional on the test not having failed. In the event that
the test fails, we now print our failure message correctly,
then also report an unclosed cluster. That's fine.
* first time quantum queries working
* implement select from timequantum columns
* skip a dax test
* make linter happy
* addressed review feedback
* reverted over eager test elimination
* correct reference for `having count(*)`
It turns out that `having count(*) ...` was always treating
the count(*) as exactly 1. After studying this a lot, I noticed
that in fact, we correctly handle other counts. The reason is
that there's already code to recognize aggregates in `having`
clauses as matching aggregates that are being computed -- but
it only covers the other aggregate clause types, not the newly
added `countStarPlanExpression` from making `count(*)` work even
if there's no `_id` field.
We add several corresponding test cases.
* fix sum(a_decimal) type conversion
Added a test case for this, and also added a fix for it.
Underlying issue: qualifiedRefPlanExpression could end up
producing an int64 instead of a pql.Decimal, even though it
had expected type Decimal.
Originally this worked by politely converting an int64 to
a pql.Decimal in the Evaluate phase, but this was not ideal;
the real question is why it was coming out as an int64 at
that step. Showed this to Pat, who spent a while studying it
and produced a better fix.
* temporarily comment out test which fails in DAX
This forward-ports a number of tests from the previous SQL
implementation. The porting is approximate in a number of ways,
and not all tests are implemented/tested yet.
In particular, several tests are currently disabled because
we don't support `limit n` constructs.
The tests that were primarily tests of the parser have been
brought forward as parser tests. One of them has been altered
to add parentheses, because our parser interprets
fld1 between 1 and 3 and fld2 = 2
as:
fld1 between (1 and 3) and (fld2 = 2)
which is invalid, while the old parser apparently interpreted it
as:
(fld1 between 1 and 3) and (fld2 = 2)
We have not yet verified the SQL spec's requirements here, but
sqlite agrees with our old parser, not our new parser, so this
may be a regression.
The old tests expected an INNER JOIN to suppress duplicate
values. Our new code does not, which is consistent with other
SQL implementations. This is a change, but the old behavior
appears to have been wrong. (You can still suppress duplicate
values by specifying DISTINCT.)
In the previous implementations, a value like `count(*)` had
`count(*)` as its column name. In the new implementation,
it has an empty string as its column name.
Related to this, the prior implementation allowed you to
write
select age, count(*) from grouper group by age having count > 1
but the new implementationt requires that to be spelled as
having count(*) > 1
This is consistent with other SQL implementations, so I think
the new behavior is correct.
The behavior of SHOW COLUMNS and SHOW TABLES has changed, in
that the specific results returned are significantly different.
Perhaps more significantly, the old system spelled the former
query as SHOW FIELDS, rather than SHOW COLUMNS. This may be
considered a regression, in that `SHOW FIELDS` no longer works,
and we should consider whether any hypothetical users might
have been relying on the output of either of these. (I hope
not, the new output is much better.)
Some of the old tests (the ones in handler_test) were accommodated
by adding a couple of specific test cases to existing tests,
specifically:
* handling timestamp values with `Z` rather than `+00:00`
* a join with a WHERE clause referring to fields in both
source tables
We introduce a new "partial" comparison type, because there's
no way for a test of `SHOW TABLES` to contain a correct table
row, because `SHOW TABLES` includes timestamps from when tables
were created. I'm not sure this is the right way to do this.
We add corresponding changes to dax_test, because the DAX tree
tests against the SQL tests.
We change the returned types of field names and field types to
plain strings, ironically because DAX needs this -- the test code
in the DAX tree is getting them back as plain strings, rather
than as dax.FieldName and dax.BaseType.
The tests using `having` are commented out because they don't
seem to be working, a ticket has been filed for this.
Two of the tests that should return strings are instead returning
untranslated integer IDs, but only for DAX, not for the regular
SQL tests, and the `delete` test has been commented out for
DAX-specific errors. If we merge this, the next step is to
ticket those and address them separately.
While fixing a bug that log messages were ending up
in the output buffer for backups, we fixed up a bunch of
things to do with log messages and output for various
commands.
Due to a subtle oversight, this means that since we did
that, executor_test's `chkSumCluster` has been dutifully
printing `hash:blahblahblah` to os.Stdout, and returning
an empty string.
This also, indirectly, fixes a very strange behavior
we've had ever since then, which is that a lot of test
output silently disappears. The reason is probably,
although I haven't found the right code path, that we
were ending up closing os.Stdout.
The caching of test results means that we get instant cached results
rather than actually running tests in some cases, which is virtually
guaranteed not to be what we want. There's almost no cases where
this actually speeds a thing up validly, and a lot where it speeds a
thing up invalidly.
* WIP: json marshal coded errors for http
* WIP: trying to see how best to implement the http-error tests
* finish stubbing out the Schemar methods in the test
* using json to move CodedErrors across boundaries and associated tests
* implemented feedback and fixes
---------
Co-authored-by: Travis Turner <travis@molecula.com>
* Removed support for EPOCH column constraint from TIMESTAMP SQL data type.
* Implicit conversion of integers to timestamp will treat the integer value as seconds since unix epoch.
* Add new ToTimeStamp(num, timeunit) SQL scalar function to help convert integer values to timestamp.
* Add meta-commands \set and \unset (for variables)
* WIP: first pass at variable replacement
* Use a mapReplacer instead of having Command implement replacer
* remove circular reference with variables
* remove the `replacer` interface; just have it be a struct
* use a lexer for variable replacement
* Separate the featurebase and fbsql make build targets
Using the same build target was problematic because they shared the same
flags. Since the `-o` output flag was used, the fbsql binary was
overwriting the featurebase binary.
* make sure the make package target builds fbsql
we want to be sure we delete the files we created during the
run even if the test run panics, but not files other runs
may have created also in /mnt/ramdisk.
So, we were special-casing creating a ramdisk, and setting a special
environment variable for it, for boltdb translate files, to improve
performance.
But actually, etcd and test cluster data and so on all go in $TMPDIR,
and if you move all of those also into a ram disk, you get way better
performance. But 2GB may not be enough for that.
So!
We unify on $TMPDIR, we bump the default size to 4GB, we make the
size configurable, and we stop using the name RAMDISK. This should
improve performance on MacOS significantly for `make test` and
things like it, and also simplifies our lives by not having a special
case for the boltdb translate files.
Also change the environment variable names used in our CI config.
(I don't see where we mount the ramdisks, but I think that's happening
in our setup.)
