* Store version-check file in the configured data-directory
This also fixes what I think is a bug.
It also un-exports everything.
I have questions.
* Version checking: clean up code, add server flags
FB-1975
Cleaned up version checking, removed a race condition, and added error checking.
Added server flags for check-in endpoint and UUID storage file.
Incorporates Travis's changes to store UUID file in data directory
and unexport most of verchk.go.
---------
Co-authored-by: Travis Turner <travis@molecula.com>
Co-authored-by: seebs <seebs@molecula.com>
When we close a task pool, we use a condition variable to wait for
workers to exit, if any workers are still running. The workers,
in turn, use the condition variable to notify that they've exited.
Unfortunately, the workers aren't using the lock (the rationale was
that it's all atomic ops so they don't need to), which means that
it's possible to have the following sequence:
Close(): obtain current live count
worker: decrement live count
worker: send broadcast to condition variable
Close(): wait on condition variable
To resolve this, we make the worker update also request the lock.
We add a simple reproducer for this. Note that simple doesn't mean
it fires completely reliably; on my laptop, the test causes a test
timeout about 60% of the time without the fix. If you add a short
delay between sampling the live count and waiting on the condition
variable, the deadlocks move from "60% chance of hitting it in
a million trials" to "nearly always".
* Add viper (for env variable) support to CLI
* Move the "featurebase cli" sub-command to its own "fbsql" command
I don't know if this is the final name, but putting it here as a
placeholder for now.
* Handle single `--command` flags.
This also adds a printer interface so we can opt NOT to print setup
information in non-interactive mode.
* Add support for multiple `--command` flags in the same call
* Add support for multiple `--file` flags
* Move members related to Config into a separate struct
* Make sure non-interactive mode can connect to a database
* comment fix
* support control-C on readline
* Prevent connection message from printing in non-interactive mode
* Return errors (instead of printing them) in non-interactive mode
* make nodeid come from the correct table
* refactored aggregates; added ability to aggregate on expressions not just references
* addressed feedback
* now with the compiler errors fixed after rebase
If we don't set an epoch, we get a cryptic message on the console.
Note, this message isn't logged properly, it doesn't use the
logger, it uses the `log` package.
2023/02/09 11:07:23 ERROR: converting timestamp options for
end_time: checking overflow: custom epoch too far from
Unix epoch: 0001-01-01 00:00:00 +0000 UTC
Because this uses the log package, it doesn't go to the same place
as other messages, making it a pain to debug.
The underlying problem is that a timestamp can't just have a zero
value for its epoch. So, we set a default epoch of 0 Unix Time.
We should possibly revisit the question of whether the conversion
in the top-level schema.go should handle an epoch which IsZero,
but I'm not sure what "base" should be in that case. In practice,
all existing usages except this one are specifying time.Unix(0, 0)
already.
Note: We now skip a test because we can't pass it but fixing
it is presently beyond my understanding. In parser_test.go,
we skip
SELECT * FROM X INNER JOIN Y ON true INNER JOIN Z ON false
because if we stringify it, we put the "ON true" in the wrong
place, and end up with something we can't parse.
The main change here is modifying AssertParseStatement and
AssertStatementStringer (and the corresponding Expression
functions) to also verify that they can clone and round-trip,
and that we can walk expressions. This gives us a ton
more coverage of Clone and conversions to string, and caught
a number of subtle typos and missing type switch cases.
Some of the changes are mostly cosmetic, such as only
including optional words when stringifying expressions
or statements if those optional words were present originally,
as shown by the Pos value stored for those words.
We also drop a lot of trailing apostrophes from some of the
parse test cases, which appear to be harmless but won't be
reproduced when converting back to strings.
We also add a number of additional test cases, or add
clauses to existing test cases, to improve coverage of a
lot of error testing. For instance, we added a decimal
field to the tests of show table, and added cases
using KEYPARTITIONS. (Although it doesn't *do* anything.)
Similarly, whenever we create a statement, we check
the behavior of requesting a list of sources from it, to
verify that source finding code at least runs.
* Refactor CLI to mimic psql's meta-commands
This PR adds support for meta-commands (also known as "backslash
commands") like those in psql, Postgres's CLI. Only a few meta-commands
are currently implemented, but this was meant to demonstrate how we
could use something like `\i file.csv` to insert local files into SQL
statements.
* Meta-commands: \file and \include
The initial implementation used `\i` as a streaming file handle.
This commit changes that to `\file`, and then implements `\i` (or
`\include`) as handling multiple sql commands.
* Add meta-command "help" (\?)
This is basically a copy of the psql help output, but includes only
those options we currently support.
* Add support for \o [file], and \timing
The \o meta-command writes query output to a file.
The \timing meta-command turns on/off the timing display sent to stdout.
* Add meta-commands: \l (show databases) and \dt (show tables)
* Add meta-command: \watch [period]
* Update meta-command \connect to take database name instead of ID
* Add support for \echo, \qecho, and \warn
This commit contains an known issue in that the `-n` option will exclude
the line feed, but if the output is the terminal, the readline package
clobbers any content on the current line (i.e. anything without a line
feed). That will need to be addressed at some point.
* Add support for \w [FILE] (write query buffer to file)
* Add SchemaAPI no-op implementation
* Refactor query handler to align with /sql and /databases endpoints
We want to standardize on:
/sql
/databases/{databaseID}/sql
* Add CLI support for expanded, border, tuples_only (and pset)
* Add help text for \pset and \t
* fb-1940 re-implemented some changes that got missed private-public
* fb-1939 fixes to between + decimals
* fb-1935 - avg() on and id type + fixed some tests
* fb-1953 add min/max for string types
* fb-1938 - remove internal_type column from show columns
* fb-1964 - fix space_used in fb_cluster_nodes to be int
* fb-1996 - make sure all Idents that are being used as object references to schema objects are lowercased
* fixed failing test
* added some missed changes
* fb-1969 found another case issue with identifier used for column idents
* Implement Schemar.SetDatabaseOption(option, value string)
This replaces the temporary `SetDatabaseOptions()` method, which
replaced the entire DatabaseOptions struct, with `SetDatabaseOption`
which takes an option/value pair of strings to set.
* Add SetDatabaseOption to controller http handler and client
This commit also:
- renames some `writeLog` to `writelog`
- updates ApplyDirective to call resource.Unlock() on any resources
being removed from the local worker
* Add Database related methods to SchemaAPI interface
Currently all implementations of this interface are implemented with
"unimplemented" errors on those methods. Next will be to implement the
necessary methods.
* SQL: CREATE DATABASE and SHOW DATABASES
* SQL: DROP DATABASE
* SQL: Add UNITS option to CREATE DATABASE
* SQL: ALTER DATABASE
* User serverlessStorage.Remove[*]Resource instead of resource.Unlock()
* Add WITH keyword to CREATE/ALTER DATABASE
* fix some WITH logic
* linter fixes
* WITH on CREATE DATABASE is not required
* tweak a bunch of logging and config
make overall logs less verbose and chatty
1 minute computer check-in interval
3 minute snapshot interval
remove CaptureLogger as it has same functionality as buffer logger
add a WithPrefix to the Logger interface so sub-services can have
different prefixes
* fix some lint
* fix lint... confused why this is coming up now
* Remove MDS and replace it with Controller
This commit removes the MDS layer (and package) and shifts Controller
package into its place.
* add pprof/fgprof to serverless http router
---------
Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
* Support Drop Table in serverless (include Snapshotter, Writelogger)
* Finish Database methods
Things like:
- `Databases`
- `DatabaseByID`
- `DatabaseByName`
- `DropDatabase`
* Change Poller to use NodeService instead of its own map
Instead of the Poller maintaining its own map of Addresses to poll, this
commit changes the Poller to use the NodeService interface to get all
known nodes from the Controller.
The next commit needs to:
Next, the logic in the boltdb NodeService implementation was moved to
the boltdb Balancer implementation. That way, the Balancer can be the
source of truth for all things nodes/workers/jobs.
* Move NodeService from Controller to Balancer
This commit moves the implementation of the NodeService into the
Balancer, and aligns `Balancer.AddWorker` with `NodeService.CreateNode`
so that they stay in sync. (Same for `Balancer.RemoveWorker` and
`NodeService.DeleteNode`).
* fix import of private repo
* Fix go vet issues
* Fix bug in DeregisterNode
We need to remove the node from the NodeService even if it's not
assigned to a database. The logic had a bug in it.
This also adds some no-op implementations for SnapshotService and
WriteloggerService. If a directory was not configured for that, then the
computer node would panic on trying to read from the Snapshotter upon
receiving a Directive.
* queryer response content-type: json
* Add support for NULL to WriteloggerDir and SnapshotterDir configs
This commit changes the way WriteLoggerDir and SnapshotterDir are
handled.
If value is empty `""`, an error will be returned on computer startup.
If value is `"NULL"`, a no-op implementation of the service will be
used. This would be for a case that wanted to run serverless on-prem
with no durable storage.
Finally, any other value will be used as the directory to use.
Some things which aren't considered here and may result in unexpected
behavior:
- a value with spaces `" "`
- any "null" which is not "NULL"... like lowercase.
* Finish the DropTable test
* Change "disable service" value to case-insensitive "off"
This commit also removes an unnecessary sleep in the tests.
* Fix docker-compose variables for IDK test
Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
* only keep the last 2000 requests and truncate sql and plan text to 4K each
* fixed import
* address review feedback
Co-authored-by: Fletcher Haynes <fletcher.haynes@generalassemb.ly>
