Commit graph

11 commits

Author SHA1 Message Date
Travis Turner
eb6c6e3105
Add kafka support to CLI (fbsql) (#2278)
* Add kafka support to CLI (fbsql)

This commit adds the ability to provide a `--kafka-config` command line
argument referncing a toml file to configure kafka.

* Move "Molecula Consumer" message to the logger; hide it in basic mode

* Fold decimal(scale) into kafka.source-type

* Build fbsql with cgo in docker for CI

* Re-organize the fbsql kafka config and setup.

Allow field config to use the table schema if no fields provided.

* Display timestamp fields with format RFC3339Nano

* remove kafkaRunner (no longer used)

* Fix cli/batch test (and make sure it's not excluded from CI)

The logic in our Makefile was exluding from tests any package with
`/batch` in the package name. This excluded `/cli/batch`, which is not
good.

This commit changes the exclusion logic to include the `/v3` portion of
the package name, so `/v3/batch`.

* Rename Basic() to SetBasic()
2023-03-07 08:18:22 -06:00
Garrison Davis
7b8b3d8e4f Remove pre_clone_script 2023-03-02 17:24:43 -07:00
Garrison Davis
5c22c9803a Add default retries 2023-03-02 11:55:02 -07:00
Garrison Davis
ffdb308472 Remove manual git clean from CI 2023-03-02 11:55:02 -07:00
Garrison Davis
f0e1b72834 Add CI_PRE_CLONE_SCRIPT to .gitlab-ci.yml 2023-03-02 11:55:02 -07:00
Julio Martinez
b8e5e1f32c
Use curl instead of find so errors propagate. (#2227)
Co-authored-by: Julio Martinez <julio.martinez@featurebase.com>
2023-03-01 11:33:44 -08:00
Garrison Davis
186da6b302 Refactor build-fbsql and upload to S3 in CI 2023-02-24 10:48:48 -07:00
Travis Turner
a633b72f3d
Separate the featurebase and fbsql make build targets (#2272)
* Separate the featurebase and fbsql make build targets

Using the same build target was problematic because they shared the same
flags. Since the `-o` output flag was used, the fbsql binary was
overwriting the featurebase binary.

* make sure the make package target builds fbsql
2023-02-22 15:33:11 -06:00
Seebs
d1dbcabb3d fix cleanup logic for ramdisk usage
we want to be sure we delete the files we created during the
run even if the test run panics, but not files other runs
may have created also in /mnt/ramdisk.
2023-02-22 14:45:56 -06:00
Seebs
8724eb09b0 improve ramdisk config in Makefile, use it for everything in tests
So, we were special-casing creating a ramdisk, and setting a special
environment variable for it, for boltdb translate files, to improve
performance.

But actually, etcd and test cluster data and so on all go in $TMPDIR,
and if you move all of those also into a ram disk, you get way better
performance. But 2GB may not be enough for that.

So!

We unify on $TMPDIR, we bump the default size to 4GB, we make the
size configurable, and we stop using the name RAMDISK. This should
improve performance on MacOS significantly for `make test` and
things like it, and also simplifies our lives by not having a special
case for the boltdb translate files.

Also change the environment variable names used in our CI config.
(I don't see where we mount the ramdisks, but I think that's happening
in our setup.)
2023-02-22 13:19:06 -06:00
Garrison Davis
7bd62952d6 Add GitLab CI pipeline 2023-01-23 10:24:52 -07:00