* Formatting adjustments made during code review.
While reviewing the BULK INSERT logic (in order to decide how best to
approach "ingest via sql" in the cloud), I made a few formatting and
comment changes. I'm just adding them here as a separate commit so they
don't muddy up my actual work.
* Parser modifications to support mulitple tuples in INSERT INTO
This commit doesn't include all of the changes required in the
planner. Fow now, the planner is simply modified to continue supporting
a single tuple (the first tuple in the list).
* Update the planner to handle multiple INSERT INTO tuples
This is part 1. It's still using the existing logic which builds an
ImportRequest for every record (and every field!).
The next step will involve using a client.Batch to handle the records.
* Introduce client.Importer interface (used by client.Batch)
Instead of the Batch having a pointer to a client, this puts an
interface there instead (which the client implements). It also allows us
to inject a different importer (i.e. other than a featurebase.client)
into the Batch.
* Decouple batch from client
This commit pulls batch-specific code out of the client package and into
a new batch package. It introduces the batch.Importer interface, the
methods of which replace all the calls that batch was previously making
directly to client methods.
Finally, it contains two implementations of the batch.Importer
interface: one is a wrapper around client, and the other is a wrapper
around featurebase.API.
* Use docker (instead of MustRunCluster) for internal batch tests
Because the `batch` package tests are internal, using
test.MustRunCluster() resulted in an import loop (because it eventually
imports `server`, and we can't have that). So this commit replaces the
use of `test.MustRunCluster()` with docker. The setup is basically the
same as that used in the idk docker tests.
Here we also remove all client-side references to `UseIngestAPI`, which
is an experimental (json) ingest api. It's still suppored on the server,
but here we remove the external usage of it.
* cherry-pick fix
* Use batch.Import() for sql3 INSERT INTO statements
* Thread logger into sql3
* fix batch test
* Fix some shadowing complaint by linter
* Address some test issues related to stringsets
* Exclude batch integration tests from CI
* Address PR feedback
- Added description to batch.README
- Consolidated grep commands in .gitlab-ci.yml
- Removed some debugging comments
- Replaces some inadvertantly removed license headers
* Add batch package to gitlab CI
* Updated CI for batch package
Updated CI include path
Update gitlab ci
Update CI
Update CI
Trying new include path for ci
Updated gitlab ci include path
Made idk race job optional for sonarcloud upload
add testdata directory
remove testenv from dockercompose file
use GIT_STRATEGY clone in batch CI
add testdata volume to dockercompose
Co-authored-by: Fletcher Haynes <fletcher.haynes@generalassemb.ly>
* unifying idk and featurebase: first pass
* resolved conflict with master for gitignore & dockerignore
* deleted binaries that were accidentally pushed to git
* combined gitlab jobs for idk & featurebase
* run go fmt for idk
* updated ssh env variable, and made docker password variable in gitlab env variables
* fixed typo assigning variable name
* trying to fix docker login error
* trying a different solution for docker password
* pass registry
* fixed docker login
* updated paths for idk
* exclude idk tests from featurebase test run
* fix vendor error
* update certificates
* grpc needs to be in version 1.38
genproto, which is imported by big query updates the grpc version to 1.47.0
grpc 1.47.0 causes etcd to deadlock when calling etcd.Close()
the fix is to have a replace in go.mod to specify a specific grpc version
* run go mod tidy
* go mod
* run go mod tidy
* exclude bigquery since it is causing issues and undo grpc replace in go.mod
* fix grpc version
* fix formatting error
* update formatting
* attempt to fix formatting
* update path for code coverage
* update to use current branch binaries, not master
* fix for building idk - path updates
* udpate path for binaries
* update job dependecies
* update docker idk tests to use the current branch registry
* update stages for jobs
* updated job dependencies
* not allow idk s3 dump to fail since it is a dependency for integration tests
* update dependecy for idk tests
* update paths for idk build and code coverage
* download featurebase binary from s3
* pass branch name to all setup scripts
* change to current branch instead of master
* updated sonarcloud
* sonarcloud fix and branch name fix
* trying to speed up pipeline run time
* update stage
* branch name fix + sonar cloud
* sonarcloud
