da4fb4ab4e
This gets complicated. For coverage output, sonarcloud supports wildcards. For test output, it doesn't. So we weren't getting meaningful results, just weird error messages. I fixed that, and got thousands of lines of other error messages because it wasn't finding the test source files. That looked like this: WARN: Failed to find test file for package github.com/molecula/featurebase/v3 and test TestTranslation_Primary But we don't actually need the test reports sent to SonarCloud, because "which parts of your test suite are being run" is sort of inherently "basically all of them" with go test. So rather than continuing to do that, we drop it. Since we're dropping that, we don't need the JSON output from go test anymore, so we drop that too, and the tee commands, and the "artifacts" from the tee commands, and now our test output is human-readable and slightly faster. We also bump SonarCloud to 4.7. We also fix the tests to use GOVERSION sometimes and GOFUTURE other times, and bump from 1.19.2 to 1.19.3. Also a couple of minor cleanup (adding explanatory comments, combining adjacent grep commands, etc.) |
||
|---|---|---|
| .. | ||
| egpool | ||
| testdata | ||
| batch.go | ||
| batch_test.go | ||
| convert.go | ||
| docker-compose.yml | ||
| Dockerfile-test | ||
| Dockerfile-wait | ||
| error.go | ||
| importer.go | ||
| Makefile | ||
| metrics.go | ||
| README.md | ||
| wait.sh | ||
batch
The batch package provides a standard tool set for batching records in a way
that is most performant for ingesting those records into FeatureBase. The main
implementation is Batch (which can be initated with the NewBatch()
function). The NewBatch() function takes an Importer which contains all of
the methods required to interact with FeatureBase; these include methods for
doing string/id translation as well as for importing shards of data.
IDK uses the batch package internally. Another example where the batch
package is used in the sql3 package. When an "INSERT INTO" statement is
executed, the SQL engine uses a Batch to do key translation and build import
batches prior to doing the final import.
Integration tests
To run the tests, you will need to install the following dependencies:
In addition to these dependancies, you will need to be added to the molecula Gitlab account.
First start the test environment. This is a docker-compose environment that includes featurebase.
make startup
To build and run the integration tests, run:
make test-run-local
Then to shut down the test environment, run:
make shutdown
The previous command is equivalent to running the following:
make startup
sleep 30 # wait for services to come up
make test-run
make shutdown
To run an individual test, you can run the command directly using docker-compose. Note that you must run docker-compose build batch-test for docker to run the latest code. Modify the following as needed:
make startup
docker-compose build batch-test
docker-compose run batch-test /usr/local/go/bin/go test -count=1 -mod=vendor -run=TestCmdMainOne .