All jobs now use GOVERSION or GOFUTURE to determine the docker image
pulled.
GOFUTURE is latest so that it will always use the latest version as new
versions are released. We can later lock it to a specific minor version
when there is another one released.
Co-authored-by: Garrison Davis <garrison.davis@featurebase.com>
the "changes" rule appears not to be good at detecting changes
in some cases. specifically, it appears that you have to be in
an "only:" clause, not a "rules" clause, to trigger the
merge-specific behavior which checks the entire merge branch
instead of the top commit, but that means that if your last
commit doesn't touch IDK, we don't run IDK tests, and I haven't
been able to fix this yet.
So for now, revert the IDK-specific change detection behavior,
which slows CI down but gets us test coverage.
We call sonar-scanner on the IDK data, and then we change
into the IDK directory and try to run it again on the same files,
which don't exist in that directory. We shouldn't be running it
twice; we should run it once on all the files.
More subtly, we created files named foo_coverage.out, then tried
to glob files named coverage*.out. (The apparent similarity of the
$(PROJECT)_coverage.out names is harmless, PROJECT is getting set
and they're using different names.)
Fixing this gets SonarCloud more reliable again.
This is a bit complicated and entangled, sorry.
First, we squash the auth-based smoke tests into the regular smoke
tests; we just run all the tests with auth on and that way we don't
need to spin up an entire separate cluster of machines just to run
a single query against them.
We improve the error detection, and standardize the jq-to-get-config
code. The purpose of this is to try to make sure that, if we actually
hit a failure and get "null" for a host name, we report *that*
as an error, rather than running ahead and producing 20+ separate
reports that ssh failed because it couldn't find a host named null.
Upgrade Go to 1.19
* Use go install to install statik for CI/CD
* Switch from stretch to buster for idk
The stretch release doesn't exist anymore for go 1.19 docker images.
buster is a newer version of Debian anyway (v10 vs v9)
Co-authored-by: Fletcher Haynes <fletcher.haynes@molecula.com>
The IDK tests should be run only when there's changes in the IDK
or client directories.
Also the shard transactional test should be run ever.
The "optional: true" flag is a fascinating quirk allowing you to
express that, *if* a job exists, we should wait for it, but if the
job doesn't exist, that's also acceptable.
Before, we were building docker images for IDK for each of the four
linux/darwin amd64/arm64 platfrom/arch combinations, which didn't make
sense. If we want to later build docker images for linux/arm64, we can
add that later.
I also cleaned up the Dockerfile for IDK to minimize creation of excess
layers (by &&-ing RUN commands), and made apt quieter to cut back some
of the noise.
We need to install base gosec tool and then the gitlab version to convert the gosec
json to the gl-sast-report.json that GitLab expects.
This lets us see the 'Security' tab under pipelines (and under the default branch
after this change is merged).
I chose to pin both of the versions of the tools to avoid any dependencies changing.
This could be an issue, but both repos are largely frozen.
We're using this because it's a builtin env variable that comes
with GitLab, and it fixes one of the annoying things about Docker
tags (e.g., you can't use all of the allowed characters in
Git branches).
One issue that I've seen a few times, is branches with either
capital letters (which was recently broken), or using the '/'
character.
This PR makes it so we always use the CI_COMMIT_REF_SLUG
when making or referencing images so that it is always consistent.
Note: this might make it slightly harder to intuit what the correct
Docker image to make (if you wanted to use the one built by CI rather
than locally). This trade off doesn't seem too hard to overcome.
* [CLOUD-934] Optionally broadcast IDK Kinesis errors/panics to external storage
- Add a minor public method `idk.Main.SetLog` to allow setting the logger instance
after initialization.
- Add a Logger implementation that captures recoverable errors and panics
and pushes to an external store. Meant to decorate an existing Logger
instance and always delegate to its implementation. Decoration happens
when all AWS resources are initialized. Before then, the wrapped Logger
implementation is used.
- If `--error-queue-name/CONSUMER_ERROR_QUEUE_NAME` specified, use an
ErrorStreamLogger to push errors and panics to an SQS queue with that name.
Omission of the option preserves current behavior.
- Parse sink ID from the `--stream-name/CONSUMER_STREAM_NAME` expecting the form
'PREFIX'-VALID_UUID. If the sink UUID is invalid, emit a warning that errors/panics
will not be written to an SQS queue but will still be logged using the decorated
Logger instance.
- The inability to push to an SQS queue leads to warnings being emitted to notify
ECS that no queue will be written to and is NOT a hard error.
- Add SQS interface mock for unit testing.
- Add IDK make targets for generating mock interfaces.
* [CLOUD-934] Execute go mod tidy and go fmt to pass CI/CD checks
* [CLOUD-934] Remove extraneous Makefile in idk/kinesis and fix install-mock-generator target
* [CLOUD-934] Add godocs to exported types and functions
* [CLOUD-934] Changed warning to not sound so ominous and update associated unit test
* [CLOUD-934] Unblock CI/CD at the IDK test stage
previously we allowed users to specify a granularity for timestamp
e.g. seconds, milli, micro, nano
however we converted everything to nano before we stored it.
This reduced the allowed range for all time units to what
was allowed by timestamp. For example, with second granularity
you can represent billions of years within the capacity of
int64 but with nano its somewhere b/w 100-200 years.
So now, for timeunits of seconds, milli, and micro the range
is year 0001 - 9999. These limits come from what Go
supports.
So this uses unit specific function to translate
timestamps to values and vice versa to increase
the time range.
In the process of increasing the range for timestamp and subsequent
testing, I found and addressed a few bugs:
- min/max queries were not using timestamp specific comparators so
added that.
- Values from Import/ingest come to FB as relative values to epoch
whereas other BSI fields come as actual values and then
becomes relative to their respective bases within FB. so some
specific handling of that was added.
- However! Set queries use timestamp strings which are, of course,
the actual value they designate. So they have to become
relative.
- When bitdepth is 0, Min/maxUnsigned functions did not run
resulting in a count of 0 when there
was an actual value that was 0.
Also, this removes (now) dead code and updates/adds tests.
* unifying idk and featurebase: first pass
* resolved conflict with master for gitignore & dockerignore
* deleted binaries that were accidentally pushed to git
* combined gitlab jobs for idk & featurebase
* run go fmt for idk
* updated ssh env variable, and made docker password variable in gitlab env variables
* fixed typo assigning variable name
* trying to fix docker login error
* trying a different solution for docker password
* pass registry
* fixed docker login
* updated paths for idk
* exclude idk tests from featurebase test run
* fix vendor error
* update certificates
* grpc needs to be in version 1.38
genproto, which is imported by big query updates the grpc version to 1.47.0
grpc 1.47.0 causes etcd to deadlock when calling etcd.Close()
the fix is to have a replace in go.mod to specify a specific grpc version
* run go mod tidy
* go mod
* run go mod tidy
* exclude bigquery since it is causing issues and undo grpc replace in go.mod
* fix grpc version
* fix formatting error
* update formatting
* attempt to fix formatting
* update path for code coverage
* update to use current branch binaries, not master
* fix for building idk - path updates
* udpate path for binaries
* update job dependecies
* update docker idk tests to use the current branch registry
* update stages for jobs
* updated job dependencies
* not allow idk s3 dump to fail since it is a dependency for integration tests
* update dependecy for idk tests
* update paths for idk build and code coverage
* download featurebase binary from s3
* pass branch name to all setup scripts
* change to current branch instead of master
* updated sonarcloud
* sonarcloud fix and branch name fix
* trying to speed up pipeline run time
* update stage
* branch name fix + sonar cloud
* sonarcloud
This PR reverts the timestamp work.
The timestamp work requires changes to FB and IDK; and there are
circular dependencies between tests in either repo preventing merging
of either. The work here is pretty stable, but required bypassing
the smoketest. Meanwhile, I found some additional things in IDK that
need addressing which means I merged this work in pre-maturely. Once
I get that worked out, I'll re-commit these commits.
This reverts following commits related to timestamp work:
bypass of smoke test b/c of circular dep with IDK: 0676790
update codec to reflect changes to timstamp range: bd5dc76
fix few bugs regarding timestamp: 33fce8a
increase time range for timestamp by using specified granularity: 5939923.
The sonarcloud job was accidentally altered to use *only* the PLG
coverage data, which is incomplete for reasons not yet fully
understood. Unfortunately, it wasn't *waiting* for the PLG coverage
data to be complete -- the job could start before the PLG coverage
ran, which mean that you could get anywhere from a few percent
to nearly total code coverage.
Also, we want to be sure to cover *both* the PLG and non-PLG coverage
data, so we add the non-PLG coverage data.
We also factor out the simulacraData package from our PKG_LIST because
it appears to be confusing sonarcloud because that package isn't
"included in project" or something.
Also remove a stray `ls` that was probably part of the original
testing/debugging of this.
* automate builds of single node featurebase for PLG
* make plg target uses go build instead of go install
* corrected artifact names in plg build stages
* add s3 dump for plg
* edits to s3 dump for plg
* create getter for monitor state
* refactor monitor
* fix http middleware
* change warn to error if attmpt to cluster on plg
* sentry: special considerations if execution is part of test
- skip test if they build a cluster as this will error by design
- skip sending messages to sentry if testing
* got single node working; refining error messaging and version info to follow
* better implementation that separates the build condition into etcd/enterprise_cluster.go and etcd/plg_cluster.go. go build will default to a clustering version and 'go build -tags plg' will build the non-clustering version
* added Makefile target for 'make plg'
* additional comments, CI/CD update
Co-authored-by: Kasey Rodgers <kaseyrodgers@Kaseys-MBP.attlocal.net>
* Changed several gauntlet tests to be manual due to cost emergency
* Made perf_able manual
Co-authored-by: Fletcher Haynes <fletcher.haynes@molecula.com>
- terraform scripts to set up cluster
- cloud-formation scripts to set up cluster
- set up ingest node with kafka server and datagen
- set up second ingest node with molecula-consumer-kafka-static
- set up datadog in all nodes (ingest + featurebase)
- set up script to execute different queries
- only run delete test on schedule
this stems from https://molecula.atlassian.net/browse/SUP-194 where the string
"standard" was being passed to viewTimePart, which output "standard" as the result.
this is not a valid time string and was causing confusing errors. now it simply
doesn't do that
this commit also adds regression testing framework and a regression test for fb-1287
This was in response to some feedback we got about the new release
format. Executables were no longer had executable permission due to
going through S3 (hence the tarballs), and we wanted a more consistent
directory structure in the final release which included the versions
of various components.
