Upgrade Go to 1.19
* Use go install to install statik for CI/CD
* Switch from stretch to buster for idk
The stretch release doesn't exist anymore for go 1.19 docker images.
buster is a newer version of Debian anyway (v10 vs v9)
Co-authored-by: Fletcher Haynes <fletcher.haynes@molecula.com>
Before, we were building docker images for IDK for each of the four
linux/darwin amd64/arm64 platfrom/arch combinations, which didn't make
sense. If we want to later build docker images for linux/arm64, we can
add that later.
I also cleaned up the Dockerfile for IDK to minimize creation of excess
layers (by &&-ing RUN commands), and made apt quieter to cut back some
of the noise.
There are currently three copies of a package called `fakeidp` in the
featurebase repo:
- ./idk/fakeidp/go.mod
- ./internal/clustertests/fakeidp/go.mod
- ./qa/fakeidp/go.mod
All three have a `go.mod` file. While this is supported under golang's
new Workspace support, what's not supported is that the modules share
the same name (in this case "fakeidp"). This commit is a sort of
temporary fix which renames the module for two of the instances. This
prevents, for example, VSCode with workspace support enabled, from
barfing.
By the way, one can enable VSCode workspace support with the following
setting:
```
// gopls
"gopls": {
"build.experimentalWorkspaceModule": true
}
```
Also...
This commit fixes the `make testv` target. It's probably not used
anywhere (which I'm assuming because it was broken), but it's a handle
target, so now it will list and run tests against all packages found in
the repo, including the root package.
* [CLOUD-934] Optionally broadcast IDK Kinesis errors/panics to external storage
- Add a minor public method `idk.Main.SetLog` to allow setting the logger instance
after initialization.
- Add a Logger implementation that captures recoverable errors and panics
and pushes to an external store. Meant to decorate an existing Logger
instance and always delegate to its implementation. Decoration happens
when all AWS resources are initialized. Before then, the wrapped Logger
implementation is used.
- If `--error-queue-name/CONSUMER_ERROR_QUEUE_NAME` specified, use an
ErrorStreamLogger to push errors and panics to an SQS queue with that name.
Omission of the option preserves current behavior.
- Parse sink ID from the `--stream-name/CONSUMER_STREAM_NAME` expecting the form
'PREFIX'-VALID_UUID. If the sink UUID is invalid, emit a warning that errors/panics
will not be written to an SQS queue but will still be logged using the decorated
Logger instance.
- The inability to push to an SQS queue leads to warnings being emitted to notify
ECS that no queue will be written to and is NOT a hard error.
- Add SQS interface mock for unit testing.
- Add IDK make targets for generating mock interfaces.
* [CLOUD-934] Execute go mod tidy and go fmt to pass CI/CD checks
* [CLOUD-934] Remove extraneous Makefile in idk/kinesis and fix install-mock-generator target
* [CLOUD-934] Add godocs to exported types and functions
* [CLOUD-934] Changed warning to not sound so ominous and update associated unit test
* [CLOUD-934] Unblock CI/CD at the IDK test stage
previously we allowed users to specify a granularity for timestamp
e.g. seconds, milli, micro, nano
however we converted everything to nano before we stored it.
This reduced the allowed range for all time units to what
was allowed by timestamp. For example, with second granularity
you can represent billions of years within the capacity of
int64 but with nano its somewhere b/w 100-200 years.
So now, for timeunits of seconds, milli, and micro the range
is year 0001 - 9999. These limits come from what Go
supports.
So this uses unit specific function to translate
timestamps to values and vice versa to increase
the time range.
In the process of increasing the range for timestamp and subsequent
testing, I found and addressed a few bugs:
- min/max queries were not using timestamp specific comparators so
added that.
- Values from Import/ingest come to FB as relative values to epoch
whereas other BSI fields come as actual values and then
becomes relative to their respective bases within FB. so some
specific handling of that was added.
- However! Set queries use timestamp strings which are, of course,
the actual value they designate. So they have to become
relative.
- When bitdepth is 0, Min/maxUnsigned functions did not run
resulting in a count of 0 when there
was an actual value that was 0.
Also, this removes (now) dead code and updates/adds tests.
* unifying idk and featurebase: first pass
* resolved conflict with master for gitignore & dockerignore
* deleted binaries that were accidentally pushed to git
* combined gitlab jobs for idk & featurebase
* run go fmt for idk
* updated ssh env variable, and made docker password variable in gitlab env variables
* fixed typo assigning variable name
* trying to fix docker login error
* trying a different solution for docker password
* pass registry
* fixed docker login
* updated paths for idk
* exclude idk tests from featurebase test run
* fix vendor error
* update certificates
* grpc needs to be in version 1.38
genproto, which is imported by big query updates the grpc version to 1.47.0
grpc 1.47.0 causes etcd to deadlock when calling etcd.Close()
the fix is to have a replace in go.mod to specify a specific grpc version
* run go mod tidy
* go mod
* run go mod tidy
* exclude bigquery since it is causing issues and undo grpc replace in go.mod
* fix grpc version
* fix formatting error
* update formatting
* attempt to fix formatting
* update path for code coverage
* update to use current branch binaries, not master
* fix for building idk - path updates
* udpate path for binaries
* update job dependecies
* update docker idk tests to use the current branch registry
* update stages for jobs
* updated job dependencies
* not allow idk s3 dump to fail since it is a dependency for integration tests
* update dependecy for idk tests
* update paths for idk build and code coverage
* download featurebase binary from s3
* pass branch name to all setup scripts
* change to current branch instead of master
* updated sonarcloud
* sonarcloud fix and branch name fix
* trying to speed up pipeline run time
* update stage
* branch name fix + sonar cloud
* sonarcloud
