We want "make test" to run in a reasonable amount of time and
actually work, the IDK tests are full of tests that only run in
a specialized docker environment with things like hosts named
pilosa and kafka and such.
Before, we were building docker images for IDK for each of the four
linux/darwin amd64/arm64 platfrom/arch combinations, which didn't make
sense. If we want to later build docker images for linux/arm64, we can
add that later.
I also cleaned up the Dockerfile for IDK to minimize creation of excess
layers (by &&-ing RUN commands), and made apt quieter to cut back some
of the noise.
There are currently three copies of a package called `fakeidp` in the
featurebase repo:
- ./idk/fakeidp/go.mod
- ./internal/clustertests/fakeidp/go.mod
- ./qa/fakeidp/go.mod
All three have a `go.mod` file. While this is supported under golang's
new Workspace support, what's not supported is that the modules share
the same name (in this case "fakeidp"). This commit is a sort of
temporary fix which renames the module for two of the instances. This
prevents, for example, VSCode with workspace support enabled, from
barfing.
By the way, one can enable VSCode workspace support with the following
setting:
```
// gopls
"gopls": {
"build.experimentalWorkspaceModule": true
}
```
Also...
This commit fixes the `make testv` target. It's probably not used
anywhere (which I'm assuming because it was broken), but it's a handle
target, so now it will list and run tests against all packages found in
the repo, including the root package.
* Fixed broken fields when packaging rpm and deb files.
* Update systemd unit files, package them into RPM's.
* Fix config file path for packages.
* Changed unit and binary paths to conform to standard locations for each vendor.
* Added featurebase owned directories.
* Create featrebase user/group and chown the right dirs
* Automate turning on featurebase
* Updated the .gitignore to include .vscode files.
* Changed RPM name to better conform to naming standards.
* Pass GOARCH when building RPM's.
* Avoid using recursive to remove files in this dir.
* automate builds of single node featurebase for PLG
* make plg target uses go build instead of go install
* corrected artifact names in plg build stages
* add s3 dump for plg
* edits to s3 dump for plg
* got single node working; refining error messaging and version info to follow
* better implementation that separates the build condition into etcd/enterprise_cluster.go and etcd/plg_cluster.go. go build will default to a clustering version and 'go build -tags plg' will build the non-clustering version
* added Makefile target for 'make plg'
* additional comments, CI/CD update
Co-authored-by: Kasey Rodgers <kaseyrodgers@Kaseys-MBP.attlocal.net>
This adds a shard-based import endpoint which takes bitmap data for
all field types and imports data for the whole shard transactionally.
It uses the BitmapRewriter interface to try to intelligently allow for
setting and clearing bits simultaneously without multiple writes which
is especially helpful when ingesting into int-like fields, but also
allows clear-and-then-set behavior for set fields.
- rip out gobby stuff
- add tokenCache, groupsCache
- refresh the token if needed
- set cookies after authenticate
- remove signature validation, the IDP does that for us
- added way more unit tests
- update older tests to use new API
- add fake idp to authcluster tests
this should be a lot more reliable than trying to construct it based
on the project name as the exact construction can differ between
docker-compose versions.
There was also an issue with the backups succeeding when they should
fail in the test. There's an arcane maze of HTTP timeouts to navigate
here, but basically there are situations where the client will just
wait forever rather than erroring if the server is paused at the
right(wrong) time. I'm not convinced we've solved every possible case
of this, so we still may see the backup succeed even when it's
supposed to fail. The ultimate hammer is to add Client.Timeout, but
that's a very blunt instrument and I'm afraid it could cause a timeout
when really we just have a lot of data to download or something.
There may be a better way to say "only time out if you literally
haven't heard a peep from the server in this long", but I haven't been
able to figure it out yet.
I also fixed how the authclustertests are run as they weren't using
the PROJECT parameter correctly. Now they can run concurrently with
clustertests, and with other copies of authclustertests without having
conflicts.
- Add auth-token for featurebase import, backup and restore
- Add auth-token to http request
- Create a cluster tests with auth enabled
- Add test for import with auth enabled
gitlab CI runs as much as 5x slower sometimes during business hours,
resulting in tests failing due to 10-11 minute timeouts that would
succeed in under 2-3 minutes outside of business hours. to allow us
to do anything at all, let's just set that to half an hour, and 90
minutes for `go test -race`.
Concern: It's possible there's a timeout that's a gitlab CI configuration
thing involved too, because we see some go test timeout panics, but we
also see some weird messages about SIGQUIT at 11 minutes, which isn't
the go test timeout, so we may need to address that too.
Note that we're changing the Makefile, and also the config for the
gitlab CI passes, which don't use the Makefile. The Makefile changes
are just to be careful and avoid retriggering this later. We may
want to revert these if we get the other issues fixed.
* fb-998 - authn/z enabled in handlers (kitchen-sink ticket)
- authorization is enabled through the use of a bearer token (using header "Authorization")
- authorization may occur through the use of an "Authorization" header or "molecula-chip" cookie
- ui is updated for changes to handler
* fb-1131 - protect grpc endpoints
- GRPC endpoints now check authorization if auth is enabled
* fb-1129 - inter-node communication
- the following endpoints use the secretKey for authentication:
- /internal/cluster/message: POST
- /internal/translate/data: GET, POST
* added test to api_test.go (TestAuth_MultiNode) testing various auth/permissions stuff on a multi-node cluster
not included:
- fb-1130 - filter response of endpoints
- fb-1109 - improved audit logging
@jaffee [are you not entertained](https://www.youtube.com/watch?v=mutgotxrcqg)
Co-authored-by: souhailanoor <90720110+souhailanoor@users.noreply.github.com>
Co-authored-by: tgruben <tgruben@gmail.com>
Co-authored-by: 54mir <48686912+54mir@users.noreply.github.com>
Co-authored-by: kcrodgers24 <49999391+kcrodgers24@users.noreply.github.com>
This adds the following test:
1. cluster comes up (node 1,2,3), status normal
2. Pause node 3
3. Insert keys making sure to filter out the keys that will go to the paused node
4. Wait for status to become degraded
5. Unpause node 3
6. Wait for status to get back to normal
7. Check that keys were replicated to all 3 nodes
The rbf_bolt tests are unusually expensive, partially because they're
run with the race detector on, but also because it's basically running
two copies of all the tests and comparing them... But they haven't
detected anything in ages, because the RBF stuff is now pretty stable,
and those tests take about twice as long as anything else in our testing,
and thus impede our workflow noticably for little-to-no return. We might
some day want to fully remove them, but for now, just taking them out of
CI should streamline our workflows a bit.
