Commit graph

368 commits

Author SHA1 Message Date
reesporte
88d2914b15 fb1172: enable refresh tokens
- rip out gobby stuff
- add tokenCache, groupsCache
- refresh the token if needed
- set cookies after authenticate
- remove signature validation, the IDP does that for us
- added way more unit tests
- update older tests to use new API
- add fake idp to authcluster tests
2022-02-07 13:42:11 -06:00
Matthew Jaffee
254bacc40c remove http subpackage and bring implementations into core
remove interfaces as necessary
2022-02-03 21:04:04 -06:00
Matthew Jaffee
d1f3b58861 remove inspect command 2022-02-03 11:25:31 -06:00
Matthew Jaffee
bff6b17a8e remove check command (was for roaring backend files) 2022-02-02 20:56:18 -06:00
Matthew Jaffee
70ea784d41 don't mind me, just submitting stuff that doesn't even compile and
then getting confused by linter errors
2022-02-02 20:56:18 -06:00
Matthew Jaffee
69c00a92ad remove a bunch of roaring backend stuff
snapshotQueue, op tracking, roaring-only tests
2022-02-02 20:56:18 -06:00
Matthew Jaffee
61783e5827 add option to set ResponseHeaderTimeout per client
this is necessary as in some cases we want a low timeout (when we
expect a quick response, e.g. with backup), but in others we may want
a very long timeout (long running query).

Now we have more granular control over timeouts so we can get things
to fail more predictably in tests.
2022-02-02 14:04:40 -06:00
Souhaila Noor
0e1cf5bbbd Enable authentication/authorization for featurebase tools
- Add auth-token for featurebase import, backup and restore
- Add auth-token to http request
- Create a cluster tests with auth enabled
- Add test for import with auth enabled
2022-01-26 17:30:26 -06:00
Ben Johnson
9ebf0e2119 Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
Seebs
375aaf8fbc don't hardcode local port for backup and restore pprof service
If we hardcode a port, we can't run on a crowded machine, like in
CI. If we use :0, we can print the value actually picked.
2022-01-21 11:12:10 -06:00
reese
2e0aaa27a9
Merge branch 'master' into redirect-url 2022-01-20 13:08:16 -06:00
reesporte
87bbca938c add a redirect-base-url config option
this allows the user to configure a url for their IDP to redirect to, rather
than relying on the bind address of the featurebase server itself
2022-01-20 12:09:18 -06:00
Samir Patel
e1d7389893
Update ctl/keygen.go
Co-authored-by: reese <45641995+reesporte@users.noreply.github.com>
2022-01-19 11:23:02 -05:00
Samir Patel
950e62aae9 update keygen subcommand
this updates the subcommand to output a single secret key
instead of two reflecting changes made to AuthN/authZ
2022-01-19 10:11:19 -06:00
reesporte
cf2410fea6 addresses multiple authn/z tickets
* fb-998 - authn/z enabled in handlers (kitchen-sink ticket)
    - authorization is enabled through the use of a bearer token (using header "Authorization")
    - authorization may occur through the use of an "Authorization" header or "molecula-chip" cookie
    - ui is updated for changes to handler
* fb-1131 - protect grpc endpoints
    - GRPC endpoints now check authorization if auth is enabled
* fb-1129 - inter-node communication
    - the following endpoints use the secretKey for authentication:
        - /internal/cluster/message: POST
        - /internal/translate/data: GET, POST

* added test to api_test.go (TestAuth_MultiNode) testing various auth/permissions stuff on a multi-node cluster

not included:
    - fb-1130 - filter response of endpoints
    - fb-1109 - improved audit logging

@jaffee [are you not entertained](https://www.youtube.com/watch?v=mutgotxrcqg)

Co-authored-by: souhailanoor <90720110+souhailanoor@users.noreply.github.com>
Co-authored-by: tgruben <tgruben@gmail.com>
Co-authored-by: 54mir <48686912+54mir@users.noreply.github.com>
Co-authored-by: kcrodgers24 <49999391+kcrodgers24@users.noreply.github.com>
2022-01-14 12:31:32 -06:00
Matthew Jaffee
34393dee09 rip out rowcache
not strictly backward compatible... hopefully no one is actually using
the rowcache config option
2022-01-11 13:49:09 -06:00
Matthew Jaffee
6335b9c801 disable retryablehttp logger because *wow* that's a lot of output 2022-01-11 10:15:58 -06:00
reesporte
8d6490329b Merge branch 'master' into protect-endpoints 2022-01-04 16:18:18 -06:00
tgruben
16600a219c
Merge branch 'master' into 54mir/authentication 2022-01-03 17:55:58 -06:00
Ben Johnson
af9795aa1a Avoid panics in RBF debug tooling 2022-01-03 13:13:02 -07:00
reesporte
42f3557c55 fix merge conflicts 2021-12-29 09:05:53 -06:00
Matthew Jaffee
1a8c10d5f3 fix backup fail test so it actually fails
A few things were going wrong here.

First, we take a "RetryPeriod" option on backup and restore which is
meant to be roughly the total amount of time we spend retrying any
given request before failing. However we were incorrectly passing that
as the RetryMaxWait which is the maximum amount of time to sleep
between any two attempts. We now do some fuzzy math to figure out
approximately how many attempts we should make given a minimum sleep
of 100ms and the fact that we double the sleep time every attempt.

Second, during the backup test, if a host was totally stopped when we
started the request, it would fail immediately and then retry, but if
the host was stopped during the request (after DNS had resolved), then
the request would wait for the DialTimeout which we default to 30s, so
turning off the cluster for 5 seconds and turning it back on resulted
in the backup completing rather than failing. Because of this, we
change the commandClient to have a default dial timeout of 1 second.

I was tempted to change the global default to 1s which I think would
be fine, but didn't want to break anything too badly.
2021-12-28 13:31:42 -06:00
Samir Patel
0ef67fd699 move query logger option to auth 2021-12-27 16:42:13 -05:00
Travis
6638fa17ee
Expose etcd.dir configuration option
The goal is to allow a user to separate FeatureBase and etcd I/O.
2021-12-27 11:13:38 -06:00
Samir Patel
684c408b93 Merge branch '54mir/protect-endpoints' into queryLoggerSetup 2021-12-22 15:12:31 -06:00
Samir Patel
448289d609 add subcommand for key generation 2021-12-22 13:04:00 -06:00
Matthew Jaffee
295fab4892 retry on >= 400, not just greater. good catch 2021-12-22 12:21:11 -06:00
Matthew Jaffee
640ba45129 use retryableHTTP in client, fix memory usage of restore
instead of awkwardly reading an entire file into a buffer, we use
retryablehttp's reader func to open the file fresh if we need to
retry, so a small fixed-size buffer can be used internally for copying
the contents onto the network.
2021-12-22 10:56:16 -06:00
Matthew Jaffee
cde3f6b5ea add profiling to backup/restore 2021-12-21 16:24:21 -06:00
Matthew Jaffee
2bce396445 add retry restore test and custom retry policy 2021-12-21 16:24:21 -06:00
Matthew Jaffee
f676fbfc51 add retryability to restore command 2021-12-21 16:24:20 -06:00
Matthew Jaffee
8486efaa79 add exponential retry logic to internal http client, use in backup 2021-12-21 16:24:20 -06:00
rachithrr
5650a24c9b query logger is set up. 2021-12-21 16:52:22 -05:00
Samir Patel
3b374a62bf Merge branch 'master' into 54mir/authentication 2021-12-20 16:42:11 -06:00
Samir Patel
6faa889bfb move logout url to conf 2021-12-20 14:30:19 -06:00
souhailanoor
55c67a9d1f
Merge branch 'master' into fb1000 2021-12-20 12:35:17 -06:00
Seebs
977a699a98 don't panic on invalid page type
debugging tools shouldn't panic when they encounter bugs. insert
"you had one job" meme.
2021-12-20 10:58:41 -06:00
souhailanoor
e82088e086
Merge branch 'master' into fb1000 2021-12-19 11:38:08 -06:00
Matthew Jaffee
1fd872b126 less write locks in fragment.importRoaring/row 2021-12-17 15:09:25 -06:00
Samir Patel
a1de086cd8 change scopes from string to slicee 2021-12-15 14:53:04 -06:00
Samir Patel
541132a176 hash and block key cmd options 2021-12-13 23:10:29 -06:00
souhailanoor
e5052a864b
Merge branch 'master' into fb1000 2021-12-10 14:09:39 -06:00
Souhaila Noor
01e4baab04 determine permission for user access to index 2021-12-10 14:07:24 -06:00
Matthew Jaffee
53373240ef make chksum process All() results correctly for unkeyed indexes 2021-12-10 11:52:39 -06:00
Matthew Jaffee
1980c8b8e5 featurebase backup: don't hide TranslateStoreNotFoundError
I think this shouldn't happen unless there's actually a problem
2021-12-10 11:52:39 -06:00
Matthew Jaffee
3d3080df8b full backup/restore test in a Go test 2021-12-10 11:52:39 -06:00
Matthew Jaffee
aff3d3ddd9 do a backup in a go test for coverage purposes
also found a weird issue with schema marshalling

if you create a field thru the api w/o specifying a field type, you
get slightly different behavior than going thru the HTTP handler which
is... not ideal. I changed the marshaler to accept an empty field type.
2021-12-10 11:52:39 -06:00
Matthew Jaffee
b8da3bc7e6 checksum All() instead of Count(All()) to cover index keys 2021-12-10 11:52:39 -06:00
reesporte
48aef0c8a4 add copyright notice back in
```bash
for file in `cat diffys`; do
   printf '%s\n%s\n' "// Copyright 2021 Molecula Corp. All rights reserved." "$(cat $file)" >$file;
done
```
2021-12-10 11:01:04 -06:00
reesporte
4c53f86e82 removed license from each go file
i used this script, a little clunky but it got the job done

```bash
for file in `find . -type f -print | grep '\.go'`; do
    sed '1,/^\/\/ limitations under the License.$/d' $file > $file.tmp;
    result=`cat $file.tmp`
    if [[ result != "" ]]; then
        gofmt $file.tmp &> /dev/null;
        if [[ $? == 0 ]]; then
            mv $file.tmp $file && gofmt -w $file;
        else
            rm $file.tmp;
        fi
    else
        rm $file.tmp;
    fi
done
```
2021-12-10 09:17:17 -06:00