Commit graph

478 commits

Author SHA1 Message Date
tgruben
09b4eed348
[FB-1388] Add optional support for continuous profiling in datadog (#2037)
Enable ability to use DataDog continuous profiling
2022-05-12 12:33:46 -05:00
tgruben
dd139cd48d
[FB-1398] Mainline etcd (#2049)
* use mainline etcd-io dependencies, not forks

this commit does lots of things around clustering with goal of increasing stability.

- upgrades from molecula/etcd to go.etcd.io/etcd@v3.5.4
- upgrades from seebs/bbolt to go.etcd.io/bbolt@v1.3.6
- update tests to use unix sockets for etcd cluster communication
    - this is what etcd uses for a lot of internal testing, so if their devs think
      it's a valid test, we can probably accept that
- cleanup etcd node-watcher shutdown process

Co-authored-by: tgruben <tgruben@gmail.com>

* moved random query to another repo

it had weird dependency issues with upgrading to mainline etcd bc of the vegeta dep
so we removed it bc no one really uses it anyway

we got this error message:
```
github.com/molecula/featurebase/v3/cmd/random-query imports
	github.com/tsenart/vegeta/v12/lib tested by
	github.com/tsenart/vegeta/v12/lib.test imports
	github.com/streadway/quantile tested by
	github.com/streadway/quantile.test imports
	.: "." is relative, but relative import paths are not supported in module mode
```

* add cleanup to EtcdUnixSocket test util

Co-authored-by: reesporte <reesedporter@gmail.com>
2022-05-11 09:49:48 -05:00
Matthew Jaffee
e0e01f9f65 expose --header-timeout option on featurebase backup 2022-04-29 16:37:40 -05:00
Travis Turner
7ccc845aac
Change Ttl to TTL (#2038)
* Change Ttl to TTL

Following go convention, acronyms should have a consistent case.
See
[Initialisms](https://github.com/golang/go/wiki/CodeReviewComments#initialisms)

This commit changes some public-facing methods, so any code importing
this package and using these methods will need to be updated.

* rewrite Ttl -> TTL

Co-authored-by: reesporte <reesedporter@gmail.com>
2022-04-27 11:20:39 -05:00
Hoang Pham
49cf5c8817 FB-1188 - improved description of TTL 2022-03-07 11:16:33 -06:00
Hoang Pham
e16171cb4b FB-1188 - Added TTL field option 2022-02-25 11:39:31 -06:00
Todd Gruben
6c512359a1 missed a fmt statement 2022-02-14 14:53:47 -06:00
Todd Gruben
cbc9bf71a1 logging 2022-02-14 14:41:22 -06:00
Todd Gruben
8a48c1b67a standard logger 2022-02-14 14:09:05 -06:00
tgruben
c4eebc6885
Update cmd/roaring-migrate/main.go
Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
2022-02-14 12:17:55 -06:00
tgruben
89598a7788
Update cmd/roaring-migrate/main.go
Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
2022-02-14 12:17:42 -06:00
tgruben
e1e968ef97
Update cmd/roaring-migrate/main.go
Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
2022-02-14 12:17:24 -06:00
Todd Gruben
7e25ca467a Merge branch 'sup-146' of github.com:molecula/featurebase into sup-146 2022-02-14 12:12:30 -06:00
Todd Gruben
9bc28091c9 better testing 2022-02-14 12:11:28 -06:00
tgruben
e963bccc93
Merge branch 'master' into sup-146 2022-02-14 08:16:37 -06:00
Matthew Jaffee
b5dae698ff remove unused env var from test
cluster.hosts is no longer a config option since move to etcd
2022-02-11 12:02:10 -06:00
Matthew Jaffee
102a6e723b more binding to 0==less port conflicts in CI 2022-02-11 12:02:10 -06:00
Souhaila Noor
7983a7506f - Need to get code coverage on the server and client side
- For server side, used an instrumented binary with a test that wraps around the main entrypoint for featurebase
- Every time, the binary is called, a new coverage file is generated.
- For the client side, used the standard -coverprofile flag for go test to generate code coverage
- For backup test that's expected to fail, needed to call Run call in backup.go directly. The code coverage is not written to disk for an instrumented binary if there is an error.
2022-02-11 11:26:00 -06:00
Todd Gruben
8d5cdbdd77 roaring-migrate bug;performance improvements 2022-02-11 11:19:44 -06:00
Matthew Jaffee
6cc5d198ee remove unused stuff and fix a bunch of random staticcheck issues
sorry... once I saw, I couldn't unsee
2022-02-07 15:10:10 -06:00
Matthew Jaffee
254bacc40c remove http subpackage and bring implementations into core
remove interfaces as necessary
2022-02-03 21:04:04 -06:00
Matthew Jaffee
d1f3b58861 remove inspect command 2022-02-03 11:25:31 -06:00
Matthew Jaffee
bff6b17a8e remove check command (was for roaring backend files) 2022-02-02 20:56:18 -06:00
Souhaila Noor
0e1cf5bbbd Enable authentication/authorization for featurebase tools
- Add auth-token for featurebase import, backup and restore
- Add auth-token to http request
- Create a cluster tests with auth enabled
- Add test for import with auth enabled
2022-01-26 17:30:26 -06:00
Ben Johnson
9ebf0e2119 Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
Samir Patel
950e62aae9 update keygen subcommand
this updates the subcommand to output a single secret key
instead of two reflecting changes made to AuthN/authZ
2022-01-19 10:11:19 -06:00
reesporte
42f3557c55 fix merge conflicts 2021-12-29 09:05:53 -06:00
Samir Patel
448289d609 add subcommand for key generation 2021-12-22 13:04:00 -06:00
Matthew Jaffee
640ba45129 use retryableHTTP in client, fix memory usage of restore
instead of awkwardly reading an entire file into a buffer, we use
retryablehttp's reader func to open the file fresh if we need to
retry, so a small fixed-size buffer can be used internally for copying
the contents onto the network.
2021-12-22 10:56:16 -06:00
Matthew Jaffee
cde3f6b5ea add profiling to backup/restore 2021-12-21 16:24:21 -06:00
Matthew Jaffee
f676fbfc51 add retryability to restore command 2021-12-21 16:24:20 -06:00
Matthew Jaffee
8486efaa79 add exponential retry logic to internal http client, use in backup 2021-12-21 16:24:20 -06:00
Todd Gruben
c25ab78b03 use higherlevel iterator in order to account for ops log 2021-12-13 13:10:16 -06:00
reesporte
48aef0c8a4 add copyright notice back in
```bash
for file in `cat diffys`; do
   printf '%s\n%s\n' "// Copyright 2021 Molecula Corp. All rights reserved." "$(cat $file)" >$file;
done
```
2021-12-10 11:01:04 -06:00
reesporte
4c53f86e82 removed license from each go file
i used this script, a little clunky but it got the job done

```bash
for file in `find . -type f -print | grep '\.go'`; do
    sed '1,/^\/\/ limitations under the License.$/d' $file > $file.tmp;
    result=`cat $file.tmp`
    if [[ result != "" ]]; then
        gofmt $file.tmp &> /dev/null;
        if [[ $? == 0 ]]; then
            mv $file.tmp $file && gofmt -w $file;
        else
            rm $file.tmp;
        fi
    else
        rm $file.tmp;
    fi
done
```
2021-12-10 09:17:17 -06:00
reesporte
76c1d237ac translaste -> translate
also remove meaingless comments
2021-12-08 16:35:26 -06:00
reesporte
b046ad5e8f fixes some more staticcheck errors 2021-12-03 16:50:02 -06:00
reese
3d3d0b7a51
Merge branch 'master' into staticcheck-issues 2021-12-03 16:02:30 -06:00
reesporte
c2964f7c7b rename to pilosa
thanks to alan's comment [here](https://molecula.atlassian.net/browse/FB-1021?focusedCommentId=11721)
2021-12-03 14:56:18 -06:00
reesporte
666baffb7d Merge branch 'master' into staticcheck-issues 2021-12-03 09:36:13 -06:00
reesporte
63c5c11108 fix some staticcheck issues 2021-12-03 09:31:45 -06:00
Todd Gruben
5c5e16af33 added a generate only flag 2021-11-19 12:26:23 -06:00
Todd Gruben
f4a40e79b9 final cleanup 2021-11-19 12:08:24 -06:00
Todd Gruben
989f9c3f48 applied suggestions 2021-11-19 11:29:17 -06:00
Todd Gruben
b12f00db59 changed default seed to current time 2021-11-19 10:57:36 -06:00
Todd Gruben
1089435218 add check for index presence 2021-11-19 09:48:45 -06:00
Todd Gruben
c72406819e added seed param 2021-11-19 09:40:39 -06:00
Todd Gruben
f6b3a81ac3 made duration minutes for query rate 2021-11-19 05:16:30 -06:00
Todd Gruben
87ededd61e Modified random query to use vegeta library 2021-11-18 16:23:12 -06:00
Todd Gruben
7de69df424 added protection against trailing slash 2021-11-15 10:54:13 -06:00