scriptoriumadmin/featurebase
1
0
Fork 0
mirror of https://github.com/featurebasedb/featurebase.git synced 2026-10-16 07:48:05 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
8472 commits 45 branches 174 tags 85 MiB
Commit graph

8472 commits

This branch
This branch
All branches
Author SHA1 Message Date
pokeeffe-molecula
44d635f407 disabling single node deploy 2022-01-03 22:43:25 -06:00
pokeeffe-molecula
56584905c2 fix gitlab pipeline; disable circleci; remove artifactory 2022-01-03 22:11:30 -06:00
pokeeffe-molecula
61c0ef9a1e
Merge pull request #1835 from molecula/fix-failed-deploy-linux-node 
Fix failed deploy linux node
 2022-01-03 21:50:21 -06:00
Fletcher Haynes
b9e8d3a103 Commented out a failing test as a meta-test 2022-01-03 19:17:11 -08:00
Fletcher Haynes
70a3af97a8 Fixed some variables in the CI file 2022-01-03 19:05:39 -08:00
Fletcher Haynes
c0708e5403 Changed the env vars a CI job was accessing 2022-01-03 16:48:42 -08:00
Fletcher Haynes
58a70863eb
Merge pull request #1823 from molecula/fb901 
Fb901
 2022-01-03 15:51:24 -08:00
Fletcher Haynes
a40d974377
Merge branch 'master' into fb901 2022-01-03 14:20:08 -08:00
pokeeffe-molecula
51ad67e06a
Update qa/tf/gauntlet/samsung/README.md 
Co-authored-by: reese <45641995+reesporte@users.noreply.github.com>
 2022-01-03 16:09:26 -06:00
pokeeffe-molecula
804b7b3146
Update qa/tf/README.md 
Co-authored-by: reese <45641995+reesporte@users.noreply.github.com>
 2022-01-03 16:06:54 -06:00
Ben Johnson
379c5e0bd1
Merge pull request #1832 from molecula/rbf-no-panic 
Avoid panics in RBF debug tooling
 2022-01-03 13:43:10 -07:00
Fletcher Haynes
e89c04acaf
Merge branch 'master' into fb901 2022-01-03 12:25:26 -08:00
Fletcher Haynes
4016a1d03d Fixed commenting in the gitlab CI file. 2022-01-03 12:21:32 -08:00
garrison.davis@molecula.com
fa9ae13363 Adds gauntlet testing framework for Samsung 
This adds the Terraform needed to create a gauntlet testing framework for a cluster that is a mirror of Samsung's. It is meant to be run once a day in CI via the GitLab scheduler.
 2022-01-03 12:15:07 -08:00
Ben Johnson
af9795aa1a Avoid panics in RBF debug tooling 2022-01-03 13:13:02 -07:00
reese
080ea6ad2e
Merge pull request #1806 from molecula/percentile-timestamp-decimal 
FB-1095 implement percentiles on timestamp/decimal
 2022-01-03 12:26:05 -06:00
reesporte
b13538e426 update doc comment 2022-01-03 11:16:14 -06:00
reesporte
72adb177ae Merge branch 'master' into percentile-timestamp-decimal 2022-01-03 11:12:23 -06:00
reesporte
6e3ce01ecb explicitly test that getScaledInt works with timestamps 2022-01-03 11:11:54 -06:00
reesporte
fa2391b948 explicitly test untested path of valcountize 2022-01-03 11:11:27 -06:00
reesporte
99f6a1c113 change min to val 
bc it could be used for things besides mins
 2022-01-03 10:45:23 -06:00
Fletcher Haynes
93b97b9831 Test push to see if pipeline is running on push to master 2021-12-30 17:00:59 -08:00
Travis Turner
56b9e2aba7
Merge pull request #1831 from molecula/tlt/ignore-down 
Stop blocking API called when cluster is DOWN or DEGRADED
 2021-12-28 14:14:29 -06:00
Travis
ffd91137e1
Stop blocking API called when cluster is DOWN or DEGRADED 
This commit effectively removes the API-level validation that was
blocking certain API methods when the cluster was in a particular state
(namely DOWN and DEGRADED). The thinking is that we shouldn't be
blocking these requests at the API level, but rather should let them
pass through and allow the fact that a node is ACTUALLY down dictate the
behavior.

With this change, two tests were modified. They were previously
expecting the error message from the API validation on DOWN, but now
they check for a "shard unavailable" error, which is what gets returned
for a particular query when the cluster is in an unhealthy state.
 2021-12-28 13:52:04 -06:00
Matthew Jaffee
6fd985c8eb
Merge pull request #1828 from molecula/errant-print 
fix retry period and change client DialTimeout for commands (e.g. restore/backup)
 2021-12-28 13:51:28 -06:00
Matthew Jaffee
1a8c10d5f3 fix backup fail test so it actually fails 
A few things were going wrong here.

First, we take a "RetryPeriod" option on backup and restore which is
meant to be roughly the total amount of time we spend retrying any
given request before failing. However we were incorrectly passing that
as the RetryMaxWait which is the maximum amount of time to sleep
between any two attempts. We now do some fuzzy math to figure out
approximately how many attempts we should make given a minimum sleep
of 100ms and the fact that we double the sleep time every attempt.

Second, during the backup test, if a host was totally stopped when we
started the request, it would fail immediately and then retry, but if
the host was stopped during the request (after DNS had resolved), then
the request would wait for the DialTimeout which we default to 30s, so
turning off the cluster for 5 seconds and turning it back on resulted
in the backup completing rather than failing. Because of this, we
change the commandClient to have a default dial timeout of 1 second.

I was tempted to change the global default to 1s which I think would
be fine, but didn't want to break anything too badly.
 2021-12-28 13:31:42 -06:00
Matthew Jaffee
fe54cbf8ae remove other print and tweak backup test timings 2021-12-28 13:31:42 -06:00
Matthew Jaffee
bb39b05d05 remove leftover fmt.Println 2021-12-28 13:31:42 -06:00
Ben Johnson
f8e13f9629
Merge pull request #1829 from molecula/metrics 
Add job & worker metrics
 2021-12-28 11:31:13 -07:00
Ben Johnson
310584b0d8 Add job & worker metrics 2021-12-28 10:01:18 -07:00
Travis Turner
145f65ab0e
Merge pull request #1827 from molecula/tlt/etcd-data-dir 
[FB-1125] Expose `etcd.dir` configuration option
 2021-12-27 11:42:59 -06:00
Travis
6638fa17ee
Expose etcd.dir configuration option 
The goal is to allow a user to separate FeatureBase and etcd I/O.
 2021-12-27 11:13:38 -06:00
Ben Johnson
4a5179f87f
Merge pull request #1824 from molecula/rbf-expvar 2021-12-27 10:02:37 -07:00
Ben Johnson
9367a62609 Add /debug/rbf endpoint for debugging 2021-12-27 09:34:43 -07:00
Matthew Jaffee
f1b525fc74
Merge pull request #1807 from molecula/backup-http-retry 
add exponential retry logic to internal http client, use in backup and restore
 2021-12-22 13:00:24 -06:00
Matthew Jaffee
295fab4892 retry on >= 400, not just greater. good catch 2021-12-22 12:21:11 -06:00
Matthew Jaffee
ea59f14d50 must use retryablehttp.NewClient to get defaults 
otherwise it won't actually retry :(
 2021-12-22 11:21:11 -06:00
Matthew Jaffee
640ba45129 use retryableHTTP in client, fix memory usage of restore 
instead of awkwardly reading an entire file into a buffer, we use
retryablehttp's reader func to open the file fresh if we need to
retry, so a small fixed-size buffer can be used internally for copying
the contents onto the network.
 2021-12-22 10:56:16 -06:00
Seebs
63d8686e22 scratch space -- need to finish updating deployPerf though 2021-12-21 16:20:25 -07:00
Seebs
b4304765e8 partial draft of perf/regression test script 2021-12-21 16:20:25 -07:00
Matthew Jaffee
cde3f6b5ea add profiling to backup/restore 2021-12-21 16:24:21 -06:00
Matthew Jaffee
2bce396445 add retry restore test and custom retry policy 2021-12-21 16:24:21 -06:00
Matthew Jaffee
d3b9193c8d try to fix data race with http lib 
WARNING: DATA RACE
Write at 0x00c008121e80 by goroutine 235:
  bytes.(*Reader).WriteTo()
      /usr/local/go/src/bytes/reader.go:139 +0x45
  github.com/molecula/featurebase/v2/http.nopCloser.WriteTo()
      <autogenerated>:1 +0x5d
  io.copyBuffer()
      /usr/local/go/src/io/io.go:391 +0x482
  io.Copy()
      /usr/local/go/src/io/io.go:368 +0x78
  net/http.(*transferWriter).doBodyCopy()
      /usr/local/go/src/net/http/transfer.go:400 +0x2f
  net/http.(*transferWriter).writeBody()
      /usr/local/go/src/net/http/transfer.go:364 +0xc9a
  net/http.(*Request).write()
      /usr/local/go/src/net/http/request.go:682 +0x887
  net/http.(*persistConn).writeLoop()
      /usr/local/go/src/net/http/transport.go:2343 +0x349

Previous write at 0x00c008121e80 by goroutine 192:
  bytes.(*Reader).Seek()
      /usr/local/go/src/bytes/reader.go:118 +0x824
  github.com/molecula/featurebase/v2/http.(*InternalClient).doWithRetry()
      /go/src/github.com/molecula/featurebase/http/client.go:1773 +0x86d
  github.com/molecula/featurebase/v2/http.(*InternalClient).executeRequest()
      /go/src/github.com/molecula/featurebase/http/client.go:1806 +0x15b
  github.com/molecula/featurebase/v2/http.(*InternalClient).CreateIndex()
      /go/src/github.com/molecula/featurebase/http/client.go:433 +0xbf8
  github.com/molecula/featurebase/v2/server_test.TestMain_Set_Quick.func1()
      /go/src/github.com/molecula/featurebase/server/server_test.go:64 +0x624
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1123 +0x202

Goroutine 235 (running) created at:
  net/http.(*Transport).dialConn()
      /usr/local/go/src/net/http/transport.go:1709 +0xc30
  net/http.(*Transport).dialConnFor()
      /usr/local/go/src/net/http/transport.go:1421 +0x151

Goroutine 192 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1168 +0x5bb
  github.com/molecula/featurebase/v2/server_test.TestMain_Set_Quick()
      /go/src/github.com/molecula/featurebase/server/server_test.go:45 +0x116
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1123 +0x202
 2021-12-21 16:24:21 -06:00
Matthew Jaffee
f676fbfc51 add retryability to restore command 2021-12-21 16:24:20 -06:00
Matthew Jaffee
3105a24542 rewind Body on retry 
this is really not ideal, and there are libraries for this kind of
thing, but I'd have to figure out how to make the libraries work with
everywhere we're already creating stdlib http clients.
 2021-12-21 16:24:20 -06:00
Matthew Jaffee
cdf4bc4c88 add clustertests testing backup's retry 2021-12-21 16:24:20 -06:00
Matthew Jaffee
8486efaa79 add exponential retry logic to internal http client, use in backup 2021-12-21 16:24:20 -06:00
Garrison Davis
d9dc613dd5
Merge pull request #1820 from molecula/pipeline-changes 
Run integration on merge to default branch
 2021-12-21 14:50:29 -07:00
garrison.davis@molecula.com
15612b8b92 Run integration on merge to default branch 
Additionally, the go version is using the GOVERSION build
variable instead.
 2021-12-21 13:58:10 -07:00
Matthew Jaffee
18d2ef3d53
Merge pull request #1818 from molecula/fb1107 
[FB-1107] slightly better lock protection around bitDepth in view
 2021-12-20 15:31:13 -06:00