scriptoriumadmin/featurebase
1
0
Fork 0
mirror of https://github.com/featurebasedb/featurebase.git synced 2026-08-28 10:54:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
8515 commits 45 branches 174 tags 85 MiB
Commit graph

8515 commits

This branch
This branch
All branches
Author SHA1 Message Date
reesporte
be66103c45 requirements when auth is enabled 
postgres binding is turned off
TLS must be turned on
 2021-12-29 11:20:19 -06:00
reesporte
2847c22a4c linter things 2021-12-29 11:06:58 -06:00
reesporte
42f3557c55 fix merge conflicts 2021-12-29 09:05:53 -06:00
Travis Turner
56b9e2aba7
Merge pull request #1831 from molecula/tlt/ignore-down 
Stop blocking API called when cluster is DOWN or DEGRADED
 2021-12-28 14:14:29 -06:00
Travis
ffd91137e1
Stop blocking API called when cluster is DOWN or DEGRADED 
This commit effectively removes the API-level validation that was
blocking certain API methods when the cluster was in a particular state
(namely DOWN and DEGRADED). The thinking is that we shouldn't be
blocking these requests at the API level, but rather should let them
pass through and allow the fact that a node is ACTUALLY down dictate the
behavior.

With this change, two tests were modified. They were previously
expecting the error message from the API validation on DOWN, but now
they check for a "shard unavailable" error, which is what gets returned
for a particular query when the cluster is in an unhealthy state.
 2021-12-28 13:52:04 -06:00
Matthew Jaffee
6fd985c8eb
Merge pull request #1828 from molecula/errant-print 
fix retry period and change client DialTimeout for commands (e.g. restore/backup)
 2021-12-28 13:51:28 -06:00
Matthew Jaffee
1a8c10d5f3 fix backup fail test so it actually fails 
A few things were going wrong here.

First, we take a "RetryPeriod" option on backup and restore which is
meant to be roughly the total amount of time we spend retrying any
given request before failing. However we were incorrectly passing that
as the RetryMaxWait which is the maximum amount of time to sleep
between any two attempts. We now do some fuzzy math to figure out
approximately how many attempts we should make given a minimum sleep
of 100ms and the fact that we double the sleep time every attempt.

Second, during the backup test, if a host was totally stopped when we
started the request, it would fail immediately and then retry, but if
the host was stopped during the request (after DNS had resolved), then
the request would wait for the DialTimeout which we default to 30s, so
turning off the cluster for 5 seconds and turning it back on resulted
in the backup completing rather than failing. Because of this, we
change the commandClient to have a default dial timeout of 1 second.

I was tempted to change the global default to 1s which I think would
be fine, but didn't want to break anything too badly.
 2021-12-28 13:31:42 -06:00
Matthew Jaffee
fe54cbf8ae remove other print and tweak backup test timings 2021-12-28 13:31:42 -06:00
Matthew Jaffee
bb39b05d05 remove leftover fmt.Println 2021-12-28 13:31:42 -06:00
Hoang Pham
936fb9e6bd UI - rename a unit test 2021-12-28 13:04:56 -06:00
Ben Johnson
f8e13f9629
Merge pull request #1829 from molecula/metrics 
Add job & worker metrics
 2021-12-28 11:31:13 -07:00
Ben Johnson
310584b0d8 Add job & worker metrics 2021-12-28 10:01:18 -07:00
Hoang Pham
0ad2bffd33 UI - added test files 2021-12-28 10:56:31 -06:00
Travis Turner
145f65ab0e
Merge pull request #1827 from molecula/tlt/etcd-data-dir 
[FB-1125] Expose `etcd.dir` configuration option
 2021-12-27 11:42:59 -06:00
Travis
6638fa17ee
Expose etcd.dir configuration option 
The goal is to allow a user to separate FeatureBase and etcd I/O.
 2021-12-27 11:13:38 -06:00
Ben Johnson
4a5179f87f
Merge pull request #1824 from molecula/rbf-expvar 2021-12-27 10:02:37 -07:00
Ben Johnson
9367a62609 Add /debug/rbf endpoint for debugging 2021-12-27 09:34:43 -07:00
Samir Patel
9bf56188f7 Merge branch '54mir/authentication' of github.com:molecula/featurebase into 54mir/authentication 2021-12-22 14:08:21 -06:00
Samir Patel
8b40c6bf7b rm comments 2021-12-22 13:47:16 -06:00
Samir Patel
448289d609 add subcommand for key generation 2021-12-22 13:04:00 -06:00
Matthew Jaffee
f1b525fc74
Merge pull request #1807 from molecula/backup-http-retry 
add exponential retry logic to internal http client, use in backup and restore
 2021-12-22 13:00:24 -06:00
Matthew Jaffee
295fab4892 retry on >= 400, not just greater. good catch 2021-12-22 12:21:11 -06:00
Samir Patel
23a884b3a2 Merge branch '54mir/authentication' of github.com:molecula/featurebase into 54mir/authentication 2021-12-22 12:05:41 -06:00
Samir Patel
b8b4425d4f clean up 2021-12-22 12:05:24 -06:00
Hoang Pham
39ebbbd88e UI - formatted tsx files with prettier (single quote) 2021-12-22 11:40:01 -06:00
Samir Patel
9612096682 Merge branch '54mir/authentication' of github.com:molecula/featurebase into 54mir/authentication 2021-12-22 11:31:56 -06:00
Samir Patel
52d941d127 more tests 2021-12-22 11:31:37 -06:00
Matthew Jaffee
ea59f14d50 must use retryablehttp.NewClient to get defaults 
otherwise it won't actually retry :(
 2021-12-22 11:21:11 -06:00
Matthew Jaffee
640ba45129 use retryableHTTP in client, fix memory usage of restore 
instead of awkwardly reading an entire file into a buffer, we use
retryablehttp's reader func to open the file fresh if we need to
retry, so a small fixed-size buffer can be used internally for copying
the contents onto the network.
 2021-12-22 10:56:16 -06:00
Hoang Pham
c67ae54b1d UI - fixed Sign In button 2021-12-21 16:44:27 -06:00
Matthew Jaffee
cde3f6b5ea add profiling to backup/restore 2021-12-21 16:24:21 -06:00
Matthew Jaffee
2bce396445 add retry restore test and custom retry policy 2021-12-21 16:24:21 -06:00
Matthew Jaffee
d3b9193c8d try to fix data race with http lib 
WARNING: DATA RACE
Write at 0x00c008121e80 by goroutine 235:
  bytes.(*Reader).WriteTo()
      /usr/local/go/src/bytes/reader.go:139 +0x45
  github.com/molecula/featurebase/v2/http.nopCloser.WriteTo()
      <autogenerated>:1 +0x5d
  io.copyBuffer()
      /usr/local/go/src/io/io.go:391 +0x482
  io.Copy()
      /usr/local/go/src/io/io.go:368 +0x78
  net/http.(*transferWriter).doBodyCopy()
      /usr/local/go/src/net/http/transfer.go:400 +0x2f
  net/http.(*transferWriter).writeBody()
      /usr/local/go/src/net/http/transfer.go:364 +0xc9a
  net/http.(*Request).write()
      /usr/local/go/src/net/http/request.go:682 +0x887
  net/http.(*persistConn).writeLoop()
      /usr/local/go/src/net/http/transport.go:2343 +0x349

Previous write at 0x00c008121e80 by goroutine 192:
  bytes.(*Reader).Seek()
      /usr/local/go/src/bytes/reader.go:118 +0x824
  github.com/molecula/featurebase/v2/http.(*InternalClient).doWithRetry()
      /go/src/github.com/molecula/featurebase/http/client.go:1773 +0x86d
  github.com/molecula/featurebase/v2/http.(*InternalClient).executeRequest()
      /go/src/github.com/molecula/featurebase/http/client.go:1806 +0x15b
  github.com/molecula/featurebase/v2/http.(*InternalClient).CreateIndex()
      /go/src/github.com/molecula/featurebase/http/client.go:433 +0xbf8
  github.com/molecula/featurebase/v2/server_test.TestMain_Set_Quick.func1()
      /go/src/github.com/molecula/featurebase/server/server_test.go:64 +0x624
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1123 +0x202

Goroutine 235 (running) created at:
  net/http.(*Transport).dialConn()
      /usr/local/go/src/net/http/transport.go:1709 +0xc30
  net/http.(*Transport).dialConnFor()
      /usr/local/go/src/net/http/transport.go:1421 +0x151

Goroutine 192 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1168 +0x5bb
  github.com/molecula/featurebase/v2/server_test.TestMain_Set_Quick()
      /go/src/github.com/molecula/featurebase/server/server_test.go:45 +0x116
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1123 +0x202
 2021-12-21 16:24:21 -06:00
Matthew Jaffee
f676fbfc51 add retryability to restore command 2021-12-21 16:24:20 -06:00
Matthew Jaffee
3105a24542 rewind Body on retry 
this is really not ideal, and there are libraries for this kind of
thing, but I'd have to figure out how to make the libraries work with
everywhere we're already creating stdlib http clients.
 2021-12-21 16:24:20 -06:00
Matthew Jaffee
cdf4bc4c88 add clustertests testing backup's retry 2021-12-21 16:24:20 -06:00
Matthew Jaffee
8486efaa79 add exponential retry logic to internal http client, use in backup 2021-12-21 16:24:20 -06:00
Hoang Pham
2bf1f0e4d4 Merge branch '54mir/authentication' of ssh://github.com/molecula/featurebase into 54mir/authentication 2021-12-21 16:06:11 -06:00
Hoang Pham
7de6c37b11 UI - cleaned up code, added comments 2021-12-21 16:06:02 -06:00
Garrison Davis
d9dc613dd5
Merge pull request #1820 from molecula/pipeline-changes 
Run integration on merge to default branch
 2021-12-21 14:50:29 -07:00
Samir Patel
fd7d905be2 fix formatting issues 2021-12-21 15:49:51 -06:00
garrison.davis@molecula.com
15612b8b92 Run integration on merge to default branch 
Additionally, the go version is using the GOVERSION build
variable instead.
 2021-12-21 13:58:10 -07:00
Samir Patel
f011587d4e add tests 2021-12-21 10:08:49 -06:00
Samir Patel
1c907281bf authz changes 2021-12-20 18:03:18 -06:00
Samir Patel
10a7aa55ea same-site strict 2021-12-20 18:00:13 -06:00
Samir Patel
d9710cb7d7 remove auth struct from authorization 2021-12-20 16:45:17 -06:00
Samir Patel
3b374a62bf Merge branch 'master' into 54mir/authentication 2021-12-20 16:42:11 -06:00
Hoang Pham
a24a1e8922 UI - added fixes for PR comments 2021-12-20 15:55:52 -06:00
Samir Patel
602145b390 add to tests 2021-12-20 15:41:18 -06:00
Matthew Jaffee
18d2ef3d53
Merge pull request #1818 from molecula/fb1107 
[FB-1107] slightly better lock protection around bitDepth in view
 2021-12-20 15:31:13 -06:00