scriptoriumadmin/featurebase
1
0
Fork 0
mirror of https://github.com/featurebasedb/featurebase.git synced 2026-08-28 10:54:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
No description
9124 commits 45 branches 174 tags 85 MiB
Find a file
Seebs 32fec70816 track closing status for index/field/view, shut down cache flush early 
This is a lot more complex than it sounds like it will be.

We shut down the cache flush when a holder is closed, but if you're
deleting an index, we don't check for that, and can have a cache flush
still creating cache files in an index which could conceivably result
in os.RemoteAll() failing. This shouldn't happen often, but it's happened
at least once.

To address this, first, we make sure that every tier of this operation
bails as quickly as it can after the thing it's working on closes. Second,
we retry RemoveAll.

Unfortunately, some things get reopened, so we have to handle that,
have mutexes covering the access to the channel, and so on. Also, some
things were getting double-closed, which was previously harmless but
could now fail. So, first, catch all the existing double-closes and
remove them, second, make the double-close fail with an error. Note
that virtually none of the tests check for errors on close.

This passes tests and should be unable to hit the original problem.
Unfortunately, it's unreasonably hard to check that, because it
requires an incredible coincidence of timing on the delete aligning
with a cache flush.
2022-03-18 14:41:44 -05:00
.github remove references to CONTRIBUTING and CHANGELOG 2021-11-19 11:22:15 -06:00
.gitlab use Go 1.18 for future tests 2022-03-17 16:08:39 -05:00
api/client Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
authn fb1172: enable refresh tokens 2022-02-07 13:42:11 -06:00
authz Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
boltdb use free id bucket to re-use ids 2022-02-28 15:56:58 -06:00
client client side retry ingestAPI requests on primary host 2022-03-10 10:22:13 -06:00
cmd FB-1188 - improved description of TTL 2022-03-07 11:16:33 -06:00
ctl listen on localhost:0 instead of :0 2022-03-18 13:38:30 -05:00
debugstats add copyright notice back in 2021-12-10 11:01:04 -06:00
disco requested idx == nil fix; add doc comment 2022-02-14 10:09:33 -08:00
encoding/proto FB-1188 - removed parseDuration error logs in proto.go 2022-03-07 11:45:29 -06:00
etcd listen on localhost:0 instead of :0 2022-03-18 13:38:30 -05:00
gcnotify Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
generator add copyright notice back in 2021-12-10 11:01:04 -06:00
gopsutil Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
hash Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
ingest Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
ingest_testdata ingest and ingest/codec testing work 2021-08-19 09:50:59 -05:00
install rip out ui/usage 2022-02-28 12:05:44 -06:00
internal Merge branch 'master' of github.com:molecula/featurebase into clustertest-retry-coverage-v2 2022-03-10 16:54:30 -06:00
lattice Revert auto-quoting in Web UI 2022-03-03 12:35:54 -07:00
logger Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
lru fix some staticcheck issues 2021-12-03 09:31:45 -06:00
mock Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
net add copyright notice back in 2021-12-10 11:01:04 -06:00
pb FB-1188 - Added TTL field option 2022-02-25 11:39:31 -06:00
pg listen on localhost:0 instead of :0 2022-03-18 13:38:30 -05:00
pql catch the panic we throw for an invalid timestamp 2022-02-22 16:20:09 -06:00
prometheus Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
proto Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
qa WIP: start integration test setup 2022-03-15 14:43:06 -05:00
rbf listen on localhost:0 instead of :0 2022-03-18 13:38:30 -05:00
roaring go fmt 2022-03-10 09:03:07 -06:00
scripts Fix benchmark file naming 2021-05-21 14:51:08 -06:00
server listen on localhost:0 instead of :0 2022-03-18 13:38:30 -05:00
shardwidth Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
short_txkey remove a bunch of commented print statements and unecessary prints 2022-01-11 10:42:44 -06:00
sql Use Distinct() call for SQL DISTINCT 2022-03-01 15:04:32 -07:00
sql2 Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
statik Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
stats only set bits after the holder is completely setup 2022-02-25 15:02:24 -06:00
statsd Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
storage remove a bunch of roaring backend stuff 2022-02-02 20:56:18 -06:00
syswrap add copyright notice back in 2021-12-10 11:01:04 -06:00
task implement a task pool 2022-02-14 09:56:19 -06:00
test listen on localhost:0 instead of :0 2022-03-18 13:38:30 -05:00
testdata Test cases for Store(Distinct) 2020-09-14 14:15:29 -05:00
testhook Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
toml add copyright notice back in 2021-12-10 11:01:04 -06:00
topology Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
tracing Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
txkey remove a bunch of commented print statements and unecessary prints 2022-01-11 10:42:44 -06:00
vprint add copyright notice back in 2021-12-10 11:01:04 -06:00
.dockerignore Add new Docker build process and continuous delivery 2021-02-26 14:34:14 -06:00
.gitignore removed tfstate.backup files 2022-02-24 10:24:27 -06:00
.golangci.yml fix govet and gofmt errors in existing code 2021-10-29 13:14:27 -05:00
api.go add server side redirect to primary for ingest requests 2022-03-10 10:22:13 -06:00
api_test.go Merge branch 'master' into fb-1188-ttl 2022-03-04 11:30:46 -06:00
apimethod_string.go FB-1188 - Added TTL field option 2022-02-25 11:39:31 -06:00
audit.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
audit_internal_test.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
audit_test.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
broadcast.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
bsi.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
bsi_test.go add copyright notice back in 2021-12-10 11:01:04 -06:00
cache.go Periodically invalidate rank cache during bulk add 2022-02-15 08:25:45 -07:00
cache_test.go Periodically invalidate rank cache during bulk add 2022-02-15 08:25:45 -07:00
catcher.go remove string keys on delete to allow for reuse 2022-02-25 16:23:54 -06:00
cluster.go remove http subpackage and bring implementations into core 2022-02-03 21:04:04 -06:00
cluster_internal_test.go track closing status for index/field/view, shut down cache flush early 2022-03-18 14:41:44 -05:00
cmd.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
const_amd64.go add copyright notice back in 2021-12-10 11:01:04 -06:00
const_other.go add copyright notice back in 2021-12-10 11:01:04 -06:00
dbshard.go remove unused stuff and fix a bunch of random staticcheck issues 2022-02-07 15:10:10 -06:00
dbshard_internal_test.go comment cleanup and removed long test 2022-02-25 16:40:04 -06:00
dbshard_test.go remove http subpackage and bring implementations into core 2022-02-03 21:04:04 -06:00
delete_test.go FB-1188 - added back disco import that was accidently removed from merge conflict 2022-03-03 16:23:55 -06:00
diagnostics.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
diagnostics_internal_test.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
doc.go add copyright notice back in 2021-12-10 11:01:04 -06:00
Dockerfile remove references to LICENSE and checks for it in source files 2021-11-19 10:38:06 -06:00
Dockerfile-clustertests bump go to go 1.17.7 2022-03-08 08:38:07 -06:00
Dockerfile-clustertests-client bump go to go 1.17.7 2022-03-08 08:38:07 -06:00
event.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
executor.go Merge branch 'master' into FB-1185 2022-03-14 14:30:37 -05:00
executor_internal_test.go track closing status for index/field/view, shut down cache flush early 2022-03-18 14:41:44 -05:00
executor_test.go track closing status for index/field/view, shut down cache flush early 2022-03-18 14:41:44 -05:00
field.go track closing status for index/field/view, shut down cache flush early 2022-03-18 14:41:44 -05:00
field_internal_test.go track closing status for index/field/view, shut down cache flush early 2022-03-18 14:41:44 -05:00
field_test.go track closing status for index/field/view, shut down cache flush early 2022-03-18 14:41:44 -05:00
filesystem.go add copyright notice back in 2021-12-10 11:01:04 -06:00
fragment.go implement a BSI-aware filter to avoid OffsetRange calls in fragment.sum 2022-02-28 14:41:30 -06:00
fragment_internal_test.go track closing status for index/field/view, shut down cache flush early 2022-03-18 14:41:44 -05:00
gc.go add copyright notice back in 2021-12-10 11:01:04 -06:00
gid.go database per shard, HolderConfig, rbf bit-wise import speedups. 2020-09-04 13:00:33 -05:00
go.mod go mod tidy 2022-03-01 10:54:53 -06:00
go.sum go mod tidy 2022-03-01 10:54:53 -06:00
hack.go hard code field options 2022-03-16 14:49:58 -05:00
handler.go remove http subpackage and bring implementations into core 2022-02-03 21:04:04 -06:00
holder.go track closing status for index/field/view, shut down cache flush early 2022-03-18 14:41:44 -05:00
holder_internal_test.go address reviewer's comments 2022-03-10 09:36:18 -06:00
holder_test.go track closing status for index/field/view, shut down cache flush early 2022-03-18 14:41:44 -05:00
http_handler.go ensure provided min/max are valid on int fields 2022-03-15 10:29:48 -05:00
http_handler_internal_test.go add /internal/disk-usage endpoint for testing 2022-03-14 15:23:15 -05:00
http_handler_test.go listen on localhost:0 instead of :0 2022-03-18 13:38:30 -05:00
http_translator.go remove http subpackage and bring implementations into core 2022-02-03 21:04:04 -06:00
http_translator_test.go remove http subpackage and bring implementations into core 2022-02-03 21:04:04 -06:00
idalloc.go add copyright notice back in 2021-12-10 11:01:04 -06:00
idalloc_test.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
index.go track closing status for index/field/view, shut down cache flush early 2022-03-18 14:41:44 -05:00
index_internal_test.go remove unused stuff and fix a bunch of random staticcheck issues 2022-02-07 15:10:10 -06:00
index_test.go track closing status for index/field/view, shut down cache flush early 2022-03-18 14:41:44 -05:00
ingest_test.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
internal_client.go Merge branch 'master' into fb-1188-ttl 2022-03-03 16:12:42 -06:00
internal_client_test.go FB-1188 - TTL - added more unit tests 2022-03-02 12:46:21 -06:00
iterator.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
iterator_internal_test.go add copyright notice back in 2021-12-10 11:01:04 -06:00
license.exceptions update license header check exceptions 2021-05-14 10:45:46 -04:00
like.go add copyright notice back in 2021-12-10 11:01:04 -06:00
like_test.go add copyright notice back in 2021-12-10 11:01:04 -06:00
main_test.go listen on localhost:0 instead of :0 2022-03-18 13:38:30 -05:00
Makefile bump go to go 1.17.7 2022-03-08 08:38:07 -06:00
metrics.go add copyright notice back in 2021-12-10 11:01:04 -06:00
nfpm.yaml FB-904: Add deb/rpm packaging of FeatureBase binary 2021-11-04 10:32:59 -06:00
NOTICE add config files for release 2021-11-19 01:26:18 +03:00
pilosa.go test multiple field creations at once on a cluster 2022-03-04 10:21:18 -06:00
pilosa_internal_test.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
pilosa_test.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
planner.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
planner_test.go track closing status for index/field/view, shut down cache flush early 2022-03-18 14:41:44 -05:00
pprof.go remove a bunch of roaring backend stuff 2022-02-02 20:56:18 -06:00
rbf.go optimize container after remove 2022-03-11 06:20:07 -06:00
README.md meaningless commit 2021-10-29 11:56:09 -05:00
row.go remove string keys on delete to allow for reuse 2022-02-25 16:23:54 -06:00
row_test.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
serializer.go add copyright notice back in 2021-12-10 11:01:04 -06:00
server.go FB-1188 - TTL - run TtlRemoval on server start, better ttl log info message 2022-03-11 11:37:36 -06:00
server_internal_test.go improve Server WaitGroup concurrent usage 2022-02-25 15:02:21 -06:00
server_test.go FB-1188 - added more unit tests 2022-03-03 16:09:37 -06:00
stattx.go remove string keys on delete to allow for reuse 2022-02-25 16:23:54 -06:00
time.go FB-1188 - added more unit tests 2022-03-03 16:09:37 -06:00
time_internal_test.go add copyright notice back in 2021-12-10 11:01:04 -06:00
tracker.go add copyright notice back in 2021-12-10 11:01:04 -06:00
tracker_test.go add copyright notice back in 2021-12-10 11:01:04 -06:00
transaction.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
transaction_test.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
translate.go remove string keys on delete to allow for reuse 2022-02-25 16:23:54 -06:00
translator_test.go remove http subpackage and bring implementations into core 2022-02-03 21:04:04 -06:00
tx.go Merge branch 'master' into tlt/tx-comments 2022-03-04 15:05:39 -06:00
tx_internal_test.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
tx_test.go remove string keys on delete to allow for reuse 2022-02-25 16:23:54 -06:00
txfactory.go . 2022-03-11 06:20:07 -06:00
txfactory_internal_test.go remove a bunch of roaring backend stuff 2022-02-02 20:56:18 -06:00
util.go add /internal/disk-usage endpoint for testing 2022-03-14 15:23:15 -05:00
util_test.go add /internal/disk-usage endpoint for testing 2022-03-14 15:23:15 -05:00
utils_internal_test.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
version.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
view.go track closing status for index/field/view, shut down cache flush early 2022-03-18 14:41:44 -05:00
view_internal_test.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00

README.md

FeatureBase, a distributed bitmap index

See our internal documentation, which includes all external documentation, plus many internal-only pages, listed under the "Internal" heading in the main navigation bar.

Follow along with the Sample Project to get a better understanding of FeatureBase's capabilities.