scriptoriumadmin/featurebase
1
0
Fork 0
mirror of https://github.com/featurebasedb/featurebase.git synced 2026-10-09 04:17:51 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
No description
7779 commits 45 branches 174 tags 85 MiB
Find a file
Seebs 9f271467fb ingest cluster support 
We add endpoints and protobuf encode/decode to allow for sending
sharded requests over the wire in protobuf, so we can take our
sharded data and send it to other nodes if needed.

This is a squash of >15 other commits, so a bit of history
is relevant:

The Request type had FieldTypes in it because the field type
information was needed for sharding because sorting requires
that information. We change this around to make the external
sharding operation require the field types, and curry that
through the codec -- the codec is needed to tell the request
how it shards. (This is because the correct sorting order
varies by field type.) Requests (and ShardedRequests) no
longer have that table in them.

And then we hit a nasty bug in production and RCA showed
that our testing wasn't good enough and we need to be more
careful, and I discovered that test coverage in this package
was around 70%.

So, the other big thing here is coverage testing; in order to
make coverage testing viable and programmatically testable,
we have added the ability to render requests *back* to
JSON. This is not a great idea, but it does allow us to do
a lot of sanity-checking and verify that the encodings we're
using are consistent and correct.

This, plus some specific tests of decoding specific flawed
inputs, has caught a number of issues. Which are now fixed!

A lot of internal API surface got slightly changed, in ways
that make it simpler to work with. For instance, the
(*FieldOperation).TranslateUnsigned function doesn't really
need to exist; we can just have a non-method translate
function for unsigned and for signed, and use them based on
field type.

The stable translation hack used for testing had a bug that
could allow it to end up producing incorrect results if you
asked it to translate an ID first rather than exclusively
asking it to translate strings first, this has been
corrected. (This is a bug fix in code that was added
partway through creating this, but is tricky enough to
mention its own comment.)

Test coverage is now just over 90%, and a lot of what's left
is error-check returns that may well be actually unreachable
unless, say, the documentation for encoding/json is full of
lies. Which it probably is.
2021-09-27 12:05:57 -05:00
.circleci Rename pilosa to featurebase 2021-09-23 10:16:14 -05:00
.github FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
api/client FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
boltdb FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
client CORE-747: Build tooling to provide datagen-like functionality that uses the API 2021-09-02 10:18:50 -04:00
cmd Implement non-aggregate SELECT query 2021-09-08 08:09:54 -06:00
ctl added sqlversion config option 2021-09-25 13:53:19 -05:00
debugstats introduce storage.Config 2021-01-20 22:05:38 -06:00
disco replace roaring.Bitmap by [][]byte 2021-05-10 18:45:23 +02:00
encoding/proto ingest cluster support 2021-09-27 12:05:57 -05:00
etcd FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
gcnotify FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
generator Address TODOs in roaring tests 2020-05-21 13:28:54 -05:00
gopsutil FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
hash FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
http ingest cluster support 2021-09-27 12:05:57 -05:00
ingest ingest cluster support 2021-09-27 12:05:57 -05:00
ingest_testdata ingest and ingest/codec testing work 2021-08-19 09:50:59 -05:00
internal Make index-key replication more resilient to network failures 2021-09-14 03:12:52 +03:00
lattice Update aggregate label to cardinality 2021-08-02 19:01:37 -05:00
logger FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
lru so much lint 2019-04-16 12:07:18 -05:00
mock FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
net Address PR comments 2021-03-24 18:09:01 +01:00
pb Add max memory limit to Extract() to prevent OOM 2021-08-02 08:20:12 -06:00
pg cleanup and applied review suggestions 2021-09-27 07:05:08 -05:00
pql FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
prometheus FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
proto FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
rbf differentiate error messages between opening file and mapping it 2021-08-09 11:42:25 -05:00
roaring test intersectionCallback more carefully 2021-09-16 14:05:35 -05:00
scripts Fix benchmark file naming 2021-05-21 14:51:08 -06:00
server silence and rebase 2021-09-25 14:06:55 -05:00
shardwidth make "subdivide list by shardwidth" available for reuse 2021-08-18 13:45:20 -05:00
short_txkey pilosa: only open views with data 2020-12-19 00:03:31 +00:00
sql basic looker connection tests pass 2021-09-25 13:51:02 -05:00
sql2 Add SQL type checking 2021-09-24 14:02:20 -06:00
statik FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
stats FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
statsd FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
storage change default backend to RBF, remove separate RBF race target 2021-03-02 17:50:56 -06:00
synthload FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
syswrap better explanation for 'cannot allocate memory' error 2021-09-23 08:32:12 -05:00
test don't use nil qcx 2021-08-18 13:45:36 -05:00
testdata Test cases for Store(Distinct) 2020-09-14 14:15:29 -05:00
testhook FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
toml Add license headers to files missing them and CI check to verify they are present. Fixes #1633 2019-04-12 11:30:41 -05:00
topology FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
tracing FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
txkey pilosa: only open views with data 2020-12-19 00:03:31 +00:00
vprint Move vprint to separate pakage 2021-03-29 14:29:19 +02:00
.dockerignore Add new Docker build process and continuous delivery 2021-02-26 14:34:14 -06:00
.gitignore add .idea to .gitignore 2021-05-24 12:18:56 -07:00
.golangci.yml fix golangci-lint complaints 2019-11-10 17:58:46 -06:00
api.go ingest cluster support 2021-09-27 12:05:57 -05:00
api_test.go add clusters to MutexCheck test, fix silly bug revealed by doing so 2021-09-13 16:25:15 -05:00
apimethod_string.go remove attributes 2021-05-14 10:28:08 -04:00
audit.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
audit_internal_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
audit_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
barrier.go Remove lmdb dependency and references, vendor Barrier 2020-12-09 08:43:23 -06:00
barrier_test.go Remove lmdb dependency and references, vendor Barrier 2020-12-09 08:43:23 -06:00
bluegreentx.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
bluegreentx_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
bolt.go fix error formatting/spelling 2021-08-18 13:45:36 -05:00
bolt_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
broadcast.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
bsi.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
bsi_test.go introduce "fields" directory between index and field 2021-03-05 16:56:44 -06:00
cache.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
cache_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
catcher.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
CHANGELOG.md Fix typo 2019-09-17 16:01:21 -05:00
client.go ingest cluster support 2021-09-27 12:05:57 -05:00
cluster.go ingest cluster support 2021-09-27 12:05:57 -05:00
cluster_internal_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
cmd.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
const_amd64.go remove lmdb as a Tx backend 2020-12-04 23:17:59 +00:00
const_other.go remove lmdb as a Tx backend 2020-12-04 23:17:59 +00:00
CONTRIBUTING.md simplify contributing instructions by removing weird upstream thing 2019-05-01 17:37:52 -05:00
dbshard.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
dbshard_internal_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
dbshard_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
delete_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
diagnostics.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
diagnostics_internal_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
doc.go Apply Apache License 2.0 2017-04-28 14:22:03 -05:00
docker-compose-3.yml More rename to work with backup and cluster tests 2021-09-22 15:41:17 -05:00
Dockerfile More rename to work with backup and cluster tests 2021-09-22 15:41:17 -05:00
Dockerfile-clustertests docker fix 2021-09-22 15:35:52 -05:00
Dockerfile.pilosa Remove prints 2021-09-22 16:02:02 -05:00
Dockerfile.runner Rename docker builds to featurebase 2021-09-22 15:41:00 -05:00
event.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
executor.go CORE-777: Added DecimalAgg field in GroupCount to output decimal sum 2021-09-16 09:43:13 -05:00
executor_internal_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
executor_test.go CORE-777: Added DecimalAgg field in GroupCount to output decimal sum 2021-09-16 09:43:13 -05:00
field.go tweak to locking which should avoid stall/deadlock w/ mutex check 2021-09-16 14:07:31 -05:00
field_internal_test.go mutex sanity-check 2021-09-07 12:41:49 -05:00
field_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
filesystem.go Add missing file 2020-08-31 09:02:46 -05:00
fragment.go make details optional and support limits on mutex checks 2021-09-08 11:59:47 -05:00
fragment_internal_test.go improve mutex import testing 2021-08-30 14:16:15 -05:00
gc.go fix some lint warnings raised in VS-Code 2019-04-17 18:10:05 -05:00
gendebug_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
generation.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
generation_debug.go testhook: leak auditing infrastructure 2020-08-24 11:26:39 -05:00
generation_nodebug.go Sources and Generations: tracking mmapped files 2019-11-12 12:14:29 -06:00
generation_test.go Tx integration milestone 2020-07-27 19:29:46 -04:00
gid.go database per shard, HolderConfig, rbf bit-wise import speedups. 2020-09-04 13:00:33 -05:00
go.mod tidy 2021-09-27 08:20:03 -05:00
go.sum removed extra command complete message 2021-09-25 13:53:19 -05:00
hack.go fix build issue 2021-07-19 13:47:47 -07:00
handler.go make import correctly reflect that it needs a single shard always 2021-08-18 13:45:36 -05:00
holder.go Make index-key replication more resilient to network failures 2021-09-14 03:12:52 +03:00
holder_internal_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
holder_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
idalloc.go restore idalloc 2021-05-21 09:27:08 -05:00
idalloc_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
index.go make import correctly reflect that it needs a single shard always 2021-08-18 13:45:36 -05:00
index_internal_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
index_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
ingest_test.go ingest cluster support 2021-09-27 12:05:57 -05:00
iterator.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
iterator_internal_test.go rename internal_tests. fix license header 2018-07-04 10:59:08 -05:00
LICENSE Apply Apache License 2.0 2017-04-28 14:22:03 -05:00
license.exceptions update license header check exceptions 2021-05-14 10:45:46 -04:00
like.go execute like queries on the primary's key translation database 2021-05-27 14:45:10 -04:00
like_test.go execute like queries on the primary's key translation database 2021-05-27 14:45:10 -04:00
main_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
Makefile Rename in docker-tag-push 2021-09-22 15:41:09 -05:00
metrics.go remove attributes 2021-05-14 10:28:08 -04:00
mmap_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
NOTES rename slice to shard 2018-06-28 14:07:07 -05:00
NOTICE Remove a few more enterprise references 2020-06-26 15:57:21 -05:00
pilosa.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
pilosa_internal_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
pilosa_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
pjobs.go run migration in parallel 2020-11-05 18:31:27 +00:00
pjobs_test.go run migration in parallel 2020-11-05 18:31:27 +00:00
planner.go cleanup and applied review suggestions 2021-09-27 07:05:08 -05:00
planner_test.go Fix SQL column mappings 2021-09-25 09:36:11 -06:00
pprof.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
rbf.go fix error formatting/spelling 2021-08-18 13:45:36 -05:00
README.md FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
row.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
row_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
rrtx.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
rrtx_internal_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
serializer.go add a gob-encoding Serializer implementation for tests 2021-02-23 10:09:55 -06:00
server.go docker fix not really related to anything 2021-09-25 13:53:19 -05:00
server_internal_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
snapshotqueue.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
stattx.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
testBackupRestore.sh More rename to work with backup and cluster tests 2021-09-22 15:41:17 -05:00
time.go Prototype ingest API 2021-08-19 09:50:59 -05:00
time_internal_test.go avoid allocations in viewsByTime 2021-08-18 13:45:36 -05:00
tournament.sh remove instances of os.Getenv("PILOSA_TXSRC") 2021-01-21 21:18:46 -06:00
tracker.go ingest and ingest/codec testing work 2021-08-19 09:50:59 -05:00
tracker_test.go Update tests 2021-01-20 10:48:35 -06:00
transaction.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
transaction_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
translate.go ingest cluster support 2021-09-27 12:05:57 -05:00
translator_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
tx.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
tx_internal_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
tx_test.go make import correctly reflect that it needs a single shard always 2021-08-18 13:45:36 -05:00
txfactory.go fix error formatting/spelling 2021-08-18 13:45:36 -05:00
txfactory_internal_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
util.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
utils_internal_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
version.go FeatureBase renaming: version info in server log 2021-07-12 12:25:43 -07:00
view.go tweak to locking which should avoid stall/deadlock w/ mutex check 2021-09-16 14:07:31 -05:00
view_internal_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00

README.md

FeatureBase, a distributed bitmap index

See our internal documentation, which includes all external documentation, plus many internal-only pages, listed under the "Internal" heading in the main navigation bar.

Follow along with the Sample Project to get a better understanding of FeatureBase's capabilities.