scriptoriumadmin/featurebase
1
0
Fork 0
mirror of https://github.com/featurebasedb/featurebase.git synced 2026-10-12 22:07:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
No description
7857 commits 45 branches 174 tags 85 MiB
Find a file
Seebs ad30a926f4 Giant Commit: drop a bunch of stuff we don't use. 
These commits are hard to disentagle, and doing them separately means
re-modifying the same chunks of code several times before removing it,
and similar things.

Basically:
(1) Drop the bolt backend storage.
(2) Drop the blue-green wrapper that compares two backends.
(3) Drop unused or barely-used Tx API components from all the
remaining backends.
(4) Minor related cleanup to simplify things related to these.

The boltdb backend existed only to verify RBF. The blue-green wrapper
was mostly used to verify RBF, but in practice we had to do a lot
of working around that, and it introduced a lot of special cases.

Types removed:

IteratorFinder: Used only to implement the roaring iterator
on top of boltdb, and to complicate the way it worked in roaring.
Reverted the complications. Also unexport NewSliceContainers
which is used only for that outside of roaring's internals.

PortMapper from cluster_internal_test.go: Used only for a test
we removed early this year. Never used for anything else.

RawRoaringData: Totally unused.

TxStore: Totally unused.

Functions removed from Tx API, and sometimes corresponding
members were removed from structs:

* Dump: debugging code, I don't think I found any actually reachable
  paths to it.
* Group: only used for debugging TxGroup stuff
* IncrementOpN: only used by fragment, fragment can increment its
  own opN.
* Options: unused?
* Pointer: debugging only
* Readonly: used only to decide how to handle Tx in a TxGrp,
  but we never add a non-readonly Tx to a TxGrp. Removed also all
  the corresponding write-aware stuff.
* RoaringBitmapReader: Used exactly once, can just be a bm.WriteTo.
* Sn (and OpenSnList): Unused
* UnionInPlace: unused and conceptually-invalid; it didn't write
  to storage and shouldn't have, and was just "create a bitmap
  then call union-in-place", which we can do directly.
* UseRowCache: just checked storage.UseRowCache.

Other things removed:

The SetRequiredForAtomicWriteTx and ClearRequiredForAtomicWriteTx
functions go away, since nothing now seems to be using them? Same
for holder_internal_test's `testHasBit` and `testMustNotHaveBit`,
which were unused.

The DBPerShard "DeleteDBPath" and "HasData" functions and related
parts were mostly unused; took out the parts that were never
actually being reached.

Changed the API of one function to simplify special cases and
remove things:
* ImportRoaringBits had a special "data" argument which gave it
  subtly different semantics for RBF and roaring (for roaring, it
  could produce a roaring bitmap *with ops log*), didn't seem to
  be adding much. Removed corresponding "readStorageFromArchive"
  which is not otherwise used.

Also took out various debugging/dumping functions that were unused
and may have bitrotted.

Dropped a test from txfactory_internal_test, and the "pjobs"
code, because those two were the only things that needed Barrier
and thus idem, which lets us drop two more dependencies. We already
have errgroup for grouping things which want to terminate as
soon as one of them errors, approximately. To do better we'd have
to have context-threading, really.

Unbroke the WriteFragment test for non-roaring tests and made it
not roaring-only.
2021-10-26 12:30:25 -05:00
.artifactory Added in various CI files 2021-10-08 12:16:03 -07:00
.circleci Rename pilosa to featurebase 2021-09-23 10:16:14 -05:00
.cloudbuild Added in various CI files 2021-10-08 12:16:03 -07:00
.github FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
.gitlab Migrated Cloud Build to GitLab 2021-10-14 18:37:06 -07:00
api/client FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
boltdb kill off a ton more fsyncs 2021-10-01 10:45:08 -05:00
client adds more detail to CSV ingest error message 2021-10-08 08:27:49 -07:00
cmd Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
ctl Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
debugstats introduce storage.Config 2021-01-20 22:05:38 -06:00
disco replace roaring.Bitmap by [][]byte 2021-05-10 18:45:23 +02:00
encoding/proto code review cleanup 2021-09-27 12:05:57 -05:00
etcd reduce etcd fsyncs during testing 2021-10-01 10:45:08 -05:00
gcnotify FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
generator Address TODOs in roaring tests 2020-05-21 13:28:54 -05:00
gopsutil FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
hash FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
http renamed webUI from Pilosa to FeatureBase and updated version 2021-09-28 16:34:52 -05:00
ingest move stableTranslator into test code 2021-09-27 12:05:57 -05:00
ingest_testdata ingest and ingest/codec testing work 2021-08-19 09:50:59 -05:00
internal kill off a ton more fsyncs 2021-10-01 10:45:08 -05:00
lattice Update aggregate label to cardinality 2021-08-02 19:01:37 -05:00
logger FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
lru so much lint 2019-04-16 12:07:18 -05:00
mock FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
net Address PR comments 2021-03-24 18:09:01 +01:00
pb introduce protobuf types for ingest ops 2021-09-27 12:05:57 -05:00
pg address ben's comments 2021-10-12 11:21:14 -05:00
pql FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
prometheus FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
proto FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
rbf Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
roaring Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
scripts Fix benchmark file naming 2021-05-21 14:51:08 -06:00
server Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
shardwidth make "subdivide list by shardwidth" available for reuse 2021-08-18 13:45:20 -05:00
short_txkey pilosa: only open views with data 2020-12-19 00:03:31 +00:00
sql basic looker connection tests pass 2021-09-25 13:51:02 -05:00
sql2 CORE-809: Aggregate COUNT() with INNER JOIN 2021-10-07 14:20:07 -06:00
statik FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
stats FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
statsd FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
storage Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
synthload FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
syswrap better explanation for 'cannot allocate memory' error 2021-09-23 08:32:12 -05:00
test kill off a ton more fsyncs 2021-10-01 10:45:08 -05:00
testdata Test cases for Store(Distinct) 2020-09-14 14:15:29 -05:00
testhook FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
toml Add license headers to files missing them and CI check to verify they are present. Fixes #1633 2019-04-12 11:30:41 -05:00
topology FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
tracing FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
txkey pilosa: only open views with data 2020-12-19 00:03:31 +00:00
vprint Move vprint to separate pakage 2021-03-29 14:29:19 +02:00
.dockerignore Add new Docker build process and continuous delivery 2021-02-26 14:34:14 -06:00
.gitignore add featurebase to .gitignore 2021-10-26 09:15:44 -05:00
.golangci.yml fix golangci-lint complaints 2019-11-10 17:58:46 -06:00
api.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
api_test.go add clusters to MutexCheck test, fix silly bug revealed by doing so 2021-09-13 16:25:15 -05:00
apimethod_string.go remove attributes 2021-05-14 10:28:08 -04:00
audit.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
audit_internal_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
audit_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
broadcast.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
bsi.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
bsi_test.go introduce "fields" directory between index and field 2021-03-05 16:56:44 -06:00
cache.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
cache_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
catcher.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
CHANGELOG.md Fix typo 2019-09-17 16:01:21 -05:00
client.go ingest cluster support 2021-09-27 12:05:57 -05:00
cluster.go code review cleanup 2021-09-27 12:05:57 -05:00
cluster_internal_test.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
cmd.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
const_amd64.go remove lmdb as a Tx backend 2020-12-04 23:17:59 +00:00
const_other.go remove lmdb as a Tx backend 2020-12-04 23:17:59 +00:00
CONTRIBUTING.md simplify contributing instructions by removing weird upstream thing 2019-05-01 17:37:52 -05:00
dbshard.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
dbshard_internal_test.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
dbshard_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
delete_test.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
diagnostics.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
diagnostics_internal_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
doc.go Apply Apache License 2.0 2017-04-28 14:22:03 -05:00
docker-compose-3.yml More rename to work with backup and cluster tests 2021-09-22 15:41:17 -05:00
Dockerfile More rename to work with backup and cluster tests 2021-09-22 15:41:17 -05:00
Dockerfile-clustertests Test paused node picks up once cluster state is back to normal 2021-09-30 20:48:01 +03:00
Dockerfile.pilosa Remove prints 2021-09-22 16:02:02 -05:00
Dockerfile.runner Rename docker builds to featurebase 2021-09-22 15:41:00 -05:00
event.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
executor.go rework executor's per-shard union to use UnionInPlace 2021-09-29 11:31:13 -05:00
executor_internal_test.go kill off a ton more fsyncs 2021-10-01 10:45:08 -05:00
executor_test.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
field.go changes error message text for additional clarity 2021-10-08 09:47:16 -07:00
field_internal_test.go kill off a ton more fsyncs 2021-10-01 10:45:08 -05:00
field_test.go kill off a ton more fsyncs 2021-10-01 10:45:08 -05:00
filesystem.go Add missing file 2020-08-31 09:02:46 -05:00
fragment.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
fragment_internal_test.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
gc.go fix some lint warnings raised in VS-Code 2019-04-17 18:10:05 -05:00
gendebug_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
generation.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
generation_debug.go testhook: leak auditing infrastructure 2020-08-24 11:26:39 -05:00
generation_nodebug.go Sources and Generations: tracking mmapped files 2019-11-12 12:14:29 -06:00
generation_test.go Tx integration milestone 2020-07-27 19:29:46 -04:00
gid.go database per shard, HolderConfig, rbf bit-wise import speedups. 2020-09-04 13:00:33 -05:00
go.mod Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
go.sum Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
hack.go fix build issue 2021-07-19 13:47:47 -07:00
handler.go make import correctly reflect that it needs a single shard always 2021-08-18 13:45:36 -05:00
holder.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
holder_internal_test.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
holder_test.go kill off a ton more fsyncs 2021-10-01 10:45:08 -05:00
idalloc.go kill off a ton more fsyncs 2021-10-01 10:45:08 -05:00
idalloc_test.go reduce etcd fsyncs during testing 2021-10-01 10:45:08 -05:00
index.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
index_internal_test.go kill off a ton more fsyncs 2021-10-01 10:45:08 -05:00
index_test.go Add comment as to why we are using os.Exit instead of panic 2021-10-14 18:07:02 +03:00
ingest_test.go ingest cluster support 2021-09-27 12:05:57 -05:00
iterator.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
iterator_internal_test.go rename internal_tests. fix license header 2018-07-04 10:59:08 -05:00
LICENSE Apply Apache License 2.0 2017-04-28 14:22:03 -05:00
license.exceptions update license header check exceptions 2021-05-14 10:45:46 -04:00
like.go execute like queries on the primary's key translation database 2021-05-27 14:45:10 -04:00
like_test.go execute like queries on the primary's key translation database 2021-05-27 14:45:10 -04:00
main_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
Makefile Test paused node picks up once cluster state is back to normal 2021-09-30 20:48:01 +03:00
metrics.go remove attributes 2021-05-14 10:28:08 -04:00
mmap_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
NOTES rename slice to shard 2018-06-28 14:07:07 -05:00
NOTICE Remove a few more enterprise references 2020-06-26 15:57:21 -05:00
pilosa.go Update pilosa.go 2021-10-08 13:46:17 -07:00
pilosa_internal_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
pilosa_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
planner.go CORE-809: Aggregate COUNT() with INNER JOIN 2021-10-07 14:20:07 -06:00
planner_test.go CORE-809: Aggregate COUNT() with INNER JOIN 2021-10-07 14:20:07 -06:00
pprof.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
rbf.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
README.md FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
row.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
row_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
rrtx.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
rrtx_internal_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
serializer.go add a gob-encoding Serializer implementation for tests 2021-02-23 10:09:55 -06:00
server.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
server_internal_test.go kill off a ton more fsyncs 2021-10-01 10:45:08 -05:00
snapshotqueue.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
stattx.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
testBackupRestore.sh partial backup/restore 2021-10-08 10:50:52 -05:00
time.go Prototype ingest API 2021-08-19 09:50:59 -05:00
time_internal_test.go avoid allocations in viewsByTime 2021-08-18 13:45:36 -05:00
tournament.sh remove instances of os.Getenv("PILOSA_TXSRC") 2021-01-21 21:18:46 -06:00
tracker.go ingest and ingest/codec testing work 2021-08-19 09:50:59 -05:00
tracker_test.go Update tests 2021-01-20 10:48:35 -06:00
transaction.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
transaction_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
translate.go kill off a ton more fsyncs 2021-10-01 10:45:08 -05:00
translator_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
tx.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
tx_internal_test.go FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
tx_test.go make import correctly reflect that it needs a single shard always 2021-08-18 13:45:36 -05:00
txfactory.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
txfactory_internal_test.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
util.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
util_test.go meaningless commit to kick off sonarcloud with new rules 2021-10-25 15:24:55 -05:00
utils_internal_test.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
version.go FeatureBase renaming: version info in server log 2021-07-12 12:25:43 -07:00
view.go Giant Commit: drop a bunch of stuff we don't use. 2021-10-26 12:30:25 -05:00
view_internal_test.go kill off a ton more fsyncs 2021-10-01 10:45:08 -05:00

README.md

FeatureBase, a distributed bitmap index

See our internal documentation, which includes all external documentation, plus many internal-only pages, listed under the "Internal" heading in the main navigation bar.

Follow along with the Sample Project to get a better understanding of FeatureBase's capabilities.