scriptoriumadmin/featurebase
1
0
Fork 0
mirror of https://github.com/featurebasedb/featurebase.git synced 2026-08-28 10:54:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
No description
9332 commits 45 branches 174 tags 85 MiB
Find a file
Samir Patel 3681feeeb2
[FB-1024 FB-1590] Increase timerange (un-revert) (#2174) 
previously we allowed users to specify a granularity for timestamp
e.g. seconds, milli, micro, nano
however we converted everything to nano before we stored it.
This reduced the allowed range for all time units to what
was allowed by timestamp. For example, with second granularity
you can represent billions of years within the capacity of
int64 but with nano its somewhere b/w 100-200 years.

So now, for timeunits of seconds, milli, and micro the range
is year 0001 - 9999. These limits come from what Go
supports.

So this uses unit specific function to translate
timestamps to values and vice versa to increase
the time range.

In the process of increasing the range for timestamp and subsequent
testing, I found and addressed a few bugs:
- min/max queries were not using timestamp specific comparators so
  added that.
- Values from Import/ingest come to FB as relative values to epoch
    whereas other BSI fields come as actual values and then
    becomes relative to their respective bases within FB. so some
    specific handling of that was added.
- However! Set queries use timestamp strings which are, of course,
    the actual value they designate. So they have to become
    relative.
- When bitdepth is 0, Min/maxUnsigned functions did not run
resulting in a count of 0 when there
was an actual value that was 0.

Also, this removes (now) dead code and updates/adds tests.
2022-08-04 15:20:45 -07:00
.github remove references to CONTRIBUTING and CHANGELOG 2021-11-19 11:22:15 -06:00
.gitlab [FB-1024 FB-1590] Increase timerange (un-revert) (#2174) 2022-08-04 15:20:45 -07:00
api/client Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
authn [FB-1379] Create a featurebase subcommand to obtain an auth token (#2079) 2022-05-26 11:35:51 -05:00
authz Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
boltdb change perms: 0666 -> 0600; 0777 -> 0750 2022-07-21 13:34:46 -05:00
client [FB-1024 FB-1590] Increase timerange (un-revert) (#2174) 2022-08-04 15:20:45 -07:00
cmd add row-col-mode to featurebase import 2022-08-02 12:12:53 -05:00
ctl add row-col-mode to featurebase import 2022-08-02 12:12:53 -05:00
debugstats add copyright notice back in 2021-12-10 11:01:04 -06:00
disco simplify, streamline, and possibly debug embedded etcd 2022-07-21 11:42:35 -05:00
encoding/proto simplify, streamline, and possibly debug embedded etcd 2022-07-21 11:42:35 -05:00
etcd removed invalid state (#2173) 2022-08-01 12:47:17 -05:00
gcnotify Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
generator add copyright notice back in 2021-12-10 11:01:04 -06:00
gopsutil Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
hash Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
idk [FB-1024 FB-1590] Increase timerange (un-revert) (#2174) 2022-08-04 15:20:45 -07:00
ingest [FB-1024 FB-1590] Increase timerange (un-revert) (#2174) 2022-08-04 15:20:45 -07:00
ingest_testdata allow ingest API delete to work at all, add test for it 2022-04-01 09:48:51 -05:00
install Fixed broken fields when packaging rpm and deb files. (#2153) 2022-07-27 12:54:46 -05:00
internal simplify, streamline, and possibly debug embedded etcd 2022-07-21 11:42:35 -05:00
lattice Remove most of the resize-related logic 2022-06-21 17:03:09 -05:00
logger change perms: 0666 -> 0600; 0777 -> 0750 2022-07-21 13:34:46 -05:00
lru fix some staticcheck issues 2021-12-03 09:31:45 -06:00
mock Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
monitor Add proper version info to sentry logs 2022-06-21 11:03:22 -05:00
net add copyright notice back in 2021-12-10 11:01:04 -06:00
pb indicate sign with a bool 2022-06-17 14:58:30 -05:00
pg remove trailing spaces en masse (#2024) 2022-04-18 09:31:35 -05:00
pql recalculate shards for each call in query if it needs them 2022-07-27 10:25:52 -05:00
prometheus Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
proto Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
qa FB-1597: unifying idk and featurebase (#2160) 2022-07-28 17:23:16 -05:00
rbf catch errors on closing written-to files 2022-07-22 12:20:16 -05:00
roaring change perms: 0666 -> 0600; 0777 -> 0750 2022-07-21 13:34:46 -05:00
scripts remove trailing spaces en masse (#2024) 2022-04-18 09:31:35 -05:00
server [FB-1024 FB-1590] Increase timerange (un-revert) (#2174) 2022-08-04 15:20:45 -07:00
shardwidth Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
short_txkey remove a bunch of commented print statements and unecessary prints 2022-01-11 10:42:44 -06:00
sql fix parens around where clause bug (#2121) 2022-06-16 22:21:24 -05:00
sql2 Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
statik Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
stats only set bits after the holder is completely setup 2022-02-25 15:02:24 -06:00
statsd Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
storage remove a bunch of roaring backend stuff 2022-02-02 20:56:18 -06:00
syswrap add copyright notice back in 2021-12-10 11:01:04 -06:00
task implement a task pool 2022-02-14 09:56:19 -06:00
test simplify, streamline, and possibly debug embedded etcd 2022-07-21 11:42:35 -05:00
testdata FB-1378: Use IP whitelisting for ingest authentication and authorization (#2070) 2022-05-20 14:57:05 -05:00
testhook Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
toml add copyright notice back in 2021-12-10 11:01:04 -06:00
tracing Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
txkey remove a bunch of commented print statements and unecessary prints 2022-01-11 10:42:44 -06:00
vprint add copyright notice back in 2021-12-10 11:01:04 -06:00
.dockerignore FB-1597: unifying idk and featurebase (#2160) 2022-07-28 17:23:16 -05:00
.gitignore FB-1597: unifying idk and featurebase (#2160) 2022-07-28 17:23:16 -05:00
.golangci.yml fix govet and gofmt errors in existing code 2021-10-29 13:14:27 -05:00
api.go [FB-1024 FB-1590] Increase timerange (un-revert) (#2174) 2022-08-04 15:20:45 -07:00
api_test.go catch errors on closing written-to files 2022-07-22 12:20:16 -05:00
apimethod_string.go Remove most of the resize-related logic 2022-06-21 17:03:09 -05:00
audit.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
audit_internal_test.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
audit_test.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
broadcast.go simplify, streamline, and possibly debug embedded etcd 2022-07-21 11:42:35 -05:00
bsi.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
bsi_test.go add copyright notice back in 2021-12-10 11:01:04 -06:00
cache.go comment cleanup 2022-05-27 11:25:17 -05:00
cache_test.go Periodically invalidate rank cache during bulk add 2022-02-15 08:25:45 -07:00
catcher.go create BitmapRewriter/ApplyRewriter, parallel to BitmapFilter 2022-04-12 12:24:22 -05:00
cluster.go simplify, streamline, and possibly debug embedded etcd 2022-07-21 11:42:35 -05:00
cluster_internal_test.go simplify, streamline, and possibly debug embedded etcd 2022-07-21 11:42:35 -05:00
cmd.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
const_amd64.go add copyright notice back in 2021-12-10 11:01:04 -06:00
const_other.go add copyright notice back in 2021-12-10 11:01:04 -06:00
dbshard.go drop the bool from a map that's only using it to check presence 2022-06-09 10:26:19 -05:00
dbshard_internal_test.go drop the bool from a map that's only using it to check presence 2022-06-09 10:26:19 -05:00
dbshard_test.go remove http subpackage and bring implementations into core 2022-02-03 21:04:04 -06:00
delete_test.go Add regression test for FB-1281 2022-03-24 15:20:57 -06:00
diagnostics.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
diagnostics_internal_test.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
doc.go add copyright notice back in 2021-12-10 11:01:04 -06:00
Dockerfile remove references to LICENSE and checks for it in source files 2021-11-19 10:38:06 -06:00
Dockerfile-clustertests remove trailing spaces en masse (#2024) 2022-04-18 09:31:35 -05:00
Dockerfile-clustertests-client remove trailing spaces en masse (#2024) 2022-04-18 09:31:35 -05:00
event.go simplify, streamline, and possibly debug embedded etcd 2022-07-21 11:42:35 -05:00
executor.go [FB-1024 FB-1590] Increase timerange (un-revert) (#2174) 2022-08-04 15:20:45 -07:00
executor_internal_test.go track closing status for index/field/view, shut down cache flush early 2022-03-18 14:41:44 -05:00
executor_test.go [FB-1024 FB-1590] Increase timerange (un-revert) (#2174) 2022-08-04 15:20:45 -07:00
field.go [FB-1024 FB-1590] Increase timerange (un-revert) (#2174) 2022-08-04 15:20:45 -07:00
field_internal_test.go Revert "FB-1491: Revert pql.Decimal commit for the 4.8.5 release (#2117)" 2022-06-17 14:58:30 -05:00
field_test.go [FB-1024 FB-1590] Increase timerange (un-revert) (#2174) 2022-08-04 15:20:45 -07:00
filesystem.go add copyright notice back in 2021-12-10 11:01:04 -06:00
fragment.go [FB-1024 FB-1590] Increase timerange (un-revert) (#2174) 2022-08-04 15:20:45 -07:00
fragment_internal_test.go change perms: 0666 -> 0600; 0777 -> 0750 2022-07-21 13:34:46 -05:00
gc.go add copyright notice back in 2021-12-10 11:01:04 -06:00
gid.go database per shard, HolderConfig, rbf bit-wise import speedups. 2020-09-04 13:00:33 -05:00
go.mod update to 1.9.1;make rm safe (#2175) 2022-08-02 15:09:20 -05:00
go.sum update to 1.9.1;make rm safe (#2175) 2022-08-02 15:09:20 -05:00
hack.go Change Ttl to TTL (#2038) 2022-04-27 11:20:39 -05:00
handler.go add doc comments 2022-05-27 11:25:17 -05:00
holder.go remove unneccessary file opens 2022-07-22 12:20:16 -05:00
holder_internal_test.go address reviewer's comments 2022-03-10 09:36:18 -06:00
holder_test.go change perms: 0666 -> 0600; 0777 -> 0750 2022-07-21 13:34:46 -05:00
http_handler.go verify available space before backup 2022-07-29 11:30:36 -05:00
http_handler_internal_test.go [FB-1444] Add sentry.io for error and perf monitoring (#2099) 2022-06-06 18:29:32 -05:00
http_handler_test.go change perms: 0666 -> 0600; 0777 -> 0750 2022-07-21 13:34:46 -05:00
http_translator.go remove http subpackage and bring implementations into core 2022-02-03 21:04:04 -06:00
http_translator_test.go remove http subpackage and bring implementations into core 2022-02-03 21:04:04 -06:00
idalloc.go change perms: 0666 -> 0600; 0777 -> 0750 2022-07-21 13:34:46 -05:00
idalloc_test.go change perms: 0666 -> 0600; 0777 -> 0750 2022-07-21 13:34:46 -05:00
index.go remove unneccessary file opens 2022-07-22 12:20:16 -05:00
index_internal_test.go remove unused stuff and fix a bunch of random staticcheck issues 2022-02-07 15:10:10 -06:00
index_test.go Revert "FB-1491: Revert pql.Decimal commit for the 4.8.5 release (#2117)" 2022-06-17 14:58:30 -05:00
ingest_test.go allow ingest API delete to work at all, add test for it 2022-04-01 09:48:51 -05:00
internal_client.go verify available space before backup 2022-07-29 11:30:36 -05:00
internal_client_test.go verify available space before backup 2022-07-29 11:30:36 -05:00
iterator.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
iterator_internal_test.go add copyright notice back in 2021-12-10 11:01:04 -06:00
license.exceptions update license header check exceptions 2021-05-14 10:45:46 -04:00
like.go add copyright notice back in 2021-12-10 11:01:04 -06:00
like_test.go add copyright notice back in 2021-12-10 11:01:04 -06:00
main_test.go listen on localhost:0 instead of :0 2022-03-18 13:38:30 -05:00
Makefile Fixed broken fields when packaging rpm and deb files. (#2153) 2022-07-27 12:54:46 -05:00
metrics.go add copyright notice back in 2021-12-10 11:01:04 -06:00
nfpm.yaml Fixed broken fields when packaging rpm and deb files. (#2153) 2022-07-27 12:54:46 -05:00
NOTICE add config files for release 2021-11-19 01:26:18 +03:00
pilosa.go Remove most of the resize-related logic 2022-06-21 17:03:09 -05:00
pilosa_internal_test.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
pilosa_test.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
planner.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
planner_test.go track closing status for index/field/view, shut down cache flush early 2022-03-18 14:41:44 -05:00
pprof.go remove a bunch of roaring backend stuff 2022-02-02 20:56:18 -06:00
rbf.go create BitmapRewriter/ApplyRewriter, parallel to BitmapFilter 2022-04-12 12:24:22 -05:00
README.md add coverage, code quality badges 2022-03-30 10:07:23 -05:00
row.go remove string keys on delete to allow for reuse 2022-02-25 16:23:54 -06:00
row_test.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
serializer.go add copyright notice back in 2021-12-10 11:01:04 -06:00
server.go simplify, streamline, and possibly debug embedded etcd 2022-07-21 11:42:35 -05:00
server_internal_test.go improve Server WaitGroup concurrent usage 2022-02-25 15:02:21 -06:00
server_test.go FB-1505 - Fixes for PR: remove underscores from test names, rename TTLRemoval to ViewsRemoval 2022-06-30 16:58:57 -05:00
stattx.go create BitmapRewriter/ApplyRewriter, parallel to BitmapFilter 2022-04-12 12:24:22 -05:00
time.go FB-1456 - TTL - fixed views not returning correct results (#2062) 2022-05-16 15:26:13 -05:00
time_internal_test.go FB-1456 - TTL - fixed views not returning correct results (#2062) 2022-05-16 15:26:13 -05:00
tracker.go add copyright notice back in 2021-12-10 11:01:04 -06:00
tracker_test.go add copyright notice back in 2021-12-10 11:01:04 -06:00
transaction.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
transaction_test.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
translate.go simplify, streamline, and possibly debug embedded etcd 2022-07-21 11:42:35 -05:00
translator_test.go simplify, streamline, and possibly debug embedded etcd 2022-07-21 11:42:35 -05:00
tx.go create BitmapRewriter/ApplyRewriter, parallel to BitmapFilter 2022-04-12 12:24:22 -05:00
tx_internal_test.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
tx_test.go remove string keys on delete to allow for reuse 2022-02-25 16:23:54 -06:00
txfactory.go . 2022-03-11 06:20:07 -06:00
txfactory_internal_test.go remove a bunch of roaring backend stuff 2022-02-02 20:56:18 -06:00
util.go [FB-1024 FB-1590] Increase timerange (un-revert) (#2174) 2022-08-04 15:20:45 -07:00
util_test.go [FB-1024 FB-1590] Increase timerange (un-revert) (#2174) 2022-08-04 15:20:45 -07:00
utils_internal_test.go simplify, streamline, and possibly debug embedded etcd 2022-07-21 11:42:35 -05:00
version.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
view.go change perms: 0666 -> 0600; 0777 -> 0750 2022-07-21 13:34:46 -05:00
view_internal_test.go Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00

README.md

FeatureBase, a distributed bitmap index

Coverage SecurityRating Quality Gate Status

See our internal documentation, which includes all external documentation, plus many internal-only pages, listed under the "Internal" heading in the main navigation bar.

Follow along with the Sample Project to get a better understanding of FeatureBase's capabilities.