scriptoriumadmin/featurebase
1
0
Fork 0
mirror of https://github.com/featurebasedb/featurebase.git synced 2026-10-10 12:57:54 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
8997 commits 45 branches 174 tags 85 MiB
Commit graph

8997 commits

This branch
This branch
All branches
Author SHA1 Message Date
Jason E. Aten
e1bb6e303a rbf: use an inlined immutable.Map<uint32, int64> for the PageMap 
- goes 7% faster on kitchen sink import test

- reduces total allocations by 5% on same test.
 2020-11-20 00:49:41 +00:00
jaten-molecula
47202b0ed2
Merge pull request #1095 from molecula/fast-write 
Implement optimized fast leaf write.
 2020-11-19 16:04:31 -06:00
Ben Johnson
a9831ad5a1 Replace literals with leafCellHeaderSize 2020-11-19 14:18:53 -07:00
Ben Johnson
d125c68275 Fix estimated page size calculation to include index padding. 2020-11-19 14:12:43 -07:00
Ben Johnson
33334511c4
Merge branch 'master' into fast-write 2020-11-19 13:44:09 -07:00
jaten-molecula
5b8cda3c88
Merge pull request #1120 from molecula/fix-root-record-overflow 
Fix rbf write root record iterator reset
 2020-11-19 13:11:09 -06:00
Ben Johnson
a31487b873 Fix rbf write root record iterator reset 2020-11-19 11:15:24 -07:00
jaten-molecula
61bdc2257b
Merge pull request #1111 from jaten-molecula/sync_pool 
rbf: reuse cursors with sync.Pool/arena
 2020-11-19 11:17:20 -06:00
Jason E. Aten
9c7bc603af rbf: reuse cursors with sync.Pool or arena 
- the sync.Pool default uses little memory under CI.

- arena approach provides ability to control the maximum memory
  used by rbf Cursors.

- cursor caching is adjustable with --rbf-cursor-cache
  currently 0 by default (meaning use sync.Pool), and
  larger than 0 meaning use an arena of this size.
  With the arena, 20 or less is needed to pass CI.

- rbf test suite runs ~ 4x faster

- kitchen sink ingest test runs 16% faster.

- report TotalAlloc in CALLSTATs

fixes #1105
 2020-11-19 17:02:49 +00:00
jaten-molecula
274a95c4da
Merge pull request #1115 from molecula/remove-rbtree 
Use immutable.SortedMap for root records
 2020-11-19 11:00:57 -06:00
Ben Johnson
26d0e9d958
Merge branch 'master' into remove-rbtree 2020-11-19 07:27:09 -07:00
tgruben
b849fe4b98
Merge pull request #1116 from jaten-molecula/pilosa_txsrc 
use pilosa server --txsrc instead of --tx to prevent viper env var shadowing
 2020-11-18 14:37:10 -06:00
Jason E. Aten
524c78623f repair pilosa/ctl/server_test too 2020-11-18 18:53:22 +00:00
Ben Johnson
0175c66756 Use immutable.SortedMap for root records 
This commit fixes a bug where the root record cache was being
updated in-place causing a race condition with other transactions
using it. The cache implementation has been changed from `rbtree`
to an `immutable.SortedMap`.
 2020-11-18 11:22:52 -07:00
Jason E. Aten
06eb3afaaa use pilosa server --txsrc to prevent viper env var shadowing from stopping PILOSA_TXSRC getting through 2020-11-18 18:21:31 +00:00
jaten-molecula
a9c1ff3047
Merge pull request #1107 from jaten-molecula/fix_tx 
restore the env variable PILOSA_TXSRC's affect
 2020-11-16 17:26:22 -06:00
Jason E. Aten
744827d5cc restore the env variable PILOSA_TXSRC's affect 2020-11-16 23:18:50 +00:00
Ben Johnson
89ae5ad4b7
Merge branch 'master' into fast-write 2020-11-16 09:10:01 -07:00
Ben Johnson
519dc5069d Implement optimized fast leaf write. 
This commit adds an optimized implementation for `putLeafCell()` if
the insert/update will not cause the page to overflow.
 2020-11-16 08:13:35 -07:00
Nia
6dfbbf91d6
Merge pull request #1098 from niaow/topn-v3 
Add TopK with perpendicular BSI bitmaps
 2020-11-16 08:39:50 -05:00
Nia Weiss
02498ce57a
add TopK with perpendicular BSI bitmaps 2020-11-13 18:31:55 -05:00
tgruben
43443d10ff
Merge pull request #1097 from jaten-molecula/bitmap_foreach 
rbf: BitN needs int32 to hold its maximum value.
 2020-11-13 17:12:19 -06:00
Jason E. Aten
4120024b10 rbf: BitN needs int32 to hold its maximum value. 
- decrease ArrayMaxSize and RLEMaxSize by 1
  to make space
- add TestForEachRange, fix bugs found in ForEach,
  where the call f() logic was backwards.
 2020-11-13 22:59:18 +00:00
tgruben
b2e9e3904a
Merge pull request #1094 from jaten-molecula/elem_rb 
rbf: keep ElemN and BitN up to date.
 2020-11-13 15:44:15 -06:00
Jason E. Aten
d46b603cd6 rbf: keep ElemN and BitN up to date. 
- fix a bug in computing leafCell.BitN in a run after a bit Remove
- shrink bitmaps on remove
- util_test.go has Cursor.DebugSlowCheckAllPages to verify;
  used by cursor_test.go
 2020-11-13 21:18:09 +00:00
Cody Soyland
dc0ac0ccd0
Merge pull request #1092 from codysoyland/ci-go-version-cve-2020-28362 
Ci: Go version update for CVE-2020-28362
 2020-11-13 11:48:16 -06:00
Cody Soyland
e53b74261b Update default Go version 2020-11-13 11:23:39 -06:00
Cody Soyland
a2f49b6a84 Fix yaml syntax, add requires section 2020-11-13 11:23:39 -06:00
Cody Soyland
dca606dea8 Add pilosa (binary build) to .gitignore 2020-11-13 11:23:39 -06:00
Cody Soyland
671250d273 Update Go version for CVE-2020-28362, use param matrix 2020-11-13 11:23:39 -06:00
tgruben
d15dc4204b
Merge pull request #1086 from jaten-molecula/rbtree_root_records 
rbf: use a red-black tree to manage the root records list
 2020-11-13 11:14:51 -06:00
Jason E. Aten
eae82b72c2 rbf: use a red-black tree to manage the root records list 
- 40% faster on ingest_test when putting 10K roots/containers.
 - 15% fewer bytes allocated total
 - clarifying renames leafCell.N -> ElemN, allocate -> allocatePgno,
   deallocate -> freePgno
 2020-11-13 16:49:26 +00:00
jaten-molecula
cb73c71e02
Merge pull request #1072 from molecula/uni-wal 
Refactor RBF to use a single WAL file
 2020-11-11 15:44:45 -06:00
jaten-molecula
563c70dc43
Merge branch 'master' into uni-wal 2020-11-11 15:27:15 -06:00
tgruben
b3d2704066
Merge pull request #1085 from jaten-molecula/uniq_dbnames 
unique db names so parallel test runs don't collide
 2020-11-11 14:26:04 -06:00
Jason E. Aten
fd9210b31c unique db names so parallel test runs don't collide 2020-11-11 20:04:02 +00:00
Ben Johnson
81a64c5902 Add RBF halting; remove time based checkpoint 2020-11-11 11:17:07 -07:00
Ben Johnson
9eba299d35 Restrict max RBF transaction size 2020-11-10 08:14:14 -07:00
Ben Johnson
78eb9e0711 Fix linter 2020-11-10 07:07:06 -07:00
Ben Johnson
52340212f6 Add RBF dirty page cache 2020-11-09 16:11:20 -07:00
Ben Johnson
8de1959938 Remove RBF exclusive/direct write. 2020-11-09 08:23:09 -07:00
Ben Johnson
3554048877 Refactor RBF to use a single WAL file 2020-11-09 08:03:22 -07:00
tgruben
e99f7f0fc3
Merge pull request #1074 from jaten-molecula/fsync_applies_to_all 
apply --fsync flag to all tx backend
 2020-11-06 12:34:28 -06:00
Jason E. Aten
a38d4fa46c apply --fsync flag to all tx backend 
- rename from --rbf-fsync to --fsync, as it
   now effects bolt, lmdb too.
 2020-11-06 18:11:02 +00:00
jaten-molecula
822c7482a5
Merge pull request #1071 from jaten-molecula/parallel_migration_rb 
run migration in parallel
 2020-11-05 14:09:31 -06:00
Jason E. Aten
7e22994a6d run migration in parallel 
- migration can be slow. parallelize it.
 2020-11-05 18:31:27 +00:00
tgruben
bc26de63b0
Merge pull request #1070 from jaten-molecula/slurp_profile_rbf_comments 
rbf default. Add TODO comments, slurp -profile returns a cpu profile
 2020-11-03 19:52:54 -06:00
Jason E. Aten
458095a707 rbf default. Add TODO comments, slurp -profile returns a cpu profile 
- default Tx is once again RBF, changed from bolt.
- document the RBF code review comments that were not addressed
  before #1052 was merged, so they don't get lost.
- they should be easily addressed by replaying the entire WAL file
  rather than from the DB meta page 0 notion of the last WalID
- cleanup rbf/cfg/cfg.go stale comments, ensure default0 respected.
  1 msec checkpoint time, 1MB wal segment defaults.
- return a specific error, ErrNoMetaFound, from findNextWALMetaPage()
  rather than io.EOF, since there actually wasn't any file IO involved.
- add http handlers for /cpu-profile/start and /cpu-profile/stop
  in http/handler.go enable CPU profiling at specific time points
  during an ingest or other operation.
 2020-11-04 01:36:17 +00:00
tgruben
a55cdad139
Merge pull request #1069 from jaten-molecula/faster_new_tx 
take a mutex off the fast path of TxFactory.NewTx
 2020-11-03 19:15:26 -06:00
Jason E. Aten
bf36f54b8d take a mutex off the fast path of TxFactory.NewTx 2020-11-04 01:04:04 +00:00