scriptoriumadmin/featurebase
1
0
Fork 0
mirror of https://github.com/featurebasedb/featurebase.git synced 2026-08-28 10:54:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
6853 commits 45 branches 174 tags 85 MiB
Commit graph

390 commits

Author SHA1 Message Date
Travis
4dd6bddf7f
remove pilosa-chk 2021-02-05 16:13:43 -06:00
Travis
c8c59b649d
remove pilosa-fsck 2021-02-04 21:34:15 -06:00
Travis
a4b37273ea
remove the rest of the gossip code (except config) 2021-02-04 13:03:02 -06:00
Antonio Navarro Perez
87ba73fa16 Stop writes on DEGRADED state 
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
 2021-02-04 17:30:14 +01:00
Travis
457194f6a8
update config to support etcd arguments 2021-01-29 19:43:48 -06:00
Travis
1c0b926eae
Merge master into disco 2021-01-28 18:03:38 -06:00
Travis
4380a05bbd
address some coord/node0 test issues 2021-01-26 22:46:37 -06:00
Travis
ace4dea46f
address some test failures due to random ordered etcd ID 2021-01-25 00:52:49 -06:00
Travis
1d55e671a2
go mod tidy and linter 
fix race

cleanup
 2021-01-15 17:48:44 -06:00
Travis
9855f4d0a0
Merge branch 'travis-test-ci' into disco-try 2021-01-15 14:31:19 -06:00
Travis
10380a1da1
Implement snap := ClusterSnapshot() 
Below is the list of instance of `ClusterSnapshot()` in the latest
`with-etcd` code. Some of these may not yet exist in the `disco` branch,
but this commit is implementing any that currently apply.

==========================
Done:
==========================
index.go
930:	snap := NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)
1072:	snap := NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)

cmd/pilosa-fsck/fsck.go
786:	snap := pilosa.NewClusterSnapshot(cfg.topo, cfg.topo.Hasher, cfg.topo.ReplicaN)

boltdb/translate.go
558:	snap := pilosa.NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)
1264:	snap := pilosa.NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)

fragment.go
3448:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
3568:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
3620:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)

==========================
Remaining:
==========================

cluster.go
371:	snap := NewClusterSnapshot(NewLocalNoder(nodes), c.Hasher, c.ReplicaN)
474:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
639:	fSnap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
640:	toSnap := NewClusterSnapshot(to.noder, c.Hasher, to.ReplicaN)
703:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1475:		snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1502:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1941:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1986:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
2049:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
2126:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)

api.go
475:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
604:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
690:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
1684:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
1946:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)

executor.go
3781:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4157:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4200:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4243:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4517:	snap := NewClusterSnapshot(NewLocalNoder(e.Cluster.Nodes()), e.Cluster.Hasher, e.Cluster.ReplicaN)

holder.go
1465:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1668:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1889:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1963:	snap := NewClusterSnapshot(c.Cluster.noder, c.Cluster.Hasher, c.Cluster.ReplicaN)
 2021-01-15 11:47:37 -06:00
Travis
7875fb8e5a
remove pilosa.DefaultPartitionN 2021-01-15 11:47:37 -06:00
Travis
ff2d235702
change all references to use subpackages: topology, net 2021-01-15 11:47:36 -06:00
Travis
27614c42f7
Finish implementing port wrapper 2021-01-13 22:56:54 -06:00
Travis
bc13834343
disco/etcd work: fix lots of races, start all cluster nodes at once. 
port mapper gives out ports from 63000-65000 for the tests

fix another race

http test uses port.MustGetPort

rbf: remove :0 port request

ocd happy

test fix for grpc listener address already in use

test/disco allocates BindGRPC port from the port mapper

dump stack on each GetPort

verify each port is usable right away

server/config.go has Config.Validate() now

panic if gossip port is 0. validate server.Config

fix another gossip port 0

builds

quiet, don't dump stack on each port alloc

builds

happy linter

even gossip fallback should not be zero but rather use the port mapper
 2021-01-12 21:06:12 -06:00
Ben Johnson
b7973e2612 Add read benchmarks 2021-01-12 13:48:09 -07:00
Travis
134abda51b
Implement snap := ClusterSnapshot() 
Below is the list of instance of `ClusterSnapshot()` in the latest
`with-etcd` code. Some of these may not yet exist in the `disco` branch,
but this commit is implementing any that currently apply.

==========================
Done:
==========================
index.go
930:	snap := NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)
1072:	snap := NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)

cmd/pilosa-fsck/fsck.go
786:	snap := pilosa.NewClusterSnapshot(cfg.topo, cfg.topo.Hasher, cfg.topo.ReplicaN)

boltdb/translate.go
558:	snap := pilosa.NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)
1264:	snap := pilosa.NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)

fragment.go
3448:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
3568:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
3620:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)

==========================
Remaining:
==========================

cluster.go
371:	snap := NewClusterSnapshot(NewLocalNoder(nodes), c.Hasher, c.ReplicaN)
474:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
639:	fSnap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
640:	toSnap := NewClusterSnapshot(to.noder, c.Hasher, to.ReplicaN)
703:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1475:		snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1502:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1941:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1986:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
2049:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
2126:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)

api.go
475:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
604:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
690:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
1684:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
1946:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)

executor.go
3781:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4157:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4200:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4243:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4517:	snap := NewClusterSnapshot(NewLocalNoder(e.Cluster.Nodes()), e.Cluster.Hasher, e.Cluster.ReplicaN)

holder.go
1465:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1668:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1889:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1963:	snap := NewClusterSnapshot(c.Cluster.noder, c.Cluster.Hasher, c.Cluster.ReplicaN)
 2021-01-06 22:53:20 -06:00
Travis
3f26d667b4
remove pilosa.DefaultPartitionN 2021-01-06 22:45:44 -06:00
Travis
4515a24e48
change all references to use subpackages: topology, net 2021-01-06 16:09:24 -06:00
Matt Jaffee
bed2cffd5e
fix test failures in case of running Pilosa on system 
If you're running a Pilosa with mostly default configuration on your
system, some of these tests would fail due to things like port
conflicts. These changes address the most common failures.
 2020-12-29 13:57:18 -06:00
Maxton Huff
dee700741a deprecate cluster.long-query-time and create long-query-time 
moved lonquerytime from cluster into server and moved cluster.longquerytime into top level config
kept cluster.longquerytime for backwards compatibility, favored if both longquerytime options are present
 2020-12-10 10:51:31 -06:00
Cody Soyland
bc94c7bcdf Remove lmdb dependency and references, vendor Barrier 2020-12-09 08:43:23 -06:00
Seebs
3d802f428b add Distinct, be more consistent about using correct rand.Rand 
The RandomQueryConfig has its own seeded RNG, but we didn't always use
it (especially in the last two commits, but also I think in one previous
thing), so let's use it more consistently.

We check for int fields (as opposed to decimal), and if we find them,
we add Distinct to our list of potential queries to use if and only
if we've got a depth of at least one so there'd be a child query under
the current query, and if we do, grab one of the int (not decimal)
fields and do a query on that.
 2020-11-23 09:54:59 -06:00
Seebs
796d646a0c handle integer fields 
Extend field handling to include int (and decimal) fields, allowing them
to have range operations specified on them.
 2020-11-23 09:51:29 -06:00
Seebs
2a2a8abbd4 support time fields 
For time fields, allow specifying a range of times, then 19/20 times,
specify "from" and "to" times in that range when querying those fields,
rather than just looking at the standard view all the time.
 2020-11-23 09:51:29 -06:00
Seebs
48b97d6ad4 report QPS after last query 
More generally, report QPS not at 0 queries, which is boring, but every
100 queries *and* after the last query if the last query wasn't at a
multiple of 100 queries. Makes the output slightly more useful, I think.
 2020-11-23 09:51:29 -06:00
Ben Johnson
8de1959938 Remove RBF exclusive/direct write. 2020-11-09 08:23:09 -07:00
Jason E. Aten
458095a707 rbf default. Add TODO comments, slurp -profile returns a cpu profile 
- default Tx is once again RBF, changed from bolt.
- document the RBF code review comments that were not addressed
  before #1052 was merged, so they don't get lost.
- they should be easily addressed by replaying the entire WAL file
  rather than from the DB meta page 0 notion of the last WalID
- cleanup rbf/cfg/cfg.go stale comments, ensure default0 respected.
  1 msec checkpoint time, 1MB wal segment defaults.
- return a specific error, ErrNoMetaFound, from findNextWALMetaPage()
  rather than io.EOF, since there actually wasn't any file IO involved.
- add http handlers for /cpu-profile/start and /cpu-profile/stop
  in http/handler.go enable CPU profiling at specific time points
  during an ingest or other operation.
 2020-11-04 01:36:17 +00:00
Ben Johnson
d5369a084e Improve RBF CLI help 2020-10-30 07:25:36 -06:00
Ben Johnson
2b1d790625 Fix tree key formatting; add --with-tree option 2020-10-27 08:24:56 -06:00
Ben Johnson
7fa1a5edd2 Add RBF CLI commands 2020-10-27 08:12:22 -06:00
jaten-molecula
20d4268676
Merge branch 'master' into rbf_config 2020-10-14 07:19:37 -05:00
Jason E. Aten
50a8db854d pilosa-fsck: the -readers flag controls parallelism 
- add path info to the panic if we find a corrupt boltdb
   translation store.
 2020-10-13 18:58:38 -05:00
J
3ad8ec7f0f rbf: add runtime options to DB struct 
- FsyncEnabled and DoAllocZero moved to DB struct.
 - deletes unused xrbrsupport.go and cmd/convert
 - fixes #941
 2020-10-12 18:35:00 -05:00
Jason Aten
8da15b18d4 pilosa-fsck: parallelize Index.ComputeTranslatorSummary 
- add -index option
 2020-10-09 15:53:29 -05:00
Jason Aten
937a1afade pilosa/cmd/random-query: generate random queries from existing schema 
- fixes molecula/molecula#136
 - flag -n 0 does continuous queries until process is killed.
 2020-10-08 18:38:01 -05:00
Jason Aten
a63c6ee2b2 pilosa-fsck: multi-index counts and repairs. 
- multiple indexes repaired at once could crosstalk. Fixed.
 - the counts of keys and ids are now broken down by index.
 2020-10-06 21:10:52 -05:00
jaten-molecula
ce3a51eb87
Merge branch 'master' into fsck_msg 2020-10-06 09:28:36 -05:00
Jason Aten
6ddeec2ee8 print time at end of fsck run, even if we return 1 to shell 2020-10-06 08:19:53 -05:00
Travis
675373f0f3
adjust some wording and typos in the fsck design doc 2020-10-05 14:38:57 -05:00
Jason E. Aten
05d4b664df groom 2020-10-05 13:29:29 -05:00
Jason E. Aten
9dacbccf8d DESIGN.md for pilosa-fsck 
- Orient users to pilosa-fsck by giving an overview of its operations.
 2020-10-05 12:51:26 -05:00
Jason E. Aten
fe425a84c0 pilosa-fsck: scan and repair of pilosa backups 
- the -fix flag repairs replication errors by copying from the primary.
 - the -fixkeys flag repairs any string key translation issues.
 - make pilosa-fsck installs pilosa-fsck and builds release-pilosa-fsck.COMMIT.GOOS.tar.gz release tarbar
 2020-10-02 16:47:56 -05:00
jaten-molecula
2cc7c829cd
Merge branch 'master' into remove-blake3-duplicated-code 2020-09-11 01:14:02 -04:00
Ben Johnson
a51c530763 Add direct import option; remove tx from fragment.Open() 2020-09-08 08:13:16 -06:00
Antonio Navarro Perez
ca340f7e62 [hash] Remove duplicated blake3 code 
blake3 code is used in several places on the code. The file was
duplicated on root and rbf package.

To avoid cyclic dependencies, I moved it to hash package. Some methods
must be public to use them in different places.

HashOfDir method was removed. Not used.

Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
 2020-09-07 12:24:22 +02:00
Ben Johnson
150c8a5b06 database per shard, HolderConfig, rbf bit-wise import speedups. 
- introduce Query Context (Qcx) for managing database-per-shard.
- replaces the MultiTx, so mtx.go is retired and removed.
- introduces the HolderConfig struct and all Holders now have
  a path from birth.
- rbf speedups on bitwise writes
- badgerdb is removed due to unresolvable write conflicts.

fixes #703 #676
 2020-09-04 13:00:33 -05:00
Jason Aten
1e87d113f1 pilosa-chk: col, row translation reported, -v shows fragment checksum 2020-09-03 18:14:49 -05:00
Jason Aten
56803e6632 pilosa/cmd/translatorchk checksums and summarizes key counts from column key translators. 
opens ~/.pilosa/index/_keys boltdbs and hashes the StringKey->ID mappings.
 2020-09-03 15:23:05 -05:00
Nia Weiss
95d261b4d7
configure slurp 2020-09-02 12:53:16 -04:00