scriptoriumadmin/featurebase
1
0
Fork 0
mirror of https://github.com/featurebasedb/featurebase.git synced 2026-09-05 08:10:50 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
7769 commits 45 branches 174 tags 85 MiB
Commit graph

29 commits

Author SHA1 Message Date
Mahesh Arumugam
858f889745 FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
Nia Weiss
c0ef297b11
standardize key translation on the find and create methods 
This migrates existing code from the old TranslateKey(s) endpoints to the newer CreateKeys and FindKeys endpoints.
The CreateKeys and FindKeys endpoints were created previously as the TranslateKeys endpoint had no way to behave sanely when the looked-up key did not exist (the parallel-arrays representation did not have a good way to represent a missing key).
This change also removes the old TranslateKey(s) functions from the translation stores.
It leaves a wrapper emulating the TranslateKey(s) endpoints so that old idk still works for now.
 2021-06-07 14:47:52 -04:00
Nia Weiss
a8f7ec4a12
execute like queries on the primary's key translation database 
This works around an issue where unreplicated keys will not be matched everywhere.
This also avoids the cost of creating millions of bolt read transactions and allocating strings.
 2021-05-27 14:45:10 -04:00
Nia Weiss
f4ba34247f
remove attributes 
Attributes are unmaintained and unused.
They have become more of a liability than a benefit.
This change eliminates them from the codebase.
The only user-visible change (assuming that attrs are not used) is that the attrs field will no longer appear in row JSON.
 2021-05-14 10:28:08 -04:00
Seebs
9e6ec3b17a break boltDB operations into chunks 
boltDB's bucket.Put() is quadratic on "new keys put into a bucket during
this transaction", which is why BoltDB has warnings not to use it with over
100k new keys at a time. The translation store logic wasn't actually using
that. The actual value picked is smaller, based on some half-baked benchmarking.

We also avoid heap-allocating separate 16-byte (not 8-byte, of course,
because make(...) is *helping*) chunks twice for each key we insert, instead
allocating a single buffer which we reuse for each new transaction.

Also fixed a check against the nilness of the wrong pointer and generally
made CreateKeys and TranslateKeys a little more similar.
 2021-05-07 13:19:15 -05:00
Seebs
1045268f01 use testhook to ensure temporary files and directories are cleaned up 
In nearly all cases, we can just switch ioutil.TempDir->testhook.TempDir
and similarly for TempFile. There's one case where we can't because we
need files to be removed before tests are over.

Also in the process give identifiable names to a lot of temporary files
and make sure they're being cleaned up, and don't use "/tmp/foo" as a
file name in a test that could be running in more than one test process
at once. :)
 2021-03-11 19:42:10 -06:00
Travis
002aee63e3
remove code related to pilosa-chk 2021-02-05 16:13:43 -06:00
Travis
134abda51b
Implement snap := ClusterSnapshot() 
Below is the list of instance of `ClusterSnapshot()` in the latest
`with-etcd` code. Some of these may not yet exist in the `disco` branch,
but this commit is implementing any that currently apply.

==========================
Done:
==========================
index.go
930:	snap := NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)
1072:	snap := NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)

cmd/pilosa-fsck/fsck.go
786:	snap := pilosa.NewClusterSnapshot(cfg.topo, cfg.topo.Hasher, cfg.topo.ReplicaN)

boltdb/translate.go
558:	snap := pilosa.NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)
1264:	snap := pilosa.NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)

fragment.go
3448:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
3568:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
3620:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)

==========================
Remaining:
==========================

cluster.go
371:	snap := NewClusterSnapshot(NewLocalNoder(nodes), c.Hasher, c.ReplicaN)
474:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
639:	fSnap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
640:	toSnap := NewClusterSnapshot(to.noder, c.Hasher, to.ReplicaN)
703:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1475:		snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1502:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1941:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1986:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
2049:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
2126:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)

api.go
475:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
604:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
690:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
1684:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
1946:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)

executor.go
3781:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4157:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4200:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4243:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4517:	snap := NewClusterSnapshot(NewLocalNoder(e.Cluster.Nodes()), e.Cluster.Hasher, e.Cluster.ReplicaN)

holder.go
1465:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1668:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1889:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1963:	snap := NewClusterSnapshot(c.Cluster.noder, c.Cluster.Hasher, c.Cluster.ReplicaN)
 2021-01-06 22:53:20 -06:00
Nia Weiss
8a130c150e
address review comments 2020-10-22 10:47:28 -04:00
Nia Weiss
22d6011d05
apply "maybe" key translation WIP 2020-10-22 10:47:25 -04:00
Seebs
76fe49d390 un-disable checkptr by fixing the memory problems 
Step one: switch to etcd.io's bbolt fork of boltdb.

The etcd-io fork of boltdb isn't archived, and has fixes for boltdb's
interactions with checkptr, allowing us to drop the checkptr-disabling
hackery.

This seems to be a drop-in replacement; etcd/bbolt says that the file
format is "fixed" (I believe in the sense of "unchanging"), and I can
run pilosa on an existing data directory with this.

Step two:

Fix missing caps in roaring.go that were also triggering the same
issues.
 2020-10-15 19:12:47 -05:00
Jason E. Aten
50a8db854d pilosa-fsck: the -readers flag controls parallelism 
- add path info to the panic if we find a corrupt boltdb
   translation store.
 2020-10-13 18:58:38 -05:00
Jason Aten
a63c6ee2b2 pilosa-fsck: multi-index counts and repairs. 
- multiple indexes repaired at once could crosstalk. Fixed.
 - the counts of keys and ids are now broken down by index.
 2020-10-06 21:10:52 -05:00
Jason E. Aten
fe425a84c0 pilosa-fsck: scan and repair of pilosa backups 
- the -fix flag repairs replication errors by copying from the primary.
 - the -fixkeys flag repairs any string key translation issues.
 - make pilosa-fsck installs pilosa-fsck and builds release-pilosa-fsck.COMMIT.GOOS.tar.gz release tarbar
 2020-10-02 16:47:56 -05:00
Kuba Podgórski
650244214d Translate only on coordinator/primary 
- This is the commit message #3:
 2020-09-17 17:03:29 -05:00
Kuba Podgórski
76715a3f9d Fix TranslateStore writable 2020-09-15 19:09:59 +02:00
Jason Aten
1e87d113f1 pilosa-chk: col, row translation reported, -v shows fragment checksum 2020-09-03 18:14:49 -05:00
Jason Aten
56803e6632 pilosa/cmd/translatorchk checksums and summarizes key counts from column key translators. 
opens ~/.pilosa/index/_keys boltdbs and hashes the StringKey->ID mappings.
 2020-09-03 15:23:05 -05:00
Nia Weiss
0a9699490b
batch the translation of field keys in results 2020-08-28 11:56:37 -04:00
Kuba Podgórski
9cf58ddb1c Add writable argument to TranslateKey functions. 2020-08-26 09:57:50 +02:00
Travis
4c311aa1a7
write to temp partition file. use io.Copy 2020-03-30 21:24:16 -05:00
Travis
2724ecfd5f
WIP: include translate partitions in cluster resize instructions 
This commit adds `TranslationSources` to the cluster
`ResizeInstruction`. These are the sources of translation
partitions which the receiving node needs in order to support
partition distribution in the new, resized cluster.

This also fixes a bug where index options were not being
encode in the proto Index object. That meant that the schema
transferred via protobuf was not correct. The reason why
things normally worked is because index creation typically
happens on the CreateIndex message, which does include the
options.

TODO:

- [ ] implement the TranslateStore interface for `InMemTranslateStore`
and `mock.TranslateStore`
- [ ] surely need some more tests around the `ReadFrom` and `WriteTo`
 2020-03-30 21:24:16 -05:00
Kuba Podgórski
f3f11f4a44
Let translate keys as empty strings (#120) 2020-02-18 12:54:24 +01:00
Travis
49c8bf01a0
WIP: Thread OpenTranslateStore through Holder to Index 2020-02-12 10:25:25 -06:00
Ben Johnson
9647d9b4bb fixing additional tests 2020-01-08 09:47:43 -07:00
Ben Johnson
e3606d6615 fix id generation 2020-01-08 09:47:43 -07:00
Ben Johnson
82910911dd refactoring id partitioning 2020-01-08 09:47:43 -07:00
Ben Johnson
7215bfd16c Implement translator store sharding 2020-01-08 09:47:43 -07:00
Ben Johnson
e844e1ad75
Translation store refactor 2019-10-09 08:59:41 -06:00