Commit graph

23 commits

Author SHA1 Message Date
Travis
002aee63e3
remove code related to pilosa-chk 2021-02-05 16:13:43 -06:00
Travis
134abda51b
Implement snap := ClusterSnapshot()
Below is the list of instance of `ClusterSnapshot()` in the latest
`with-etcd` code. Some of these may not yet exist in the `disco` branch,
but this commit is implementing any that currently apply.

==========================
Done:
==========================
index.go
930:	snap := NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)
1072:	snap := NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)

cmd/pilosa-fsck/fsck.go
786:	snap := pilosa.NewClusterSnapshot(cfg.topo, cfg.topo.Hasher, cfg.topo.ReplicaN)

boltdb/translate.go
558:	snap := pilosa.NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)
1264:	snap := pilosa.NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)

fragment.go
3448:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
3568:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
3620:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)

==========================
Remaining:
==========================

cluster.go
371:	snap := NewClusterSnapshot(NewLocalNoder(nodes), c.Hasher, c.ReplicaN)
474:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
639:	fSnap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
640:	toSnap := NewClusterSnapshot(to.noder, c.Hasher, to.ReplicaN)
703:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1475:		snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1502:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1941:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1986:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
2049:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
2126:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)

api.go
475:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
604:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
690:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
1684:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
1946:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)

executor.go
3781:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4157:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4200:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4243:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4517:	snap := NewClusterSnapshot(NewLocalNoder(e.Cluster.Nodes()), e.Cluster.Hasher, e.Cluster.ReplicaN)

holder.go
1465:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1668:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1889:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1963:	snap := NewClusterSnapshot(c.Cluster.noder, c.Cluster.Hasher, c.Cluster.ReplicaN)
2021-01-06 22:53:20 -06:00
Nia Weiss
8a130c150e
address review comments 2020-10-22 10:47:28 -04:00
Nia Weiss
22d6011d05
apply "maybe" key translation WIP 2020-10-22 10:47:25 -04:00
Seebs
76fe49d390 un-disable checkptr by fixing the memory problems
Step one: switch to etcd.io's bbolt fork of boltdb.

The etcd-io fork of boltdb isn't archived, and has fixes for boltdb's
interactions with checkptr, allowing us to drop the checkptr-disabling
hackery.

This seems to be a drop-in replacement; etcd/bbolt says that the file
format is "fixed" (I believe in the sense of "unchanging"), and I can
run pilosa on an existing data directory with this.

Step two:

Fix missing caps in roaring.go that were also triggering the same
issues.
2020-10-15 19:12:47 -05:00
Jason E. Aten
50a8db854d pilosa-fsck: the -readers flag controls parallelism
- add path info to the panic if we find a corrupt boltdb
   translation store.
2020-10-13 18:58:38 -05:00
Jason Aten
a63c6ee2b2 pilosa-fsck: multi-index counts and repairs.
- multiple indexes repaired at once could crosstalk. Fixed.
 - the counts of keys and ids are now broken down by index.
2020-10-06 21:10:52 -05:00
Jason E. Aten
fe425a84c0 pilosa-fsck: scan and repair of pilosa backups
- the -fix flag repairs replication errors by copying from the primary.
 - the -fixkeys flag repairs any string key translation issues.
 - make pilosa-fsck installs pilosa-fsck and builds release-pilosa-fsck.COMMIT.GOOS.tar.gz release tarbar
2020-10-02 16:47:56 -05:00
Kuba Podgórski
650244214d Translate only on coordinator/primary
- This is the commit message #3:
2020-09-17 17:03:29 -05:00
Kuba Podgórski
76715a3f9d Fix TranslateStore writable 2020-09-15 19:09:59 +02:00
Jason Aten
1e87d113f1 pilosa-chk: col, row translation reported, -v shows fragment checksum 2020-09-03 18:14:49 -05:00
Jason Aten
56803e6632 pilosa/cmd/translatorchk checksums and summarizes key counts from column key translators.
opens ~/.pilosa/index/_keys boltdbs and hashes the StringKey->ID mappings.
2020-09-03 15:23:05 -05:00
Nia Weiss
0a9699490b
batch the translation of field keys in results 2020-08-28 11:56:37 -04:00
Kuba Podgórski
9cf58ddb1c Add writable argument to TranslateKey functions. 2020-08-26 09:57:50 +02:00
Travis
4c311aa1a7
write to temp partition file. use io.Copy 2020-03-30 21:24:16 -05:00
Travis
2724ecfd5f
WIP: include translate partitions in cluster resize instructions
This commit adds `TranslationSources` to the cluster
`ResizeInstruction`. These are the sources of translation
partitions which the receiving node needs in order to support
partition distribution in the new, resized cluster.

This also fixes a bug where index options were not being
encode in the proto Index object. That meant that the schema
transferred via protobuf was not correct. The reason why
things normally worked is because index creation typically
happens on the CreateIndex message, which does include the
options.

TODO:

- [ ] implement the TranslateStore interface for `InMemTranslateStore`
and `mock.TranslateStore`
- [ ] surely need some more tests around the `ReadFrom` and `WriteTo`
2020-03-30 21:24:16 -05:00
Kuba Podgórski
f3f11f4a44
Let translate keys as empty strings (#120) 2020-02-18 12:54:24 +01:00
Travis
49c8bf01a0
WIP: Thread OpenTranslateStore through Holder to Index 2020-02-12 10:25:25 -06:00
Ben Johnson
9647d9b4bb fixing additional tests 2020-01-08 09:47:43 -07:00
Ben Johnson
e3606d6615 fix id generation 2020-01-08 09:47:43 -07:00
Ben Johnson
82910911dd refactoring id partitioning 2020-01-08 09:47:43 -07:00
Ben Johnson
7215bfd16c Implement translator store sharding 2020-01-08 09:47:43 -07:00
Ben Johnson
e844e1ad75
Translation store refactor 2019-10-09 08:59:41 -06:00