Commit graph

44 commits

Author SHA1 Message Date
Travis
002aee63e3
remove code related to pilosa-chk 2021-02-05 16:13:43 -06:00
Travis
bc13834343
disco/etcd work: fix lots of races, start all cluster nodes at once.
port mapper gives out ports from 63000-65000 for the tests

fix another race

http test uses port.MustGetPort

rbf: remove :0 port request

ocd happy

test fix for grpc listener address already in use

test/disco allocates BindGRPC port from the port mapper

dump stack on each GetPort

verify each port is usable right away

server/config.go has Config.Validate() now

panic if gossip port is 0. validate server.Config

fix another gossip port 0

builds

quiet, don't dump stack on each port alloc

builds

happy linter

even gossip fallback should not be zero but rather use the port mapper
2021-01-12 21:06:12 -06:00
Travis
134abda51b
Implement snap := ClusterSnapshot()
Below is the list of instance of `ClusterSnapshot()` in the latest
`with-etcd` code. Some of these may not yet exist in the `disco` branch,
but this commit is implementing any that currently apply.

==========================
Done:
==========================
index.go
930:	snap := NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)
1072:	snap := NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)

cmd/pilosa-fsck/fsck.go
786:	snap := pilosa.NewClusterSnapshot(cfg.topo, cfg.topo.Hasher, cfg.topo.ReplicaN)

boltdb/translate.go
558:	snap := pilosa.NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)
1264:	snap := pilosa.NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)

fragment.go
3448:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
3568:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
3620:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)

==========================
Remaining:
==========================

cluster.go
371:	snap := NewClusterSnapshot(NewLocalNoder(nodes), c.Hasher, c.ReplicaN)
474:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
639:	fSnap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
640:	toSnap := NewClusterSnapshot(to.noder, c.Hasher, to.ReplicaN)
703:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1475:		snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1502:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1941:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1986:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
2049:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
2126:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)

api.go
475:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
604:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
690:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
1684:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
1946:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)

executor.go
3781:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4157:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4200:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4243:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4517:	snap := NewClusterSnapshot(NewLocalNoder(e.Cluster.Nodes()), e.Cluster.Hasher, e.Cluster.ReplicaN)

holder.go
1465:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1668:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1889:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1963:	snap := NewClusterSnapshot(c.Cluster.noder, c.Cluster.Hasher, c.Cluster.ReplicaN)
2021-01-06 22:53:20 -06:00
Travis
3f26d667b4
remove pilosa.DefaultPartitionN 2021-01-06 22:45:44 -06:00
Nia Weiss
8a130c150e
address review comments 2020-10-22 10:47:28 -04:00
Nia Weiss
22d6011d05
apply "maybe" key translation WIP 2020-10-22 10:47:25 -04:00
Seebs
76fe49d390 un-disable checkptr by fixing the memory problems
Step one: switch to etcd.io's bbolt fork of boltdb.

The etcd-io fork of boltdb isn't archived, and has fixes for boltdb's
interactions with checkptr, allowing us to drop the checkptr-disabling
hackery.

This seems to be a drop-in replacement; etcd/bbolt says that the file
format is "fixed" (I believe in the sense of "unchanging"), and I can
run pilosa on an existing data directory with this.

Step two:

Fix missing caps in roaring.go that were also triggering the same
issues.
2020-10-15 19:12:47 -05:00
Jason E. Aten
50a8db854d pilosa-fsck: the -readers flag controls parallelism
- add path info to the panic if we find a corrupt boltdb
   translation store.
2020-10-13 18:58:38 -05:00
Jason Aten
a63c6ee2b2 pilosa-fsck: multi-index counts and repairs.
- multiple indexes repaired at once could crosstalk. Fixed.
 - the counts of keys and ids are now broken down by index.
2020-10-06 21:10:52 -05:00
Jason E. Aten
fe425a84c0 pilosa-fsck: scan and repair of pilosa backups
- the -fix flag repairs replication errors by copying from the primary.
 - the -fixkeys flag repairs any string key translation issues.
 - make pilosa-fsck installs pilosa-fsck and builds release-pilosa-fsck.COMMIT.GOOS.tar.gz release tarbar
2020-10-02 16:47:56 -05:00
Nia Weiss
1b210080c6
fix race condition when resetting translation
Previously, we never waited for translation sync goroutines to stop.
That issue should be mostly harmless in the normal path.
Additionally, this waits for the translation sync to shut down when stopping the server.
2020-09-29 13:54:16 -04:00
Kuba Podgórski
650244214d Translate only on coordinator/primary
- This is the commit message #3:
2020-09-17 17:03:29 -05:00
Kuba Podgórski
76715a3f9d Fix TranslateStore writable 2020-09-15 19:09:59 +02:00
Jason Aten
1e87d113f1 pilosa-chk: col, row translation reported, -v shows fragment checksum 2020-09-03 18:14:49 -05:00
Jason Aten
56803e6632 pilosa/cmd/translatorchk checksums and summarizes key counts from column key translators.
opens ~/.pilosa/index/_keys boltdbs and hashes the StringKey->ID mappings.
2020-09-03 15:23:05 -05:00
Nia Weiss
0a9699490b
batch the translation of field keys in results 2020-08-28 11:56:37 -04:00
Kuba Podgórski
9cf58ddb1c Add writable argument to TranslateKey functions. 2020-08-26 09:57:50 +02:00
Travis
182c1d3c42 alter tests to allow for shardwidth22
also, reset BitDepth on field and bsiGroup during
importRoaringOverwrite
2020-04-02 17:32:49 -05:00
Travis
4c311aa1a7
write to temp partition file. use io.Copy 2020-03-30 21:24:16 -05:00
Travis
2724ecfd5f
WIP: include translate partitions in cluster resize instructions
This commit adds `TranslationSources` to the cluster
`ResizeInstruction`. These are the sources of translation
partitions which the receiving node needs in order to support
partition distribution in the new, resized cluster.

This also fixes a bug where index options were not being
encode in the proto Index object. That meant that the schema
transferred via protobuf was not correct. The reason why
things normally worked is because index creation typically
happens on the CreateIndex message, which does include the
options.

TODO:

- [ ] implement the TranslateStore interface for `InMemTranslateStore`
and `mock.TranslateStore`
- [ ] surely need some more tests around the `ReadFrom` and `WriteTo`
2020-03-30 21:24:16 -05:00
Kuba Podgórski
f3f11f4a44
Let translate keys as empty strings (#120) 2020-02-18 12:54:24 +01:00
Travis
49c8bf01a0
WIP: Thread OpenTranslateStore through Holder to Index 2020-02-12 10:25:25 -06:00
Ben Johnson
9647d9b4bb fixing additional tests 2020-01-08 09:47:43 -07:00
Ben Johnson
e3606d6615 fix id generation 2020-01-08 09:47:43 -07:00
Ben Johnson
82910911dd refactoring id partitioning 2020-01-08 09:47:43 -07:00
Ben Johnson
7215bfd16c Implement translator store sharding 2020-01-08 09:47:43 -07:00
Ben Johnson
e844e1ad75
Translation store refactor 2019-10-09 08:59:41 -06:00
Ben Johnson
c7c9c1e1d7
v2.0.0
Co-authored-by: Cody Soyland <codysoyland@gmail.com>
2019-10-08 14:56:17 -06:00
Seebs
449c853850 address meta-lint or half-baked lint fixes
Clean up some spelling and consistency issues for the lint
fixes.
2019-04-16 12:08:40 -05:00
Seebs
20a8c48552 boltdb/attrstore.go: fix up lint about error checking
There's two kinds of unchecked errors here. Writes to a hash
(we don't care, hash functions usually don't error in ways we
care about), and rollbacks of non-writing transactions to a
database. After studying the boltdb docs, I concluded that
the recommended solution is to use the `.View(...)` function
instead of directly controlling the transaction, so I switched
the functions to do that.
2019-04-16 12:06:28 -05:00
Cody Soyland
e76c523135
Merge branch 'master' into 1517-metalinter-unconvert 2018-08-02 11:45:24 -05:00
Cody Soyland
dc50204846 Fix linter issues: unconvert 2018-07-20 11:51:55 -05:00
Cody Soyland
e9523063b5 Fix linter issues: unused 2018-07-20 10:33:29 -05:00
Cody Soyland
f01d850b17 Fix linter issues: gosimple 2018-07-20 09:06:43 -05:00
Cody Soyland
0a9e6bca7a Re-add return variable names removed in 2aa4d6b1. 2018-07-19 08:09:05 -05:00
Cody Soyland
2aa4d6b12f Fix linter issues: nakedret 2018-07-18 15:23:36 -05:00
Cody Soyland
f9625ef4fa Fix linter issues: unparam 2018-07-17 12:05:07 -05:00
Cody Soyland
9db2109278 Unexport boltdb.NewAttrCache 2018-07-05 23:11:56 -05:00
Cody Soyland
fcf6517c7e Unexport boltdb.AttrStore 2018-07-05 23:11:56 -05:00
Cody Soyland
43cac45d40 Unexport boltdb.AttrCache 2018-07-05 23:11:56 -05:00
Cody Soyland
817ede49e6 Unexport boltdb.AttrBlockSize 2018-07-05 23:11:56 -05:00
Alan Bernstein
eb888b74f7 Update test error strings 2018-05-10 12:12:55 -05:00
Cody Soyland
f9efa6c579 Use type func(string) AttrStore in place of interface AttrStoreGenerator for simplicity 2018-03-23 13:57:29 -05:00
Travis Turner
b66bedd1ef
put BoltDB behind AttrStore interface 2018-03-19 11:30:41 -05:00