Commit graph

319 commits

Author SHA1 Message Date
Travis
2bbe1fdde0
remove remaining references to "coordinator" 2021-02-23 17:23:09 -06:00
Travis
24c5b654c6
change IndexOptions to reference by value 2021-02-23 10:09:54 -06:00
Kuba Podgórski
bfc24a1745 Check state in shardsByNode once stator is implemented 2021-02-17 16:02:28 +01:00
Seebs
82a975fd2a avoid race conditions on nodes
Turns out we sometimes modify returned nodes. Handle this better, but
also fix up some cases where we were generating node lists we didn't really
need to answer simple questions.
2021-02-15 10:20:31 -06:00
Travis
9fa271a7b8
remove dead code (api.HostStates()) 2021-02-12 21:51:27 -06:00
Travis
8fe1b9a37c
store views in etcd via Schemator 2021-02-10 11:47:26 -06:00
Travis
114f6a8751
add withViews argument to api.Schema() method 2021-02-08 10:42:55 -06:00
Antonio Navarro Perez
b1ff8e55cd
Unify state
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
2021-02-06 16:53:42 -06:00
Travis
30d4687a99
remove type Topology 2021-02-05 16:13:43 -06:00
Travis
ec91dad198
remove dead code 2021-02-05 11:31:33 -06:00
Kuba Podgórski
35b9d41da1
Merge branch 'disco-config-noder' into resizer-interface 2021-02-05 11:57:51 +01:00
Kuba Podgórski
ab37bf5c7b Apply resizer interface (remove and add node) 2021-02-04 20:26:41 +01:00
Travis
a4b37273ea
remove the rest of the gossip code (except config) 2021-02-04 13:03:02 -06:00
Travis
afc53e1163
remove ReceiveEvent 2021-02-03 23:31:38 -06:00
Travis
6e4ea21ce5
remove gossip listenForJoins 2021-02-03 22:34:45 -06:00
Travis
da804ee6d5
linter fixes 2021-02-03 15:11:24 -06:00
Travis
f9661b7b81
add PrimaryNodeID() method to Noder interface 2021-02-03 15:11:24 -06:00
Kuba Podgórski
6601835ba1 Remove public mutex from Node 2021-02-03 19:37:54 +01:00
Kuba Podgórski
6826997852 Remove state member from cluster.
Remove all function SetState like. Stop broadcasting cluster state.
2021-02-03 15:16:59 +01:00
Travis
4c1d94da14
remove some dead code related to coordinator 2021-02-02 15:59:36 -06:00
Antonio Navarro Perez
c45e21640c
Change coordinator to primary
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
2021-02-02 15:56:12 -06:00
Kuba Podgórski
a16a83445b Apply stator 2021-02-02 19:36:34 +01:00
Travis
629bfa3ac8
add more AwaitState calls in the tests 2021-02-01 21:21:06 -06:00
Travis
26176c15eb
fix linter issues (wrap all ClusterStates in string() until we update the type) 2021-02-01 16:57:35 -06:00
Kuba Podgórski
b611480499
disco State 2021-02-01 16:37:04 -06:00
Travis
855e1b35f5
more use of noder; remove c.nodes
disable some of the gossip logic

implement some of the stator logic
2021-01-31 23:42:49 -06:00
Travis
b80f5099b2
more coordinator/primary cleanup 2021-01-25 23:20:08 -06:00
Travis
ace4dea46f
address some test failures due to random ordered etcd ID 2021-01-25 00:52:49 -06:00
Travis
2f66501160
finish implementing snap := ClusterSnapshot() 2021-01-24 23:22:32 -06:00
Travis
a196e1e74c
use etcd for node.ID
this commit adds a temporation interface for starting gossip.
we needed this so we can start gossip AFTER setting up the node,
but before waitingForJoins.
2021-01-23 19:52:58 -06:00
Travis
bc13834343
disco/etcd work: fix lots of races, start all cluster nodes at once.
port mapper gives out ports from 63000-65000 for the tests

fix another race

http test uses port.MustGetPort

rbf: remove :0 port request

ocd happy

test fix for grpc listener address already in use

test/disco allocates BindGRPC port from the port mapper

dump stack on each GetPort

verify each port is usable right away

server/config.go has Config.Validate() now

panic if gossip port is 0. validate server.Config

fix another gossip port 0

builds

quiet, don't dump stack on each port alloc

builds

happy linter

even gossip fallback should not be zero but rather use the port mapper
2021-01-12 21:06:12 -06:00
Travis
8dbfae1d86
temporarily have cluster implement Noder 2021-01-07 13:45:46 -06:00
Travis
134abda51b
Implement snap := ClusterSnapshot()
Below is the list of instance of `ClusterSnapshot()` in the latest
`with-etcd` code. Some of these may not yet exist in the `disco` branch,
but this commit is implementing any that currently apply.

==========================
Done:
==========================
index.go
930:	snap := NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)
1072:	snap := NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)

cmd/pilosa-fsck/fsck.go
786:	snap := pilosa.NewClusterSnapshot(cfg.topo, cfg.topo.Hasher, cfg.topo.ReplicaN)

boltdb/translate.go
558:	snap := pilosa.NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)
1264:	snap := pilosa.NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)

fragment.go
3448:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
3568:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
3620:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)

==========================
Remaining:
==========================

cluster.go
371:	snap := NewClusterSnapshot(NewLocalNoder(nodes), c.Hasher, c.ReplicaN)
474:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
639:	fSnap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
640:	toSnap := NewClusterSnapshot(to.noder, c.Hasher, to.ReplicaN)
703:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1475:		snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1502:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1941:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1986:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
2049:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
2126:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)

api.go
475:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
604:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
690:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
1684:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
1946:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)

executor.go
3781:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4157:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4200:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4243:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4517:	snap := NewClusterSnapshot(NewLocalNoder(e.Cluster.Nodes()), e.Cluster.Hasher, e.Cluster.ReplicaN)

holder.go
1465:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1668:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1889:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1963:	snap := NewClusterSnapshot(c.Cluster.noder, c.Cluster.Hasher, c.Cluster.ReplicaN)
2021-01-06 22:53:20 -06:00
Travis
3f26d667b4
remove pilosa.DefaultPartitionN 2021-01-06 22:45:44 -06:00
Travis
4515a24e48
change all references to use subpackages: topology, net 2021-01-06 16:09:24 -06:00
Matt Jaffee
16fd6a7edd
add tests for GroupBy(Distinct), fix various problems
We execute the aggregate Distinct calls after the GroupBy is complete,
and we need these to act like non-remote calls in that they forward to
all nodes, but like remote calls in that they bypass key
translation. Added a "PreTranslated" flag to the QueryRequest to
achieve this.

Discovered an issue where a nil *Row in EmbeddedData would cause a
panic in the protobuf serialization. Changed the encoding code we
control to never pass a nil *Row.

Got fed up with lack of context on errors and added wrapping to all
calls under executor.executeCall as well as a few other places.

Handled a situation where not having data on a shard for a particular
field could cause a query to error instead of just treating that
fragment as being empty. (see the switch in executeDistinctShardSet)

Stopped GroupBy from executing the Count(Distinct) aggregate on Remote
calls.

Fixed a longstanding issue where errors retrieved from remote query
calls had a garbage character at the front due to treating a protobuf
payload as an error message instead of decoding it. (see
http/client.go)
2020-12-30 08:13:41 -06:00
Maxton Huff
dee700741a deprecate cluster.long-query-time and create long-query-time
moved lonquerytime from cluster into server and moved cluster.longquerytime into top level config
kept cluster.longquerytime for backwards compatibility, favored if both longquerytime options are present
2020-12-10 10:51:31 -06:00
Matt Jaffee
a48bf28be2
don't try to translate keys on unkeyed indexes
this causes a few things to error earlier than they otherwise would
have, hence the changed tests.
2020-11-20 14:48:28 -06:00
Nia Weiss
b0a588f8dd
finish translate-if-exists 2020-10-22 10:50:09 -04:00
Nia Weiss
a74eda1eb0
cleanup of key translation fix 2020-10-22 10:50:08 -04:00
Nia Weiss
13a19aeb42
query translation WIP 2020-10-22 10:47:29 -04:00
Nia Weiss
8a130c150e
address review comments 2020-10-22 10:47:28 -04:00
Nia Weiss
22d6011d05
apply "maybe" key translation WIP 2020-10-22 10:47:25 -04:00
Alan Bernstein
94e82aea86 Consider available shards 2020-10-14 21:11:02 -05:00
alan
2a902b3a7b Finish basic shard-distribution endpoint 2020-10-14 21:11:02 -05:00
Alan Bernstein
98cd2dc441 WIP shard distribution endpoint 2020-10-14 21:11:02 -05:00
Jason Aten
a63c6ee2b2 pilosa-fsck: multi-index counts and repairs.
- multiple indexes repaired at once could crosstalk. Fixed.
 - the counts of keys and ids are now broken down by index.
2020-10-06 21:10:52 -05:00
Jason E. Aten
fe425a84c0 pilosa-fsck: scan and repair of pilosa backups
- the -fix flag repairs replication errors by copying from the primary.
 - the -fixkeys flag repairs any string key translation issues.
 - make pilosa-fsck installs pilosa-fsck and builds release-pilosa-fsck.COMMIT.GOOS.tar.gz release tarbar
2020-10-02 16:47:56 -05:00
Matt Jaffee
c2a93cc886
retrun confirmDownRetries to 10, tourney mac compatible
At some point the cluster code was modified to do 120 tries to confirm
if a node was down which is a bit excessive for production. My
understanding is that this was done to help trigger or fix a problem
during testing which is hopefully no longer relevant.

Also changed the tournament script to use what I think are more
standard bash-isms that work on mac. Please confirm this still works
on Linux as well.
2020-10-02 14:24:16 -05:00
Alan Bernstein
5705b45864 Add a CLI flag for cluster name and include it in /status response 2020-10-01 10:43:49 -05:00