This commit removes the previous `MaxShard` tracking and replaces
it with an `Available Shards` set tracking. This allows sparse shard
tracking without implicitly tracking all shards in between.
monitorReplication is now not allowed to return until the goroutine it starts
cancels the context. Previously, it could return just before the context was
canceled which caused a race between its internal goroutine and
handlePrimaryStoreEvent which recreates a channel which that internal goroutine
was listening on.
handlePrimaryStoreEvent already correctly made sure that monitorReplication had
returned before recreating the channel, so proper handling of the sub-goroutine
of monitorReplication was all that was needed to avoid this race.
node (as opposed to sending to shard0, which may or may not be the
coordinator). It adds a `Nodes()` method to the `InternalClient`
which is used by the importer to determine which node is the
coordinator.
previously, losing a node could cause the cluster to go from NORMAL->DEGRADED,
but adding a node would not cause it to go from STARTING->DEGRADED. Cody brought
this up in code review.
now, nodes which have failed and been removed from the running cluster state can
still be manually removed to trigger a cluster resize event. This is important
because otherwise there is no way to cause the cluster to resize itself if a
node fails and you don't want to add a node to replace it.
cluster is in degraded state when some number of nodes greater than 0 but less
than replicaN have failed. This is sort of a hybrid of "STARTING" and "NORMAL"
states because we can still respond to queries as in the NORMAL state, but we
need to be alert to re-add nodes to the cluster if they come back online which
required some changes to the cluster logic.
In order to make debugging easier, the test.MustRunCluster functionality now
names the nodes in the cluster explicitly as "node0", "node1", etc. "node0" is
the coordinator.
A number of TODOs are left in the test for scenarios that need to be checked.
