There was a situation where availableShards on a new
node were not in sync with the cluster, so queries
following a resize were incorrect.
- Start a one-node cluster.
- Write data to shards 0 and 1
- Start a second node.
In the case where the hash algo was moving shard 0 to
node1, then node1 only knew about shard 0, so queries
to node1 would be incomplete.
This PR modifies the ResizeInstruction message to replace
`Schema` with `NodeStatus` (which contains both `Schema` and
`AvailableShards`). So now when a resize instruction is received,
the receiving node is able to sync its schema and availableShards.
running
"make clustertests
DOCKER_COMPOSE=internal/clustertests/docker-compose-replication2.yml"
shows this issue (just remove the change in cluster.go).
Also removed two unrelated lines of code that appear to be doing absolutely nothing.
I'd like to add stat tracking to Roaring, which means it
has to be able to import the stats package, which means
stats has to be a package rather than part of the pilosa
package. If stats stops being in pilosa, it still needs
a way to import logger, so logger also has to leave the
pilosa package. Then everything using them needs to import
them and use package selectors on their names.
This doesn't actually add the stats support to roaring,
it just makes it so there's a way to import the stats
code from something in the roaring package.
This commit adds the ability to remove an 'available shard'
from the shard cache. This does not affect shards known to be
available because of local data.
This commit removes the previous `MaxShard` tracking and replaces
it with an `Available Shards` set tracking. This allows sparse shard
tracking without implicitly tracking all shards in between.
previously, losing a node could cause the cluster to go from NORMAL->DEGRADED,
but adding a node would not cause it to go from STARTING->DEGRADED. Cody brought
this up in code review.
now, nodes which have failed and been removed from the running cluster state can
still be manually removed to trigger a cluster resize event. This is important
because otherwise there is no way to cause the cluster to resize itself if a
node fails and you don't want to add a node to replace it.
cluster is in degraded state when some number of nodes greater than 0 but less
than replicaN have failed. This is sort of a hybrid of "STARTING" and "NORMAL"
states because we can still respond to queries as in the NORMAL state, but we
need to be alert to re-add nodes to the cluster if they come back online which
required some changes to the cluster logic.
In order to make debugging easier, the test.MustRunCluster functionality now
names the nodes in the cluster explicitly as "node0", "node1", etc. "node0" is
the coordinator.
A number of TODOs are left in the test for scenarios that need to be checked.
