previously, losing a node could cause the cluster to go from NORMAL->DEGRADED,
but adding a node would not cause it to go from STARTING->DEGRADED. Cody brought
this up in code review.
now, nodes which have failed and been removed from the running cluster state can
still be manually removed to trigger a cluster resize event. This is important
because otherwise there is no way to cause the cluster to resize itself if a
node fails and you don't want to add a node to replace it.
cluster is in degraded state when some number of nodes greater than 0 but less
than replicaN have failed. This is sort of a hybrid of "STARTING" and "NORMAL"
states because we can still respond to queries as in the NORMAL state, but we
need to be alert to re-add nodes to the cluster if they come back online which
required some changes to the cluster logic.
In order to make debugging easier, the test.MustRunCluster functionality now
names the nodes in the cluster explicitly as "node0", "node1", etc. "node0" is
the coordinator.
A number of TODOs are left in the test for scenarios that need to be checked.
gossip now takes a single "MemberServer" which is implemented by server. Several
interfaces have been removed.
MemberServer contains ReceiveMessage which is a superset of the functionality of
ReceiveEvent, LocalStatus and HandleRemoteStatus are all that's left of
StatusHandler - ClusterStatus was not used and is gone. The Node() method is
actually a subset of LocalStatus() functionality. Maybe we should break up
localstatus or remove Node... not sure.
Remove BroadcastReceiver test which was a bit silly.
NodeEvent can now be unexported, and is.
