cluster is in degraded state when some number of nodes greater than 0 but less
than replicaN have failed. This is sort of a hybrid of "STARTING" and "NORMAL"
states because we can still respond to queries as in the NORMAL state, but we
need to be alert to re-add nodes to the cluster if they come back online which
required some changes to the cluster logic.
In order to make debugging easier, the test.MustRunCluster functionality now
names the nodes in the cluster explicitly as "node0", "node1", etc. "node0" is
the coordinator.
A number of TODOs are left in the test for scenarios that need to be checked.
The generate-config command was printing a fixed string rather than calling
NewConfig() which is the canonical source for default config. I also noticed
that we were depending on two different toml libraries, and so collapsed that to
a single one. We have to use pelletier rather than BurntSushi because the viper
library that we use depends on pelletier.
datadog statsd client contained a race condition - was fixed in master
Server.Logger contained a race where multiple loggers could write to the same
output io.Writer
TestMain_FrameRestore contained a race where it tried to change a cluster's
nodes while it was running (which conflicted with antiEntropy reading that
state).
