previously, losing a node could cause the cluster to go from NORMAL->DEGRADED,
but adding a node would not cause it to go from STARTING->DEGRADED. Cody brought
this up in code review.
now, nodes which have failed and been removed from the running cluster state can
still be manually removed to trigger a cluster resize event. This is important
because otherwise there is no way to cause the cluster to resize itself if a
node fails and you don't want to add a node to replace it.
cluster is in degraded state when some number of nodes greater than 0 but less
than replicaN have failed. This is sort of a hybrid of "STARTING" and "NORMAL"
states because we can still respond to queries as in the NORMAL state, but we
need to be alert to re-add nodes to the cluster if they come back online which
required some changes to the cluster logic.
In order to make debugging easier, the test.MustRunCluster functionality now
names the nodes in the cluster explicitly as "node0", "node1", etc. "node0" is
the coordinator.
A number of TODOs are left in the test for scenarios that need to be checked.
The generate-config command was printing a fixed string rather than calling
NewConfig() which is the canonical source for default config. I also noticed
that we were depending on two different toml libraries, and so collapsed that to
a single one. We have to use pelletier rather than BurntSushi because the viper
library that we use depends on pelletier.
trying to separate internal an external concerns in pilosa.Server - it should
handle Cluster, Holder, etc. while pilosa/server handles things with external
deps - e.g. Logger, Stats, Handler, etc. Using functional options in
pilosa.Server now.
datadog statsd client contained a race condition - was fixed in master
Server.Logger contained a race where multiple loggers could write to the same
output io.Writer
TestMain_FrameRestore contained a race where it tried to change a cluster's
nodes while it was running (which conflicted with antiEntropy reading that
state).
This commit adds support for allocating a gossip transport
and a server listener prior to opening server (and cluster).
Doing that allows tests to use a dynamically allocated port
by supplying bind port: 0.
Create a custom memberlist NetTransport (which will bind to an available
port when port = 0 in the configuration). This allows us to bind
to dynamic ports in tests while at the same time determining a valid
seed for the cluster.
