When doing tests, we create a ton of one-off clusters. This
turns out to be expensive and slow. Fixing it is surprisingly hard.
Fundamentally: If we're sharing clusters, we need to use different
indexes for each test, to avoid clashes. This changes index names.
As a side-effect, this reorders many partition-based things, like
the order keys are returned in. Thus, to fix this, we change a lot
of tests to no longer depend on the *order* in which strings are
returned.
Having done that, we can also discard the ModHasher behavior, since
that only existed to allow us to reliably predict partitioning.
The basic design is as follows: Instead of a cluster being a
[]*Command, a "shareable" cluster is now a []*Command plus some
flags, and a "cluster" is a pointer to a possibly-shared cluster,
plus a link to the specific test using this specific cluster,
and correspondingly, its test name suitably coerced to be a valid
index name prefix.
The "test.Cluster" object now has methods to allow retrieving an
index name, and also implemnts fmt.Formatter to let you use,
e.g., `%i` with it in Sprintf to get "the index name, plus an i".
(This works for everything but %p and %T.)
This allows us to consistently rework all the many things that
use index names in a persistent way.
We also have `MustUnshared` and `MustRunUnsharedCluster` methods
which allow us to specify that a given test needs its own cluster
for some reason. For instance, the tests that want to run backups
need their own isolated cluster, and the tests that want to close
or reopen nodes need their own cluster because a reopened cluster
won't have working GRPC for some reason.
On "closing" a shared cluster (actually the test-specific wrapper
that reflects a given sharing), we delete any indexes starting with
that test's index name prefix. Otherwise, the huge pile of open
indexes prevents `go test -race` from working on MacOS, where we
run out of address space too quickly.
This is fairly enormous but most of the individual changes are
fairly trivial things like replacing the string "i" with "c.Idx()".
We also tweaked a test that failed for me a couple of times to
not depend on sort order.
* removes unused filesize function
* removes ioutil usage
* updates ioutil.ReadAll to io.ReadAll
* updates ioutil.TempFile to os.CreateTemp
* updates ioutil.TempDir to os.MkdirTemp
* updates ioutil.ReadAll to os.ReadAll
* update ioutil.WriteFile to os.WriteFile
* updates ioutil.Discard to io.Discard
* updates ioutil.ReadDir to os.ReadDir where applicable
* removes unused code in idk
* creates type to use for context value keys
* replaces assert.Nil with assert.NoError for error checks
This change should have been adding the "GetClientCertificate"
function in server/tlsconfig.go. This is in addition to the
GetCertificate func which is only used by servers. It ended up being
much more involved for a few reasons:
1. We had no way of passing a configured HTTP client into the
translate store stuff.
2. Our cluster tests assumed http, not HTTPS, and didn't have any way
to pass the necessary configuration in.
3. I encountered what turned out to be an unrelated bug in
cmd/server_test.go which is why I moved "close(m.Started)" in
server/server.go. Basically, I was running something on port 10111
which caused the test to fail (because it was trying to bind to that),
but the failure was not immediately caught during server startup
because the m.Started channel got closed which allowed the test code
to fall through to where it called m.Close() which then got a nil
pointer exception because m.Handler had never been set up.
4. Our test code was assuming that it could create clients that
ignored the config, which meant they didn't do TLS. I added an
InternalClient() method to pilosa.Server to expose the configured
client.
we are experiencing issues with CI where it fails with race: limit on
8128 simultaneously alive goroutines is exceeded, dying
this, despite the fact that closing the executor should clean up all
worker goroutines. Apparently in CircleCI runtime.NumCPU() reports 36,
so the goroutines added up quickly.
the buffers were unused internally and external users had no access to them.
Those wishing to read the logs of the cluster in tests may replace stdout/stderr
with buffers on the Command struct.
The race occurred when a node was stopped and then started again. some
memberlist goroutines might not be completely cleaned up by the time the node
restarted, and then two loggers were using the same output buffer.
cluster is in degraded state when some number of nodes greater than 0 but less
than replicaN have failed. This is sort of a hybrid of "STARTING" and "NORMAL"
states because we can still respond to queries as in the NORMAL state, but we
need to be alert to re-add nodes to the cluster if they come back online which
required some changes to the cluster logic.
In order to make debugging easier, the test.MustRunCluster functionality now
names the nodes in the cluster explicitly as "node0", "node1", etc. "node0" is
the coordinator.
A number of TODOs are left in the test for scenarios that need to be checked.
refactor handler Close func to use errgroup to be a bit less messy.
refactor pilosa.Server closing to actually return an underlying error if one occurs
add option to pilosa/test.Cluster and pilosa/server.Command to control close
timeout. currently is only used by the http handler, but conceivably could be
passed as a parameter to other subsystems of pilosa/server.Command
since the gossip MemberSet has access to Server, it wasn't really necessary to
pass it a Node object when calling Open on it from Cluster. The end goal is to
have it be removed from Cluster entirely, and have it be Opened externally, and
this is a step toward that.
Exposing Node method on Server doesn't really expose any more than was already
there as the same info can be gotten from LocalStatus with a bit of type
casting. I figured adding the method was a little cleaner, and we could collapse
all the functionality when the dust has settled.
The Cluster.open method has been broken into two parts - one of which happens
earlier (at NewServer time), and the other will eventually just be "waiting to
make sure we've joined the cluster". Right now it's calling Memberset.Open, and
then waiting to make sure the cluster has been joined.
pilosa.Server now implements StatusHandler and EventReceiver and needs only
start a gossip memberset. A gossip member set now takes a server as an argument
explicitly and the maze of handlers and receivers and the starting sequence is
somewhat simplified.
Server now trivially implements EventHandler by passing the call along to its
Cluster object which has the actual implementation. This means that less things
will need to refer to cluster.
cluster.open waits for node to join cluster if it is not the coordinator, and
currently this relies on having the http handler able to receive messages, so
handler needs to be started first.
