if we got an error, we don't have to merge it. so either ctx.Err or
resp.err being non-nil means we shouldn't be reducing, but we still need
to grab the responses to make sure we waited for them all.
This fixes a variety of bugs where API requests would read uninitialized state, causing crashes or race conditions.
Co-authored-by: Antonio Navarro Perez <antnavper@gmail.com>
It's not enough to cancel jobs so their goroutines *will* exit; we have
to be certain that they *have exited* before we finish returning from,
e.g., mapReduce(), or a query can "complete" at a time when there are
still running goroutines accessing data that we're about to invalidate
when we terminate the Qcx.
A better solution would integrate this logic and control into the Qcx
and pass it through everything, rather than having the Qcx bypass
the mapper/mapperLocal and be passed into the mapFn/reduceFn via
closures. But a better solution would be a lot larger.
This gives more consistency with the other tests and allows us to get audit
checks on the server/ tests. The tests on the clients being closed are
temporarily disabled because they tend to think the last test's clients
are "still open" for a few seconds after the test completes.
etcd runs a LOT more goroutines during server startup. Fix a
goroutine/for loop bug causing us to run four 7-node clusters
instead of 1/3/4/7-node clusters, also have the test/cluster
code reduce import workers. We can't do much about the spamminess
of the Raft stuff, but this should tone it down some.
Every usage of this just ran keepAlive func as a goroutine with a timer, using
a parent context, but the keepAlive func didn't know about that context, so
it couldn't use that context for its own messages or interactions. Change
it to create its own cancelable context from a provided parent, and use
that to control its inner behavior.
Note that we *do* still need to send the revoke at least sometimes -- otherwise
cluster states don't update correctly. But we can time that send out
rather than using context.Background(), because after a TTL's worth of time,
there's no lease to revoke anyway.
Also, add hooks for testhook tracking so we can confirm/deny that things
are getting shut down, which they weren't.
Long story short: Once we create a server and start it, we can't start
it again. We can't close it and restart it, and we can't just start
it without closing it.
Unfortunately, if the server's config needs to change, we have a Problem
here.
This ultimately means that the retry logic for GetListeners can't actually
retry successfully; if we fail on the first attempt, we necessarily fail
on any later attempts also, and if we try to fix that, we get panics.
But!
We don't actually NEED to retry. We just need to ensure that we can
open a :0 port, extract the actual port number, and use that in places
where the port number mattered, without having to rebind it.
The only actual place we needed to rebind things was opening gRPC
servers, so we introduce a gRPC Listener that can be used instead of
trying to bind to a specified port.
In a bunch of other cases where we had similar logic to try to allocate
and then use a port, we can switch to just using a provided listener.
For instance, net/http has `Serve(net.Listener, handler)`, not just
ListenAndServe(addr, handler).
This should eliminate the weird CI failures from eaddrinuse.
NOT fixed: server/cluster_test.go/TestClusterResize_AddNode isn't working
right now. The new node isn't actually being added to the existing cluster.
I attempted this but was outsmarted by it, and I think fixing the
rest of this is worth it as a separate thing.
