Ensure that mapReduce always waits on its ErrGroup, even if it wants to return
early due to a failure somewhere. Also check logic a bit more carefully on
the error returns; we don't want a transient failure from one node to result
in the whole query failing, we just want it to retry on the next node, so that
shouldn't cancel the whole ErrGroup.
if we got an error, we don't have to merge it. so either ctx.Err or
resp.err being non-nil means we shouldn't be reducing, but we still need
to grab the responses to make sure we waited for them all.
This fixes a variety of bugs where API requests would read uninitialized state, causing crashes or race conditions.
Co-authored-by: Antonio Navarro Perez <antnavper@gmail.com>
It's not enough to cancel jobs so their goroutines *will* exit; we have
to be certain that they *have exited* before we finish returning from,
e.g., mapReduce(), or a query can "complete" at a time when there are
still running goroutines accessing data that we're about to invalidate
when we terminate the Qcx.
A better solution would integrate this logic and control into the Qcx
and pass it through everything, rather than having the Qcx bypass
the mapper/mapperLocal and be passed into the mapFn/reduceFn via
closures. But a better solution would be a lot larger.
This gives more consistency with the other tests and allows us to get audit
checks on the server/ tests. The tests on the clients being closed are
temporarily disabled because they tend to think the last test's clients
are "still open" for a few seconds after the test completes.
etcd runs a LOT more goroutines during server startup. Fix a
goroutine/for loop bug causing us to run four 7-node clusters
instead of 1/3/4/7-node clusters, also have the test/cluster
code reduce import workers. We can't do much about the spamminess
of the Raft stuff, but this should tone it down some.
Every usage of this just ran keepAlive func as a goroutine with a timer, using
a parent context, but the keepAlive func didn't know about that context, so
it couldn't use that context for its own messages or interactions. Change
it to create its own cancelable context from a provided parent, and use
that to control its inner behavior.
Note that we *do* still need to send the revoke at least sometimes -- otherwise
cluster states don't update correctly. But we can time that send out
rather than using context.Background(), because after a TTL's worth of time,
there's no lease to revoke anyway.
Also, add hooks for testhook tracking so we can confirm/deny that things
are getting shut down, which they weren't.
