If we explicitly shut a node down, we don't want everyone else
thinking it's up for the next 5 seconds. Worse, in CI, we have random
long delays (10+ seconds) with no CPU activity at all, so we have to
set the TTL longer there. Which makes any test checking for responsive
detection of a node going down take even longer. So! We revoke
leases on our way down, and this makes the tests not take so
long, and allows us to have a reasonable timeout on the test, while
having a completely unreasonable HeartbeatTTL to make CI stop
breaking randomly.
This is a significant overhaul! Quite a lot of things changed here.
Basically: Prior to this, every request for data from etcd implies
requesting the current live data from etcd, and then unpacking it or
extracting it in some way. This is expensive, which is why we have
a cache in front of it.
We don't need to do that! We can use a Watch, which notifies us
of changes as changes happen. However, there's some challenges and
difficulties along the way, and there's a couple of other changes
which are included here because it's a pain to try to separate them
out.
1. We require a logger to be provided to create our internal Etcd
wrapper. We then use that logger, instead of `fmt.Printf`. This makes
debugging messages work better, and also diagnostics, and so on.
2. The internal client that we are reusing can enter a failed state
after a leader election, in which case we have to recreate the client
to have a working client. We add a new internal-use method,
`retryClient`, which wraps a function which takes an etcd client
and returns an error, and checks for leader-election type errors
and retries creating the client when they happen. That last bit
has not been successfully tested because it's actually really hard
to trigger this now. (Because it was related in part to the
amount of etcd traffic we were producing, which is reduced.)
3. The general swap over from looking things up to unpacking things
as they come in, then returning those already-unpacked things when
we get requests.
With this change, *many tests will fail*. That is addressed by
a separate commit which addresses the secondary problem, which is
that some of our test harness code was relying on the assumption
that if any node in a cluster thinks the cluster is up, every node
will. That was usually true when we were doing everything as
expensive fully-synchronized cluster checks, but becomes significantly
less reliably true in real-world cases where nodes are also
going down sometimes, or nodes are going up and down unexpectedly.
There's some loose ends here because really we probably want to be
using the top-level server logger, and we should fix that, but in the
mean time, let's not swallow the errors as much, because the last
line printed doesn't actually show what the error was, but it could.
To do this, we distinguish between the current error (which might
be a wrapper around DeadlineExceeded) and a previous error which
we might prefer to return, if one exists, since it's more likely
the "real" cause.
