It's not enough to back up each index's translation keys after
backing up that index's data; we also have to back them up after
backing up any index data from indexes which have foreign key
references to that index. So we do the per-index passes separately.
Since the individual backup data files are being created separately,
the expected output is unchanged for a quiescent database, the only
difference is that the amount of translation info which might be
newer than the data stored for shards is potentially increased.
Previously the backup tool only fsync'ed the files.
Since the directories were not synced, it was possible for the references to be lost.
Now we sync the entire output directory tree and its parent.
If we explicitly shut a node down, we don't want everyone else
thinking it's up for the next 5 seconds. Worse, in CI, we have random
long delays (10+ seconds) with no CPU activity at all, so we have to
set the TTL longer there. Which makes any test checking for responsive
detection of a node going down take even longer. So! We revoke
leases on our way down, and this makes the tests not take so
long, and allows us to have a reasonable timeout on the test, while
having a completely unreasonable HeartbeatTTL to make CI stop
breaking randomly.
After continuing to see weird test failures, did some more careful testing,
discovered that CI can pause a machine entirely for up to 29 seconds
or so very rarely, and 5-10 seconds quite frequently, which causes
cascading heartbeat failures and so on. Remove those.
CircleCI is providing a ramdisk as /mnt/ramdisk. Using the ramdisk
instead of local storage makes fsync operations essentially free,
which removes (some of) the frequent multi-second delays we see during
runs otherwise.
It turns out that it's desireable to be able to configure the bootstrap
timeout for etcd, because during startup, we end up delaying that long
(N-1) times in series during each cluster creation, which is pointless
when we're starting the whole cluster. Reduces test runtime by several
minutes.
