Seebs 0638101d2a cluster state checking cleanups and fixes ... A while back we started just polling the reported cluster state of one node when starting a cluster for tests. This works fine if we're doing fresh new etcd queries for every single operation -- but that's insanely expensive, it turns out. When we use the watcher, some nodes will report stale data for "a while", where "a while" appears to be easily a couple dozen milliseconds. This is probably irrelevant in most real-world cases, because the common case (detecting a node going down) means that we have at least five seconds after a node goes down before etcd notices the lease expiring, and a few milliseconds more or less won't matter. But we have tests that assume either that node 0 is always the coordinator (wrong) or that waiting for node 0 to think the cluster is up means that every node in the cluster thinks the cluster is up, or at least that it means that the coordinator thinks the cluster is up. We retried later operations but not the initial ones against the coordinator. In fact, we probably want to wait for the entire cluster to think it's up before we start trying things on clusters. We also replace the "CheckClusterState" function with the existing AwaitState call, or a new AssertState which errors out since that's the way we usually use AwaitState anyway. In the AwaitPrimaryState function, which used to be AwaitCoordinatorState in a different long-lost revision, we have to delay until a primary node is available, or fail if one does not become available, to avoid a panic. This probably shouldn't happen anymore, because of the last change: Also, rovide dummy topology.Node entries before metadata is read. During initial startup, we want to be able to do things like determine which node is the primary, even before we've read metadata from them. To do this, we populate the node list with dummy entries that just have the ID (the only part we need to sort our list), and a node state of UNKNOWN. This breaks the fancy logic for determining whether or not to update the node data, because the initial status of UNKNOWN matches what we get from SetMetadata giving us new data so we end up not realizing that this was actually a meaningful change. But actually, that's a pretty niche optimization; we usually only get state changes when there's an actual change in state. The updates here are cheap and only happen after a write (or on the first query) so it's not worth making the logic a lot fancier to make it work, when we can just do the simple thing and update any time the dirty flag is set. We also standardize on a 50ms delay, because 1ms delays were really expensive when each check was hitting etcd multiple times, and 50ms is Usually Long Enough.		2021-04-13 12:37:07 -05:00
..
embed.go	cluster state checking cleanups and fixes	2021-04-13 12:37:07 -05:00
leasedkv.go	switch to using a watcher to watch etcd changes	2021-04-13 12:37:07 -05:00
leasedkv_test.go	switch to using a watcher to watch etcd changes	2021-04-13 12:37:07 -05:00