Prior to using etcd for node membership, the data director was created
during the cluster topology setup. Since that no longer exists, we
weren't actually creating the data directory before getting to
logStartup(). So this change ensure that the data directory exists.
There's no need to have two different translation readers, a single
reader can handle both partitions and fields at the same time, so we
can combine them. This may not actually change things much but was
a useful step in diagnosing a different problem with translate readers,
and I think it is a minor improvement so I'm preserving the patch
just in case.
The functional option and returned closure combine to result in
us using the same sync.Mutex object for every TranslateReader on
a given server, which means that if one of them isn't producing anything,
we eventually end up waiting on that with all the others blocked
waiting for the lock. Use separate locks for each, of the same
type as the one initially provided as a template. This does mean
that multiple readers can be operating at once, but in theory
no two readers should ever be writing to the same stores, we
think.
If we are using replication, we can be a replica translate store for a
partition, which means we start a translate store reader to replicate
data for it. The translation logic does not admit *stopping* the
translate reader, only "resetting" it (stopping and immediately
restarting), so the translate reader just runs until it hits an error
and terminates, which it does even if perhaps it shouldn't. Oops.
Anyway, one potential failure mode is that if you hit timing just
right, you can end up trying to process translation *while* the
index is being closed, and the index can close its translation stores,
and make them all nil, right before we request a store and try to use
it. Another is a similar error, but during the initial startup of the
translate store readers. Either way, we want to error out of the
process cleanly if this happens.
This could also happen during initial creation, perhaps.
We're aborting translation sync on these errors, because otherwise
we'd continue accepting new keys, and then end up with our highest
known key being higher than some keys we missed; this way the next
restart will restart from the last key we have.
this doesn't quite work as-is, but I verified that it reproduced/fixed
the issue by adding a panic where the problem log statement
is. There's a follow up ticket to fix the test... it's just a bit open
ended as to the best way to do that.
we were sending pilosa.Message objects from a spool, but actually
passing a pointer to them rather than the Message itself. I'm
concerned this wasn't caught in any test, and also curious if that
needed to be a pointer for some reason or if it's a typo.
Definitely need to write a test still.
