We add endpoints and protobuf encode/decode to allow for sending
sharded requests over the wire in protobuf, so we can take our
sharded data and send it to other nodes if needed.
This is a squash of >15 other commits, so a bit of history
is relevant:
The Request type had FieldTypes in it because the field type
information was needed for sharding because sorting requires
that information. We change this around to make the external
sharding operation require the field types, and curry that
through the codec -- the codec is needed to tell the request
how it shards. (This is because the correct sorting order
varies by field type.) Requests (and ShardedRequests) no
longer have that table in them.
And then we hit a nasty bug in production and RCA showed
that our testing wasn't good enough and we need to be more
careful, and I discovered that test coverage in this package
was around 70%.
So, the other big thing here is coverage testing; in order to
make coverage testing viable and programmatically testable,
we have added the ability to render requests *back* to
JSON. This is not a great idea, but it does allow us to do
a lot of sanity-checking and verify that the encodings we're
using are consistent and correct.
This, plus some specific tests of decoding specific flawed
inputs, has caught a number of issues. Which are now fixed!
A lot of internal API surface got slightly changed, in ways
that make it simpler to work with. For instance, the
(*FieldOperation).TranslateUnsigned function doesn't really
need to exist; we can just have a non-method translate
function for unsigned and for signed, and use them based on
field type.
The stable translation hack used for testing had a bug that
could allow it to end up producing incorrect results if you
asked it to translate an ID first rather than exclusively
asking it to translate strings first, this has been
corrected. (This is a bug fix in code that was added
partway through creating this, but is tricky enough to
mention its own comment.)
Test coverage is now just over 90%, and a lot of what's left
is error-check returns that may well be actually unreachable
unless, say, the documentation for encoding/json is full of
lies. Which it probably is.
The view.go change is straightforward and fairly obviously more
correct.
The field.go change avoids holding the field read lock for the
duration of the mutex check request. The thinking was that while the
read lock was held something else was attempting to get a write lock,
which blocked all other read locks and something was getting into a
loop. Seebs might have a more detailed explanation, but that's as far
as my understanding goes at the moment. I believe this change is safe
though as we don't read/modify any field level data structures after
grabbing the standard view.
This takes our reasonably broad selection of predefined container
types and tries intersectionCallback on each pair of them, comparing
results against the results of plain old intersect(). We've had
several intersectionCallback fixes recently; every one of them
produces test failures here if reverted or broken, so I have at
least some confidence in this coverage.
Similarly, test everything on containerCallback, verifying that
we get the same set of values called back that we get from Slice().
Both of these were verified with -coverprofile to actually be
hitting all the lines of code that aren't insane edge case
checks like "what if a run is in the wrong order".
The inner loop of intersectionCallbackArrayArray's "fast"
case has
for len(ca) > 0 && ca[0] < va {
}
so we do not leave that loop unless len(ca) is 0, or
ca[0] >= va.
We then return from the whole function if len(ca) is 0,
so the only way we finish one iteration of the outer for
loop is if ca[0] >= va. Thus, this can be an `if` rather
than a `for`.
We also fix the logic for ArrayRun to make it require fewer
tests and be clearer about why the tests work and clearer about
always making progress.
And, finally, the bitmap/range callback logic, and the underlying
"callback per bit in word" logic, were both badly broken. In
particular, if a range started and ended in the same word, it would
hit the values in that word twice, once with them incorrectly
shifted, but then it would further garble any offsets past the first
in a word. Eww.
