The view.go change is straightforward and fairly obviously more
correct.
The field.go change avoids holding the field read lock for the
duration of the mutex check request. The thinking was that while the
read lock was held something else was attempting to get a write lock,
which blocked all other read locks and something was getting into a
loop. Seebs might have a more detailed explanation, but that's as far
as my understanding goes at the moment. I believe this change is safe
though as we don't read/modify any field level data structures after
grabbing the standard view.
This takes our reasonably broad selection of predefined container
types and tries intersectionCallback on each pair of them, comparing
results against the results of plain old intersect(). We've had
several intersectionCallback fixes recently; every one of them
produces test failures here if reverted or broken, so I have at
least some confidence in this coverage.
Similarly, test everything on containerCallback, verifying that
we get the same set of values called back that we get from Slice().
Both of these were verified with -coverprofile to actually be
hitting all the lines of code that aren't insane edge case
checks like "what if a run is in the wrong order".
The inner loop of intersectionCallbackArrayArray's "fast"
case has
for len(ca) > 0 && ca[0] < va {
}
so we do not leave that loop unless len(ca) is 0, or
ca[0] >= va.
We then return from the whole function if len(ca) is 0,
so the only way we finish one iteration of the outer for
loop is if ca[0] >= va. Thus, this can be an `if` rather
than a `for`.
We also fix the logic for ArrayRun to make it require fewer
tests and be clearer about why the tests work and clearer about
always making progress.
And, finally, the bitmap/range callback logic, and the underlying
"callback per bit in word" logic, were both badly broken. In
particular, if a range started and ended in the same word, it would
hit the values in that word twice, once with them incorrectly
shifted, but then it would further garble any offsets past the first
in a word. Eww.
The merge lists behavior was flawed in that it would drop one item
from the list per merge, which means that, with high replication
and low number of distinct items, it could even produce an empty
list.
The actual "is there anything wrong" logic is fine, but the list of
clashing values set for a given record is not.
Unfortunately this also doubles the time the test takes, to
21 seconds on MacOS. OW.
It's hard to do this remotely sanely for the fragments, but the
translation and collation process itself could be fairly slow on
large data sets, so we should check occasionally for canceled
context and return early if no one needs the result anyway.
Also, take out no-longer-correct comments from the test case.
We support query parameters for details (default false) which
request additional data, and for a limit (default 0/MaxInt32)
on number of results returned to limit the amount of spam
produced if there's a lot of results. The simpler default
output should reduce load and runtime significantly, and the
ability to specify limits makes it easier to get reasonably
small responses.
There's some context support here, but the underlying filters
don't take contexts or check for them, which is probably
a flaw but might be a bit large to correct for this.
Despite being large, this set of changes is actually
fairly well contained within the mutex-checking code.
The rbf_bolt tests are unusually expensive, partially because they're
run with the race detector on, but also because it's basically running
two copies of all the tests and comparing them... But they haven't
detected anything in ages, because the RBF stuff is now pretty stable,
and those tests take about twice as long as anything else in our testing,
and thus impede our workflow noticably for little-to-no return. We might
some day want to fully remove them, but for now, just taking them out of
CI should streamline our workflows a bit.
Mutex fixes -- these address a couple of cases in which mutexes could end up with duplicate values, and also improve the testing so they're more likely to get caught.
This implements a fairly straightforward sanity-check for mutexes,
implemented as a bitmapfilter at the fragment level, and with higher
levels combining results. There's two endpoints, an internal endpoint
which only checks the local node's shards, and an external one which
forwards requests (using the internal endpoint) to all the other nodes.
The internal endpoint does not do key translation, the external one
does.
The transmission format is a probably-inefficient JSON blob, and
returns data separated per-shard so we don't have as much merging
work to do.
This introduces a horrifying monstrosity function which tries to
sneakily corrupt mutex fields and which has to be exported (EWWWWW)
but which is only present in _test code (!??!! THIS WORKS WHY).
Also one typo fix in unrelated code caused by not wanting to keep
fighting with gofmt about this.
Two of the intersectionCallback functions were broken.
In intersectionCallbackArrayArray, when checking to see whether we can
skip ahead 8, we need to check whether that last value is lower than
the one we're looking for, not whether the first value is.
For intersectionCallbackArrayBitmap, actually implement it at all;
it had never gotten modified significantly from the original
intersectionCount, so it still counted and returned intersections, but
never called the callback at all.
When doing the import tests, import all the data sets if there's
multiple data sets, and check that we're producing the correct number of
results including overwriting previous values, not just that we produce
the same number of values that we set, which shouldn't happen if there's
any overlap.
Also add a specific test that triggers the case I first ran into this for.
This is a design to let us write test cases for ingest with schema setup
and data in the json formats we want to use, and results as alternating
queries and expected results, so we can just create new test files and
run the tests against them. We also have to report back what we created
when creating things.
In the process of developing this, I noticed that the documentation describes
ingest schema as allowing more than one schema operation, but we didn't support
this, and also it wouldn't do much good because there was no way to do partial
things like "just add a field". Fixed.
Also we implement comparison for ops, so the test output is actually
a test rather than just some data to visually eyeball.
In the process, realize that the handling of timestamps was wrong; we said that we
take them as raw numbers relative to the epoch, not as raw Unix timestamps.
Also a couple of related cleanups caught by doing the testing.
This is a rework of Nia's radix sort. Still using stdlib sort for the
tail ends of things, and should probably replace it at some point
because it's still woefully inefficient, but this gets decent
performance, and lets us do the fancy thing of doing quick partial
sorting by record-key-only to get to shards, then deciding whether
to sort by value-then-record (as for a set field) or just by record
(as for int fields), which lets us reduce the amount of re-sorting
the same data by different criteria we do.
We also use a messy code-duplication basically-bubblesort for the
inner loops because it's much cheaper for small N.
This also lets us use field-aware sorting for shards, sorting them
correctly for a corresponding field type, and add corresponding API
support and fragment support for an option to tell the fragment
code that we already ordered things in the order that's most
efficient there, to avoid a second sort that we don't otherwise
need.
This partially-implemented prototype of the ingest API is based on our
programmatic ingest API reference. It has noticable limitations, most
crucially that it doesn't handle multi-node clusters right now. However,
it basically implements the expected semantics.
There's some noticeable performance issues to do with the high overhead
of sorting bits in order to import them efficiently, but this is fixable.
We also add the hooks to the internal client, and make the finisher logic
a bit smarter.
Much of this code was originally by Nia Weiss, but it's been merged
and restructured a bit to get things broken into logical commits.
