There's no reason to have 10-20 seconds of delays for testing this,
because in testing, we're running things on the local machine and don't
need to worry about significant network lag. Make retry count and delay
settable options, and set them lower. Moves the Replica2 test in
server/server_test.go from ~21s to ~2s.
The random-value tests can be pathological, and in particular, the
test of arbitrarily-spaced values is in effect O(N^2), and with race
testing on, that test *alone* can take ten minutes to run, but
it's not really all that exciting. We just reduce a bunch of values
and/or test fewer things for these, which doesn't significantly alter
coverage, but reduces test runtime on my laptop with `-race` from
21 minutes to a bit under 5.
When a mapper hits an error, we want it to immediately tell the
other things in that same mapper that they can stop now. But we
don't want to propagate that all the way back up; if a specific
node has a failure executing a query, we will in some cases want
to send a new query to other backup nodes, so the overall
context isn't cancelled yet.
In general, mapFn and reduceFn have been closures that inherit
a context from the function defining them -- but we don't want
that! We want them to be stopped if their specific mapper gets
cancelled, too, because otherwise they can consume a lot of
resources long after the mapper has stopped being interested
in them. So now those are parameters passed into them,
and mapperLocal puts *those* contexts in the jobs shoved into
the job queue, and the workers pass the context in to the
mapFn/reduceFn.
We also check responses from reduceFn now; both mapReduce
and mapperLocal check for a possible error, and return that,
and reduce functions doing anything nontrivial check their
context.
We also add a few more explicit checks for context cancellation
in various places, especially in the GroupByIterator which is
what bit us that one time. The explicit check against ctx.Err
is officially safe as of Go 1.9 or so. (It was previously
unspecified, but on further study, the Go team concluded that
no actual implementation did anything else, and existing code
was already depending on that.) This also affects the rows
function, because that could potentially take quite a while to
run for a large fragment.
I think when this code was written, I thought "freeze" would be
really cheap. It's not actually that cheap. As a result, freezing
things preemptively when it may be that nothing ever tries to write
to them anyway is possibly disadvantageous, to the tune of being
roughly 20% of a sample profile we were shown. Instead, we don't
mark the components "writable", so if anything wants to write to
them, it'll end up freezing itself new copies of their bitmaps
later. But in practice that probably doesn't happen.
This modifies the parser to properly "unquote" incoming strings. So if
a string comes in double or single quoted, we approximately follow Go
rules for removing the quotes and processing escape sequences.
The differences from Go are:
1. we only support backslash, quote, tab and newline escape
sequenences.
2. Single quoted strings are supported and work just like double
quoted strings.
3. The peg parser won't actually accept backquoted strings (I don't
think)
Fixes: #411
We only have 4 bytes for offsets, but what if a file is
over 4GB? Someone came to us with a file with 265 *million* containers,
in a single fragment, which means that over 3GB of their 4.7GB file
is actually just the container headers alone. But we can't easily make
the offsets larger, or change the file format.
So we don't. We just track how many 4GB hunks of the file we've
been through and bump that every time the 32-bit offset wraps. And this
appears to... just work.
This is fixed for both the roaring iterator and the old unmarshalBinary
logic. The logic to handle this will work on 32-bit hosts in the sense
that it will correctly error out for excessively large file sizes or
container counts, but it doesn't actually handle the large files since
it can't.
In addition to adding some tests, this commit moves the
`GenerateUint64Slice()` helper function into a new `generator` package
so that it can be used in both internal and non-internal tests.
