most of this is stolen from the 3.x branch
for building the release artifacts I removed the restriction of only
master branch, so we should be able to see if it works before the PR
is merged.
When writing things that cause additions to the cache, mark it dirty and
flag it for recomputing, but only sometimes actually do the recalculation,
currently implying a 10-second window. We still mark the cache dirty,
so if a request comes in, we'll get fresh data, but the query will be
slowed down because the recomputation will happen then. But that's better
than doing thousands of recalculations which are never used...
When searching for a small array in a large array, scanning ahead
is productive. The switch from counting indexes to reslicing the
slice appears to improve performance in this case. The fairly arbitrary
value `na << 2` is like `nb / 4 > na` except that it computes faster,
and lets us avoid the expensive overhead unless we have reason to
expect that there's significantly more items in b than in a.
Improvements: Not huge in some cases, but sometimes quite noticeable,
especially as the frequency of overlap increases, which is also
the expensive case in other ways.
name old time/op new time/op delta
ImportMutexSampleData/64K/2Kr/40/none/write-0-8 501ms ± 4% 486ms ± 2% ~ (p=0.052 n=6+5)
ImportMutexSampleData/64K/2Kr/40/none/write-1-8 756ms ± 5% 698ms ± 5% -7.62% (p=0.002 n=6+6)
ImportMutexSampleData/64K/2Kr/80/none/write-0-8 292ms ± 3% 276ms ± 4% -5.46% (p=0.002 n=6+6)
ImportMutexSampleData/64K/2Kr/80/none/write-1-8 511ms ± 6% 482ms ± 4% -5.72% (p=0.015 n=6+6)
ImportMutexSampleData/64K/2Kr/240/none/write-0-8 153ms ± 3% 132ms ± 5% -13.91% (p=0.008 n=5+5)
ImportMutexSampleData/64K/2Kr/240/none/write-1-8 354ms ± 2% 215ms ± 6% -39.41% (p=0.004 n=5+6)
ImportMutexSampleData/1K/2Kr/40/none/write-0-8 565ms ± 3% 543ms ± 3% -3.89% (p=0.015 n=6+6)
ImportMutexSampleData/1K/2Kr/40/none/write-1-8 807ms ± 6% 778ms ± 3% ~ (p=0.180 n=6+6)
ImportMutexSampleData/1K/2Kr/80/none/write-0-8 317ms ± 3% 300ms ± 1% -5.40% (p=0.002 n=6+6)
ImportMutexSampleData/1K/2Kr/80/none/write-1-8 462ms ± 3% 437ms ± 4% -5.31% (p=0.009 n=6+6)
ImportMutexSampleData/1K/2Kr/240/none/write-0-8 141ms ± 1% 119ms ± 2% -15.85% (p=0.004 n=5+6)
ImportMutexSampleData/1K/2Kr/240/none/write-1-8 213ms ± 3% 171ms ± 3% -19.70% (p=0.002 n=6+6)
In BitmapBitmapFilter.ConsiderData, we intersect things solely in order
to perform callbacks on them. Creating these intermediate arrays is
actually somewhat expensive, and all we're going to do with them is
make callbacks anyway.
So, we add a new `intersectCallback`, which behaves similarly to
`intersectionCount`, but which dramatically reduces the amount of memory
allocation associated with doing the callbacks; in some test cases
on mutex data, this code was >90% of all memory allocations, and
getting rid of that helps a lot.
At that point, we no longer need the separate intersectAny check,
because it doesn't save us any time anymore.
The mutex tests had weird and un-idiomatic definitions for b.N, and
in particular would report ludicrously low times for high values of
b.N because they'd still only do a small amount of importing, then
get counted as having done a much larger number of iterations. Also,
the computation of the number of values to create was pretty noticably
wrong so the secondary data set was unduly tiny.
Do tests with ranked cache and larger row counts because we have
reason to suspect that the cache behavior is mattering. We adjust the
range of tests performed to reflect real world data a bit. We also
drop the "don't do large mutex tests" thing because the insanely
bad performance on larger mutex data should be fixed now, we hope.
The "batched" flag creates a complexity which is that the return value of Add
might or might not be meaningful, but it doesn't really buy us very much.
If we are concerned about the ops log size of writing single ops as 21-byte
arrays of 1 op rather than as 13-byte ops, we can make the AddN code smarter
about how it writes ops. And probably should.
Along with this, change Remove to use the batched operation form, which
writes a more meaningful ops log, and return a meaningful value for changes
made. Otherwise, it ends up writing potentially thousands of ops to the
ops log without reporting any OpN, because the number of ops written isn't
the same as the number of changes those ops made. This could result in
files growing by megabytes without OpN changing.
There was a comment here about a test failing with RemoveN. I can't prove
it, but I strongly suspect that this was actually a result of that test
case hitting a particular bug that we eventually fixed, and which we might
have fixed sooner if we'd realized why using RemoveN made that test
fail.
When unmarshalling ops, we weren't adding a meaningful OpN to them,
resulting in misleading reports from `pilosa inspect`. Also, we were
mistakenly reporting things as "mapped" when they were actually
using their internal storage (as with small array containers).
Add the "sanity check" to `pilosa inspect` so that errors like the
above get noticed more easily and corrected. Also, to make that work,
have roaring.InspectBinary actually put containers in the bitmap
it creates rather than just creating info entries for them.
bitmap.BitwiseEqual had a couple of subtle bugs, and the net result
is that if the bitmap you were comparing to had an empty container after
the original bitmap ran out of containers, we'd spuriously report
the container as existing and being... the last container in the original,
actually.
Issues are both that we were grabbing the value from the wrong iterator,
and also that we were iterating twice per loop, and thus could also
have missed a non-empty container immediately following an empty one.
When *fragment.openStorage is invoked in both f.importValue and
f.importValueSmallWrite and it returns an error, this means there's
some underlying error with the storage device and at the point of this
commit, the sane thing to do is to close the process, otherwise the
operation of Pilosa might proceed in an inconsistent state thus
precipiatting other silent but hairy errors along the way such as
dereferencing *fragment.gen later on which is set to nil once
openStorage fails.
