Have found some potential performance or stability issues associated
with lots of mutex blocking in getting a parent span's context. Want
the ability to totally disable tracing to help debugging.
Annoyingly, this is actually the only place we can make
a read-only lock, because the row() call might write to
the row cache, so it needs the write lock. We might be
able to fix that later, though.
Addressing various lint.
incrementOpN no longer returns errors, because it no longer waits for
the snapshot, so checking those errors is unnecessary.
Several fields in a common embedded structure were "unused" according
to a naive checker.
Other tiny style things, and one actual unchecked error. Yay linters!
If you don't hold the fragment lock when computing rows, it's
pretty reasonable for other stuff to be able to modify it -- which
could invalidate or race the enumeration.
Some calls to f.rows were being made with the lock held, others
weren't, so we introduce `f.unprotectedRows` which has the obvious
semantics. (Without which this looked great except that several
of the tests deadlocked.)
The Pilosa roaring format uses two bytes of its
header, next to the magic number, for a version. The
official roaring format uses them for a container
count, if and only if it's the version of the format
that uses run-length containers.
But if it is, it really does need those bits. Also,
since we never use the official format in our internals
or snapshots, we don't have any reason to support
reading flag bits in it, since the flag bits are used
only for internals of fragments and snapshots. So
we revert the change to support flags with official
roaring bitmaps.
A couple of the fuzz tests happened to rely on this,
and we may find more issues with more fuzzing.
I didn't think of this, because we don't use it much in the
client. This is a bit hairy because really official roaring
is two fairly different formats, one with runs and one without.
As the size of a fragment grows, the cost of snapshots
increases; with a large fragment getting a lot of large writes,
every write will trigger a snapshot, while any other writes have
to wait for that snapshot before they, too, can trigger a snapshot.
To address this, we introduce a background queue of snapshots.
In general, operations which were omitting their ops log writes
and just snapshotting no longer do; they emit an ops log. This does
mean that, in some cases, the ops log is written and then a snapshot
takes place essentially immediately, which costs us some performance.
However, that only actually happens under very light load; under
heavier load, there's generally going to be multiple writes coalesced
into each snapshot, and the ops log writes for them will be much
cheaper than a full snapshot.
When we do a snapshot, we may end up with containers which are
mmapped to the old file, and containers which have allocated storage
identical to the contents of the new file. It would be nicer if they
were mapped to it. But unmarshalling the entire file is expensive.
Instead, we remap it. (Or, if we couldn't mmap it, just make sure
the old stuff is no longer using the old storage space before we
munmap it.)
Instead of fancy bitmap ops or ImportPositions, we use the
recently-added ImportRoaringBits operations, which can dump
themselves to op logs much more efficiently, and which are
also usually much more efficient than things like "create a
new bitmap which is a copy of the old one".
The new op type code changed the failure mode for
one of the fuzz test issues -- and the fuzz test revealed a
bug in the code. Fixed the code, updated the test to expect
the newer, better, message.
Also fixed capitalization on the old message.
We add a new ops log type(pair), AddRoaring and RemoveRoaring,
which set and clear the bits from a provided roaring bitmap.
This also compels us to consider additional sanity checking
during tests.
It turns out there's some significant potential improvements to
be had in the case where there's no cache being used on a field, so
we add it to the benchmarks, to allow testing that.
We also make sure that `getUpdataInto` picks the requested number
of columns; if N was a point at which something weird happens,
we might only sometimes see it.
If you delete a fragment while something else is calling allFragments,
you can cause a race. This almost never happens in practice, because
deleting fragments is rare, and the only likely overlap would be with
something like the holder cache flush, which only happens once a
minute. But if you slowed down the rest of the tests enough, and ran
with -race, you might see it.
We check v.fragments directly instead of calling v.Fragment, because
v.Fragment also needs a lock, and we don't want to drop the lock between
the check for existence and the delete operation.
