The GetTx logic is deeply broken, this DOES NOT fix the underlying
bug.
When any call anywhere in a given set of calls has a top-level write,
we perform all transactions as write transactions, and we do not cache or
share those transactions. This means that anything which causes a
second GetTx for the same index/shard deadlocks against itself.
The two easy to find cases by casual inspection are time quantums
and Not queries, so this addresses those, but this should NOT be
considered a general fix.
TxBitmap was a workaround for performance problems with doing
individual-bit operations directly on RBF, used only in the
large-writes path of importValue. With importValue no longer
using that path, ever, there are zero remaining users of TxBitmap,
and the test for it no longer exercises it.
Solution: Remove it.
sort.Stable has horrible runtime -- O(n*logn*logn) -- but if we
don't use sort.Stable, our logic for ensuring that we apply the
"last" value for a given column is actually completely wrong in
the first place.
We've got a fairly consistent thing of the API splitting data up
into shards before sending it to a field, which it has to do because
of clustering, so we don't intend to support the case where you
have data from another shard in a data set.
Also drop the identical but mislabeled test from TestIntField's
corresponding case.
There's only ever one view in importValue, but there's also only ever
one shard, because importValue is only called by things called from
the API after it has split everything up by shard.
Since we don't always have "snapshots" anymore, the arguable benefit of
avoiding the snapshot is reduced, and the primary expense of
importPositions has been dramatically reduced as well, so let's
just use that all the time, and simplify life.
We also want to make it faster. We don't know how many bits there
are to set or clear in the input set, but we do know exactly how
many bits there are to set AND clear. We can subdivide these into
batches by rows, then process each batch by storing sets at the
bottom and clears at the top. We can also do batches by columns,
reducing the memory overhead of unpacking all the bits at once.
(For extra credit, we could alternate set/clear settings, and
thus do batches of "the clears from row 0, followed by the clears
from row 1" and "the sets from row 1, followed by the sets from
row 2", and so on, but this is too fancy.)
Every caller of importValue is in fact already providing values
with column IDs sorted. As such, we don't need a map for checking
the previously-set columns; we just need to check against the
previous value.
We also implement, but disable for now, a check for sortedness of
inputs. This check was useful in development but it's expensive (about
5% of CPU time for large inputs!) and once we've verified that we
can make it through tests without triggering it, we're probably fine.
With timestamps, we probably want to at least check larger BSI fields,
so we add that. Also, tweak the interpretation of b.N (making each
N count for 10,000 bits) so we can see allocation load at all. But we
also reduce the sparse set to be about one bit per 19 bits, because
if we do one per 70,000, and are doing field-at-a-time imports, we're
getting hundreds of imports to try to match a target of, say, around
a million values.
We also sort the inputs, because ImportValue is about to start requiring
that, since the API does it anyway.
Also, extend this to be available on Fields, because field.ImportValue
is ALSO doing things which could be inefficient or expensive.
Attributes are unmaintained and unused.
They have become more of a liability than a benefit.
This change eliminates them from the codebase.
The only user-visible change (assuming that attrs are not used) is that the attrs field will no longer appear in row JSON.
