Did some benchmarking with b+tree values. The actual interactions
appear to be slightly inconsistent; some values seem to help more
in cases with higher OpN in benchmarks, others with lower OpN. It
appears that the practical consideration may be what happens
when a snapshot gets triggered; smaller kx/kd appear to reduce
costs there, but increase costs between snapshots. This is a
bit of guesswork.
Numbers are slighly under powers of 2, because that means that the
total actual sizes of k and d end up fitting nicely in alloc
pool sizes.
The benchmarks take an absurdly long time to run, and I think these are the
largest offenders. Dropping to two concurrency cases 2 and 16 should give a
pretty good idea.
report the approximate hardware specs (CPU speed, cores, memory)
of the server in the /info endpoint. This may be useful when
benchmarking.
We do some workarounds because gopsutil's core count output is
confusingly different between Linux and Darwin, and the MHz output
is usually wrong on Linux. Intel's app notes say to just parse
the model string. Whyyyyyyy.
set sane defaults. The performance overhead seems to be negligible, and this will allow us to obtain mutex and blocking profiles from running Pilosas by default.
close files after using them if global max is passed.
I originally implemented this without the global count—just always closing files
when done with them, and reopening for new writes. This was crazy slow for that
one test that uses mustSetBits in a big loop. I modified the test to use
importRoaring and everything worked better (though much more slowly).
After adding the global counter, I ran the tests with that one test using
mustSetBits again, and the performance was similar to master. After completing
this PR, I ran the tests with the max limit set to 5—they still passed but were
much slower.
You might wonder why `i <= bitmapN-4`. Answer: The compiler isn't
smart enough for the stride analysis to figure out that `i <= bitmapN`
actually guarantees that. If you set the limit to something not a
multiple of stride, though, it can't figure out *anything* about
things. But for some reason, `i < bitmapN-3` fails badly (it
actually adds bounds checks not present with `i < bitmapN`), but
`i <= bitmapN - 4` works.
This reduces runtime of bitmapRepair by about 14%.
Treating a pointer as a pointer to a large array of bytes,
or converting back the other way, isn't totally insane, but
it does create slices with an extremely large cap. This bit
me while I was trying to build the 16-byte packed Container
structure, but it's probably actually worth fixing in general.
It used to be useful/desireable to set the other slices to nil when
setting a new slice type, it's no longer useful, take some of those
out.
Also reuse the already-computed run count when converting arrays
and bitmaps to runs.
It turns out the logic for "don't update everything if
the incoming slice pointer is the stash" is wrong; it should
really be "don't update everything if the incoming slice
pointer is the one we already have".
The reason this breaks is that one of the tests directly
sets the mapped bit. This breaks my assumption that we'd
never be using the stash and have the mapped bit set, and
that in turn breaks my assumption that the pointer
of an incoming array can't be the stash address unless
we were previously using the stash. If unmap moved us
to non-stashed memory, then a future write could try to
write, notice that it would fit in the stash, copy the
data ... and not update the pointer because the stash
pointer was handled separately.
This way, if you do that, you can end up not using the
stash when you probably could, but you get the expected
behavior. But also, don't set the mapped bit directly.
(I guess there's a good reason to for the test case,
which is using it to verify that unmaps happen when
modifications happen.)
Also the unmap functions should indicate that they have
successfully unmapped, which may help performance in
some test cases.
Noticed in profiling that unmap wasn't being inlined. Also noticed
that every call is on a specific container type, so now they're
specialized and small enough to inline.
This implementation, controlled by the build flag "container24s",
is similar to the single-slice container implementation, but goes
a bit further. First, instead of using a native slice as its internal
storage, it uses pointer/len/cap as distinct values, and only int32
ranges for len and cap. Second, it has a small region of additional
storage which it uses as a backing store by default for arrays or
runs. The idea is that, if you request a new empty array container,
you get one with a pre-allocated virtual slice big enough for five
values, actually stored in the Container. This is useful because
Go's allocator has size classes for 16 and 32 bytes, and the
Container comes in at 24 bytes worth of storage -- meaning that if
you allocate a container, you're allocating 32 bytes anyway, so we
might as well use that space to avoid extra allocations.
This includes some test fixups because DeepEqual was testing
too much equality in some tests.
Also, we simplify unionArrayArray to postpone creating a Container
until we're ready.
On a 64-bit machine, the slices in a Container consume 72
bytes, and the Container itself is 80. But we only use one
slice at a time! This patch shifts us to keeping a single
slice in the Container, and converting provided slices to
and from that type when we want to update it. (It is not
safe to access the slice through the wrong type.)
We also add some new tests, conditional on a build tag
called `roaringparanoia`. These tests will be optimized
away entirely by the compiler when the tag isn't
present, because the conditionals use a const. These catch
possible errors like trying to access the bitmap slice
of a non-bitmap container.
We also eliminate all direct creation of Container literals,
so we can mess with the internals more. (On reflection
and study, we decided not to go to the fancier design where
references to .n and .typ were also converted to function
calls, which would have allowed packing those attributes
more tightly, because it was a lot more overhead and a lot
of work to keep track of.)
There's some circumstances where we appear to have been
relying on incorrect guesses about the nature of containers.
For instance, in xorBitmapRun, there's logic that makes sense
only if the output's a run container, but it's not, it's a
bitmap container. This creates strange behavior sometimes,
though. Several of these are corrected now.
