stays consistent with the array type containers. array and rle are using uint32
in case we want to change the size of roaring containers as an optimization.
This has been done in the past
a8feb985e1
The downside is that more storage is used, so we may want to convert to uint16
until we decide to play with that optimization again, but for now I'll keep the
rle implementation in sync with the array implementation.
in order of the diff:
1. When the start and end of the range fall in the same word, special handling
is needed to "mask" off the beginning and end of the word simultaneously to
avoid counting bits at the beginning or end of the word that aren't in the
range.
2. `i++` is needed at the end of the first partial word to avoid counting this
word in the next block.
3. the shift amount for right shifts is 64 - (end % 64) rather than just end
% 64. If end is (e.g.) 68, then 68 - 64 is 4 and we are only interested in the
first 4 bits of the word, so we must right shift by 60 bits, not 4 bits.
it's no longer reponsible for updating the count cache.
This commit also helps SetBit/ClearBit performance by allowing them
to work against data from `bitmapCache` instead of loading bitmaps
from fragment.storage every time.
- Ingore asserts in fragment container.
- Only log queries that take longer than 90 seconds.
TODO:
- address the TODOs that make the asserts configurable.
Introduces new `pilosactl sort` to sort import files by bit
position so they can be inserted faster. Also optimizes container
scanning and adds a `-buffer-size` flag to `import`.
This commit makes several changes to optimize the TopN() query:
- Reduce highbits() back from 24-bits to 16-bits.
- Reduce MaxArraySize back from 2^20 to 4096.
- Optimize bitmap count invalidation.
- Parallelize TopN() across nodes.
- Parallelize TopN() across slices.
Reworks the `roaring.intersectionCountArrayBitmap()` call to avoid
using an iterator. Performance of the included benchmark went from
2.5ms to 1.1ms.
Some of the issue with intersectionCount is the increased size of
bitmaps and slices and I need to do additional testing with various
sizes.
This commit adds `roaring.Bitmap.CountRange()` to return the number
of bits in a subrange of a bitmap. This significantly improves
server start time and is needed for upcoming zero copy bitmap
optimizations.
This commit moves the cache flush to the `Index` and only serializes
a single fragment at a time.
Also included in this commit is the `inspect` command for the
`pilsoactl` binary. This provides insight into pilosa data files.
This commit changes the `roaring.Bitmap` to use a 40/24 split in
the high and low bits. This change also requires the low bits
to use `uint32` instead of `uint16`.
This commit moves bitmap intersection into the `roaring` package
in order to reduce allocations.
Benchmarks against real world datasets show a speed improvement of
353%. Previously, intersection operated at approximately 5.6M
bits/sec against two 60K bit bitmaps with 30% overlap. The new
intersection operates at approximately 20M bits/sec.
