in order of the diff:
1. When the start and end of the range fall in the same word, special handling
is needed to "mask" off the beginning and end of the word simultaneously to
avoid counting bits at the beginning or end of the word that aren't in the
range.
2. `i++` is needed at the end of the first partial word to avoid counting this
word in the next block.
3. the shift amount for right shifts is 64 - (end % 64) rather than just end
% 64. If end is (e.g.) 68, then 68 - 64 is 4 and we are only interested in the
first 4 bits of the word, so we must right shift by 60 bits, not 4 bits.
When the random number generator was coming up 0,
then `a = []uint64{}`. This caused the bitmap `bm` to be empty.
In this case, `bm.Slice()` is a nil slice, while the expected
slice is an empty slice (i.e. they are not considered equal):
got: ([]uint64) <nil>
exp: ([]uint64) {}
it's no longer reponsible for updating the count cache.
This commit also helps SetBit/ClearBit performance by allowing them
to work against data from `bitmapCache` instead of loading bitmaps
from fragment.storage every time.
- Ingore asserts in fragment container.
- Only log queries that take longer than 90 seconds.
TODO:
- address the TODOs that make the asserts configurable.
Introduces new `pilosactl sort` to sort import files by bit
position so they can be inserted faster. Also optimizes container
scanning and adds a `-buffer-size` flag to `import`.
This commit makes several changes to optimize the TopN() query:
- Reduce highbits() back from 24-bits to 16-bits.
- Reduce MaxArraySize back from 2^20 to 4096.
- Optimize bitmap count invalidation.
- Parallelize TopN() across nodes.
- Parallelize TopN() across slices.
Reworks the `roaring.intersectionCountArrayBitmap()` call to avoid
using an iterator. Performance of the included benchmark went from
2.5ms to 1.1ms.
Some of the issue with intersectionCount is the increased size of
bitmaps and slices and I need to do additional testing with various
sizes.
This commit adds `roaring.Bitmap.CountRange()` to return the number
of bits in a subrange of a bitmap. This significantly improves
server start time and is needed for upcoming zero copy bitmap
optimizations.
This commit moves the cache flush to the `Index` and only serializes
a single fragment at a time.
Also included in this commit is the `inspect` command for the
`pilsoactl` binary. This provides insight into pilosa data files.
This commit changes the `roaring.Bitmap` to use a 40/24 split in
the high and low bits. This change also requires the low bits
to use `uint32` instead of `uint16`.
This commit moves bitmap intersection into the `roaring` package
in order to reduce allocations.
Benchmarks against real world datasets show a speed improvement of
353%. Previously, intersection operated at approximately 5.6M
bits/sec against two 60K bit bitmaps with 30% overlap. The new
intersection operates at approximately 20M bits/sec.
This commit moves the computation of the intersection count to
the `roaring` package so that no allocations are required. The
implementation operates at the roaring container level and has
specialized functions for array-array, array-bitmap, and
bitmap-bitmap container pairs.
This commit changes the underlying storage for `Bitmap` from a
red-black tree to a roaring bitmap. It also removes bitmaps from
the cache and only stores the bitmap count.
This commit refactors the anti-entropy system to fetch data from
all replicated blocks and only set/clear bits which deviate from
the consensus between all blocks.
An example of this is if 3 nodes had the following bits set for
a single bitmap:
Node A: 1 2 3
Node B: 2 4
Node C: 1 2 4
Then only bits which are set on a majority will be set. In this
case bits 1, 2, & 4 are set but 3 only exists on a single node.
The node performing the merge would then determine the following
set/clear diffs for each node:
Node A: clear(3), set(4)
Node B: set(1)
Node C: none
Once the merge is performed and all nodes receive their diff
instructions then the nodes will be in sync:
Node A: 1 2 4
Node B: 1 2 4
Node C: 1 2 4
There still exists situations where bits can be reset. If Node A
is up and Node B & C are down then Node A's bits will be reset
once B & C come back online. We should add write consistency
settings for incoming writes so that we can ensure that a quorum
is written to before returning a success. This is outside the
scope of this commit though.
This commit changes the `pql.Query` so that it can accept one or more
top-level calls instead of only one.
The query request format change because a query with a single call is
still valid. However, the result format now returns a `results` field
that has one result for each top-level call. The `profiles` field is
still the same, however, it combines all profiles from all bitmap
responses into one return so that there's not duplicate attributes.
Fixes#59
This commit refactors the roaring bitmaps to use uint64 values
instead of uint32 values. This is required for the size of
values we need in pilosa.
This change is not backwards compatible with the previous data
format so any existing data directories need to be removed before
using this code.
