Turning off IPv6 causes the tests to fail because the Go
`net.Listen()` function obtains a `[::]` address when using
a `localhost:0` bind address. Looking through the Go source
code, I don't see a good way to fix that.
This change adds a test flag to allow users to specify the
network name when running the test:
```sh
go test github.com/pilosa/pilosa/server -network tcp4
```
in order of the diff:
1. When the start and end of the range fall in the same word, special handling
is needed to "mask" off the beginning and end of the word simultaneously to
avoid counting bits at the beginning or end of the word that aren't in the
range.
2. `i++` is needed at the end of the first partial word to avoid counting this
word in the next block.
3. the shift amount for right shifts is 64 - (end % 64) rather than just end
% 64. If end is (e.g.) 68, then 68 - 64 is 4 and we are only interested in the
first 4 bits of the word, so we must right shift by 60 bits, not 4 bits.
When the random number generator was coming up 0,
then `a = []uint64{}`. This caused the bitmap `bm` to be empty.
In this case, `bm.Slice()` is a nil slice, while the expected
slice is an empty slice (i.e. they are not considered equal):
got: ([]uint64) <nil>
exp: ([]uint64) {}
it's no longer reponsible for updating the count cache.
This commit also helps SetBit/ClearBit performance by allowing them
to work against data from `bitmapCache` instead of loading bitmaps
from fragment.storage every time.
- Ingore asserts in fragment container.
- Only log queries that take longer than 90 seconds.
TODO:
- address the TODOs that make the asserts configurable.
Introduces new `pilosactl sort` to sort import files by bit
position so they can be inserted faster. Also optimizes container
scanning and adds a `-buffer-size` flag to `import`.
This commit makes several changes to optimize the TopN() query:
- Reduce highbits() back from 24-bits to 16-bits.
- Reduce MaxArraySize back from 2^20 to 4096.
- Optimize bitmap count invalidation.
- Parallelize TopN() across nodes.
- Parallelize TopN() across slices.
Reworks the `roaring.intersectionCountArrayBitmap()` call to avoid
using an iterator. Performance of the included benchmark went from
2.5ms to 1.1ms.
Some of the issue with intersectionCount is the increased size of
bitmaps and slices and I need to do additional testing with various
sizes.
This commit adds `roaring.Bitmap.CountRange()` to return the number
of bits in a subrange of a bitmap. This significantly improves
server start time and is needed for upcoming zero copy bitmap
optimizations.
This commit moves the cache flush to the `Index` and only serializes
a single fragment at a time.
Also included in this commit is the `inspect` command for the
`pilsoactl` binary. This provides insight into pilosa data files.
This commit changes the `roaring.Bitmap` to use a 40/24 split in
the high and low bits. This change also requires the low bits
to use `uint32` instead of `uint16`.
