28b9d6d7fc
UpdateEvery can change every key, and I think it strongly suggests no reasonable expectation of repeated access to a previously-accessed key, but also it can change the containers and replace them. We were avoiding caching mapped containers in some but not all cases, and that was causing segfaults. But really, the *problem* is that the remap operation wasn't clearing (or updating) the cache. Cleaning that up allows us to take advantage of the caching performance advantage even when working with read-only/mapped bitmaps. The only way to hit this: * Have mmapped containers to begin with. * Do reads so those containers get frozen. * Access, either reading or writing, a specific container with key K. * Snapshot, so the bitmap gets its containers replaced. * Remember, they have to be frozen -- if they aren't frozen, we'll update the containers in place. * Now have GC run so it actually unmaps the data. * Now try to write to the container with key K *before reading or writing any other key*. You have to get through the whole snapshot and GC process without any other reads or writes. * You get the cached value. You try to use it. You explode. The sliceContainers code was also setting lastKey to 0 in some cases, but also setting lastContainer to nil, so this wouldn't have caused problems, but just to be careful, I've standardized on ^uint64(0) for everything. |
||
|---|---|---|
| .. | ||
| testdata | ||
| btree.go | ||
| btree_test.go | ||
| container_stash.go | ||
| containers_btree.go | ||
| containers_slice.go | ||
| containers_test.go | ||
| fuzz_test.go | ||
| fuzzer.go | ||
| generation_debug.go | ||
| generation_nodebug.go | ||
| inst.go | ||
| naive.go | ||
| naive_test.go | ||
| nop_inst.go | ||
| README.md | ||
| roaring.go | ||
| roaring_helpers_test.go | ||
| roaring_internal_test.go | ||
| roaring_nop_paranoia.go | ||
| roaring_nop_sentinel.go | ||
| roaring_nop_stats.go | ||
| roaring_paranoia.go | ||
| roaring_sentinel.go | ||
| roaring_stats.go | ||
| roaring_test.go | ||
| source.go | ||
| unmarshal_binary.go | ||
The Fuzzer
For complete documentation on go-fuzz, please see: https://github.com/dvyukov/go-fuzz
The fuzzer in relation to the roaring package checks the Bitmap.UnmarshalBinary function found in roaring.go. In order to use the fuzzer, you can follow these steps:
cd $GOPATH/src/github.com/pilosa/pilosa/roaring
go-fuzz-build ./
You must now make the workdir/corpus directory. This is achieved by:
mkdir workdir/corpus
The fuzzer needs some input to start the fuzzing with. Copy some sample Pilosa fragments into the workdir/corpus folder. For example:
cp ~/.pilosa/my-index/my-field/views/standard/fragments/0 workdir/corpus
Once you have copied your sample inputs, you are ready to run the fuzzer:
go-fuzz -bin=roaring-fuzz.zip -workdir=workdir -func=FuzzBitmapUnmarshalBinary
Understanding the Fuzzer Output
The fuzzer will output something similar to the follwoing:
2015/04/25 12:39:53 workers: 8, corpus: 124 (12s ago), crashers: 37, restarts: 1/15, execs: 35342 (2941/sec), cover: 403, uptime: 12s
The most important part of the output is the crashers and cover. The crashers records how many combinations were discovered that fail and the cover tells you how much code is being accessed.
For a complete explanation of the output, please see: https://github.com/dvyukov/go-fuzz.
The fuzzer will document the crashers in a folder labeled "crashers." It will record the fragment and the error that was produced in two separate files within this folder. This is the final product.
Happy Fuzzing!