4d494f6699
This is sort of large, but it's annoyingly difficult to separate out. The basic idea is to allow us to have a single holder-iterating block of code, which is associated with the holder, that can be used for various things, like the snapshot queue background scan, or for inspect operations. We invent the concept of a HolderFilter, which is a thing that can decide what things in a holder it cares about, and a HolderOperator, which can also process those things selectively. In the process, we fix up a couple of subtle bugs in the inspect logic; specifically, the assumption that the mapped flag could tell you whether a container was modified by the ops log doesn't work with mmap, so we have a shiny new flag which is used to track that, internal to the roaring/container code. All of this leads to the actual *point* of this exercise, which is making it easier to create an /inspect endpoint which produces almost the same data we'd have gotten from `pilosa inspect` on a data directory; the distinction is that it doesn't try to identify the distinction between data from disk and data from operations since the file was loaded. Possibly it should, but it doesn't yet. The snapshot queue is now implemented using the HolderOperator design, which requires some subtle changes to how it works, but overall makes it easier to follow the snapshot queue logic, and also shares that logic with the way Inspect works. The holder's snapshot queue is now provided by the server, in a default environment. The queueless snapshot queue no longer triggers snapshots on enqueue -- it turns out that breaks badly, because a key point about enqueueing a snapshot is that it's safe to do it *during* a transaction on that fragment, and triggering a snapshot during a transaction actually causes horrible errors as the ops log ends up being the old file, which we close. Related to this, we also need to prevent closed fragments from trying to snapshot, so we track fragment openness when opening or closing, and bail on trying to snapshot a fragment which is closed. We also stop using the queueless snapshot queue during tests, because that's a horrible idea. We copy a little bit of the partition logic from the cluster code so we don't have to expose it all, this lets us check whether the node we're looking at is the one which should be primary for a given shard, and if not, identify which node would be. This works only when pointed at a data directory, for now. The test cases for the holder have to be internal, because pilosa doesn't export view/fragment, just Index/Field. This means that the holder test cases can't just use the test/* package, so they duplicate some of its logic, approximately. |
||
|---|---|---|
| .. | ||
| testdata | ||
| btree.go | ||
| btree_test.go | ||
| container_stash.go | ||
| containers_btree.go | ||
| containers_slice.go | ||
| containers_test.go | ||
| fuzz_test.go | ||
| fuzzer.go | ||
| generation_debug.go | ||
| generation_nodebug.go | ||
| inst.go | ||
| naive.go | ||
| naive_test.go | ||
| nop_inst.go | ||
| README.md | ||
| roaring.go | ||
| roaring_helpers_test.go | ||
| roaring_internal_test.go | ||
| roaring_nop_paranoia.go | ||
| roaring_nop_sentinel.go | ||
| roaring_nop_stats.go | ||
| roaring_paranoia.go | ||
| roaring_sentinel.go | ||
| roaring_stats.go | ||
| roaring_test.go | ||
| source.go | ||
| unmarshal_binary.go | ||
The Fuzzer
For complete documentation on go-fuzz, please see: https://github.com/dvyukov/go-fuzz
The fuzzer in relation to the roaring package checks the Bitmap.UnmarshalBinary function found in roaring.go. In order to use the fuzzer, you can follow these steps:
cd $GOPATH/src/github.com/pilosa/pilosa/roaring
go-fuzz-build ./
You must now make the workdir/corpus directory. This is achieved by:
mkdir workdir/corpus
The fuzzer needs some input to start the fuzzing with. Copy some sample Pilosa fragments into the workdir/corpus folder. For example:
cp ~/.pilosa/my-index/my-field/views/standard/fragments/0 workdir/corpus
Once you have copied your sample inputs, you are ready to run the fuzzer:
go-fuzz -bin=roaring-fuzz.zip -workdir=workdir -func=FuzzBitmapUnmarshalBinary
Understanding the Fuzzer Output
The fuzzer will output something similar to the follwoing:
2015/04/25 12:39:53 workers: 8, corpus: 124 (12s ago), crashers: 37, restarts: 1/15, execs: 35342 (2941/sec), cover: 403, uptime: 12s
The most important part of the output is the crashers and cover. The crashers records how many combinations were discovered that fail and the cover tells you how much code is being accessed.
For a complete explanation of the output, please see: https://github.com/dvyukov/go-fuzz.
The fuzzer will document the crashers in a folder labeled "crashers." It will record the fragment and the error that was produced in two separate files within this folder. This is the final product.
Happy Fuzzing!