mirror of
https://github.com/featurebasedb/featurebase.git
synced 2026-08-28 10:54:59 +00:00
4d494f6699
This is sort of large, but it's annoyingly difficult to separate out. The basic idea is to allow us to have a single holder-iterating block of code, which is associated with the holder, that can be used for various things, like the snapshot queue background scan, or for inspect operations. We invent the concept of a HolderFilter, which is a thing that can decide what things in a holder it cares about, and a HolderOperator, which can also process those things selectively. In the process, we fix up a couple of subtle bugs in the inspect logic; specifically, the assumption that the mapped flag could tell you whether a container was modified by the ops log doesn't work with mmap, so we have a shiny new flag which is used to track that, internal to the roaring/container code. All of this leads to the actual *point* of this exercise, which is making it easier to create an /inspect endpoint which produces almost the same data we'd have gotten from `pilosa inspect` on a data directory; the distinction is that it doesn't try to identify the distinction between data from disk and data from operations since the file was loaded. Possibly it should, but it doesn't yet. The snapshot queue is now implemented using the HolderOperator design, which requires some subtle changes to how it works, but overall makes it easier to follow the snapshot queue logic, and also shares that logic with the way Inspect works. The holder's snapshot queue is now provided by the server, in a default environment. The queueless snapshot queue no longer triggers snapshots on enqueue -- it turns out that breaks badly, because a key point about enqueueing a snapshot is that it's safe to do it *during* a transaction on that fragment, and triggering a snapshot during a transaction actually causes horrible errors as the ops log ends up being the old file, which we close. Related to this, we also need to prevent closed fragments from trying to snapshot, so we track fragment openness when opening or closing, and bail on trying to snapshot a fragment which is closed. We also stop using the queueless snapshot queue during tests, because that's a horrible idea. We copy a little bit of the partition logic from the cluster code so we don't have to expose it all, this lets us check whether the node we're looking at is the one which should be primary for a given shard, and if not, identify which node would be. This works only when pointed at a data directory, for now. The test cases for the holder have to be internal, because pilosa doesn't export view/fragment, just Index/Field. This means that the holder test cases can't just use the test/* package, so they duplicate some of its logic, approximately. |
||
|---|---|---|
| .. | ||
| check.go | ||
| check_test.go | ||
| common.go | ||
| config.go | ||
| config_test.go | ||
| doc.go | ||
| export.go | ||
| export_test.go | ||
| generate_config.go | ||
| generate_config_test.go | ||
| import.go | ||
| import_test.go | ||
| inspect.go | ||
| inspect_test.go | ||
| server.go | ||
| server_test.go | ||