mirror of
https://github.com/featurebasedb/featurebase.git
synced 2026-10-17 08:13:35 +00:00
No description
b25eb8f596
This code represents an attempt at providing reliable tracking of whether any bitmaps still in use have access to a given block of mmapped data, allowing us to unmap the data when nothing is using it anymore. The basic approach is as follows: Each mmap is associated with a new object, called a "generation". A generation reflects a particular instance of a given file being mapped. When a bitmap is built from an mmapped data source, the bitmap is given a pointer to the generation as its Source. When bitmap operations combine containers from other bitmaps, they produce new bitmaps that are tagged with the combined set of sources. When we snapshot a file, or for some other reason wish to remap it, the corresponding bitmap has all its containers updated to use the new storage, and the bitmap's source is changed. However, previously-handed-out containers might still have references to the old storage. Those containers would be in bitmaps with the old source. After a bunch of study of trying to reference-count and track this, I realized: We don't actually need to do that, because we already have something suitable for determining whether anything can reach a given object. It's the garbage collector. So we set a finalizer on the generation object, which handles unmapping. There's additional sanity-checks here to confirm things like "we thought this generation should be expiring", and we track timestamps. We could also have things check whether a given bitmap's source was marked as obsolete "a while ago", but that isn't implemented yet. There's a debug version of this which tracks finalization, creation, and ending timestamps, and has a call to provide diagnostics for this. Identical generation IDs get separated out with random suffixes in this case -- there's sometimes a second or third instance of the same name due to a holder closing and reopening, but this basically only happens in testing. Note that generations are still used even when there's no mmapping, but unless debugging is turned on, they shouldn't propagate much -- we don't consider a generation to be the source of a bitmap unless the bitmap actually mapped things from that generation's mmapped storage, or debugging is on. There's a couple of other, possibly more subtle, changes and bug fixes that got caught by the testing on this: * If a fragment is partially opened and then opening some later part fails, we close the earlier parts before returning the error so we aren't leaving it partially open. * Several operations on segments which were requesting that a frozen copy of a bitmap be created are now actually *replacing* their bitmap with the frozen bitmap, rather than discarding it. * intersectRunRun, if it decides to create an array or bitmap, will yield that container instead of discarding it. And why all of this? Why, so we can actually implement the thing where when a fragment has a valid roaring bitmap, but the ops log is corrupt, we can truncate the corrupt part of the ops log and reopen it. Which I did. When the generationdebug build tag is in use, every generation has a finalizer all the time. When it's not, they only get finalizers when we expect them to be done -- say, when closing a fragment. This is because finalizers appear to be possibly-expensive. There's some logical cleanup to openStorage here, dividing part of its work into applyStorage and importStorage, which have a common case for handling "there's no data in this file". |
||
|---|---|---|
| .circleci | ||
| .github | ||
| api/client | ||
| boltdb | ||
| cmd | ||
| ctl | ||
| docs | ||
| encoding/proto | ||
| enterprise | ||
| gcnotify | ||
| gopsutil | ||
| gossip | ||
| http | ||
| internal | ||
| logger | ||
| lru | ||
| mock | ||
| pql | ||
| prometheus | ||
| proto | ||
| roaring | ||
| server | ||
| shardwidth | ||
| stats | ||
| statsd | ||
| syswrap | ||
| test | ||
| testdata/sample_view | ||
| toml | ||
| tracing | ||
| .gitignore | ||
| .golangci.yml | ||
| api.go | ||
| api_test.go | ||
| apimethod_string.go | ||
| attr.go | ||
| attr_test.go | ||
| broadcast.go | ||
| cache.go | ||
| cache_test.go | ||
| CHANGELOG.md | ||
| client.go | ||
| cluster.go | ||
| cluster_internal_test.go | ||
| cmd.go | ||
| CONTRIBUTING.md | ||
| diagnostics.go | ||
| diagnostics_internal_test.go | ||
| doc.go | ||
| Dockerfile | ||
| Dockerfile-clustertests | ||
| event.go | ||
| executor.go | ||
| executor_internal_test.go | ||
| executor_test.go | ||
| field.go | ||
| field_internal_test.go | ||
| field_test.go | ||
| fragment.go | ||
| fragment_internal_test.go | ||
| gc.go | ||
| generation.go | ||
| generation_debug.go | ||
| generation_nodebug.go | ||
| generation_test.go | ||
| go.mod | ||
| go.sum | ||
| handler.go | ||
| holder.go | ||
| holder_internal_test.go | ||
| holder_test.go | ||
| index.go | ||
| index_internal_test.go | ||
| index_test.go | ||
| iterator.go | ||
| iterator_internal_test.go | ||
| LICENSE | ||
| license.exceptions | ||
| Makefile | ||
| NOTES | ||
| NOTICE | ||
| pilosa.go | ||
| pilosa_internal_test.go | ||
| pilosa_test.go | ||
| README.md | ||
| row.go | ||
| row_test.go | ||
| server.go | ||
| server_internal_test.go | ||
| time.go | ||
| time_internal_test.go | ||
| translate.go | ||
| translator_test.go | ||
| uri.go | ||
| uri_internal_test.go | ||
| utils_internal_test.go | ||
| version.go | ||
| view.go | ||
| view_internal_test.go | ||
An open source, distributed bitmap index.
Want to contribute? One of the easiest ways is to tell us how you're using (or want to use) Pilosa. We learn from every discussion!
Docs
See our Documentation for information about installing and working with Pilosa.
Getting Started
-
Start Pilosa with the default configuration:
pilosa serverand verify that it's running:
curl localhost:10101/nodes -
Follow along with the Sample Project to get a better understanding of Pilosa's capabilities.
Data Model
Check out how the Pilosa Data Model works.
Query Language
You can interact with Pilosa directly in the console using the Pilosa Query Language (PQL).
Client Libraries
There are supported libraries for the following languages:
Licenses
The core Pilosa code base and all default builds (referred to as Pilosa Community Edition) are licensed completely under the Apache License, Version 2.0.
If you build Pilosa with the enterprise build tag (Pilosa Enterprise Edition), then that build will include features licensed under the GNU Affero General
Public License (AGPL). Enterprise code is located entirely in the github.com/pilosa/pilosa/enterprise
directory. See github.com/pilosa/pilosa/NOTICE and
github.com/pilosa/pilosa/LICENSE for more information about Pilosa licenses.
Get Support
There are several channels available for you to reach out to us for support.
Contributing
Pilosa is an open source project. Please see our Contributing Guide for information about how to get involved.