This changes the API of the types but does not alter the functionality
since only the `context.Background()` is currently being used. Adding
`Context` will help handle fault tolerance in the future by allowing
timeouts to be propagated across calls to different nodes.
Previously the index would create a fragment if it didn't exist
when clearing a bit. This causes additional file handles to be
created when it's already known that the bit is not set.
This commit makes several changes to optimize the TopN() query:
- Reduce highbits() back from 24-bits to 16-bits.
- Reduce MaxArraySize back from 2^20 to 4096.
- Optimize bitmap count invalidation.
- Parallelize TopN() across nodes.
- Parallelize TopN() across slices.
This commit adds a fast path for optimizing queries that only
have `SetBitmapAttrs()` calls in them. It does this by grouping
the calls into a single Bolt transaction and also by merging
multiple attribute updates per ID into one update.
This commit changes the fragment allocation algorithm in the cluster
to make use of the `DB` name. This allows each database to use a
different slice distribution.
Initially, the `frame` was going to be used for allocation, however,
this was problematic since queries can span multiple frames so it's
impossible to choose a single frame to use.
This commit changes the underlying storage for `Bitmap` from a
red-black tree to a roaring bitmap. It also removes bitmaps from
the cache and only stores the bitmap count.
This commit adds an endpoint to restore an entire frame from
another cluster. Multiple hosts can use this endpoint to copy
and rebalance a cluster to a new cluster.
This commit changes the `pql.Query` so that it can accept one or more
top-level calls instead of only one.
The query request format change because a query with a single call is
still valid. However, the result format now returns a `results` field
that has one result for each top-level call. The `profiles` field is
still the same, however, it combines all profiles from all bitmap
responses into one return so that there's not duplicate attributes.
Fixes#59
This commit refactors the profile and bitmap attribute stores so
that they share the same code. There is a new `AttrStore` which
associates key/value pairs with a `uint64` identifier.
The previous JSON encoding has been fixed to use protobufs which
fixes encoding issues for int64.
This commit adds the ability to set string, integer, and boolean
values on bitmaps and profiles within Pilosa. Bitmap attributes
are automatically returned when making a `Bitmap()` call. Profile
attributes must be requested by setting `profile=true` in the
URL.
The function names have also been renamed to initial caps so that
the PQL query language can support math operations in the future.
This commit attaches the mmap-backed roaring bitmaps as storage
to the fragments. Currently snapshotting is not supported from
Pilosa so the WAL will grow indefinitely, however, it is not a
difficult change to make.
This commit adds `set()` execution to the Executor. When a set is
issued and the replication factor is greater than 1 then the
query will be forwarded to all other nodes that own the partition.
Currently there is not a way to recover from a `set()` that only
succeeds on some nodes.
This commit refactors the pilosa codebase. It makes several major
changes:
* Removes bitmap handles
* Removes dispatch/hold/transport
* Removes etcd dependency
* Adds consistent hash ring for slice placement
* Refactors parser/lexer
* Adds strong typing to PQL AST
* Flattens package hierarchy
