heap sort order backwards
WIP TopN accuracy
adjusted first phase topn to collect all slices id's
incorrect handling of large topns
cache performance enhancement
fix for failed test TestMain_FrameRestore
remove unused code and fix some variable names
Databases and frames now require explicit creation and have the
option of setting row & column labels. If no labels are provided
then the default `id` and `profileID` labels are used.
Consolidates all the pql.Call implementations into a single
pql.Call struct. This is needed to support user-defined schemas
on frames.
This also has the added benefit that a lot of redundant parsing
code has been consolidated.
This changes the API of the types but does not alter the functionality
since only the `context.Background()` is currently being used. Adding
`Context` will help handle fault tolerance in the future by allowing
timeouts to be propagated across calls to different nodes.
Previously the index would create a fragment if it didn't exist
when clearing a bit. This causes additional file handles to be
created when it's already known that the bit is not set.
This commit makes several changes to optimize the TopN() query:
- Reduce highbits() back from 24-bits to 16-bits.
- Reduce MaxArraySize back from 2^20 to 4096.
- Optimize bitmap count invalidation.
- Parallelize TopN() across nodes.
- Parallelize TopN() across slices.
This commit adds a fast path for optimizing queries that only
have `SetBitmapAttrs()` calls in them. It does this by grouping
the calls into a single Bolt transaction and also by merging
multiple attribute updates per ID into one update.
This commit changes the fragment allocation algorithm in the cluster
to make use of the `DB` name. This allows each database to use a
different slice distribution.
Initially, the `frame` was going to be used for allocation, however,
this was problematic since queries can span multiple frames so it's
impossible to choose a single frame to use.
This commit changes the underlying storage for `Bitmap` from a
red-black tree to a roaring bitmap. It also removes bitmaps from
the cache and only stores the bitmap count.
This commit adds an endpoint to restore an entire frame from
another cluster. Multiple hosts can use this endpoint to copy
and rebalance a cluster to a new cluster.
This commit changes the `pql.Query` so that it can accept one or more
top-level calls instead of only one.
The query request format change because a query with a single call is
still valid. However, the result format now returns a `results` field
that has one result for each top-level call. The `profiles` field is
still the same, however, it combines all profiles from all bitmap
responses into one return so that there's not duplicate attributes.
Fixes#59
