Because the *pb.go files get auto-generated, we can't easily
keep the license block on the top of the *.pb.go files.
Instead, we keep a copy of the LICENSE file in the same directory
as the generated *.pb.go files.
NodeState is shared among nodes in the cluster (via gossip
in a gossip implementation) and cached locally in Cluster.Nodes
in order to be available to the /status endpoint.
this affects the scanner/parser and makes it produce signed ints rather than
uints, and so it affects all code that is expecting unsighed ints to come out of
the parser.
Previously the cache type (`lru` or `ranked`) was determined by
checking the ending suffix of the frame. If it ended with a `.n`
then it was ranked. Otherwise it was an LRU.
This commit changes frames so that a `cacheType` option can be
passed in during creation to set either `lru` or `ranked`.
Previously, multiple frames with different prefixes were used to separate
different data layouts. This included separating standard row/column
layouts from inverted column/row layouts as well as storing aggregate
information for timestamp data.
Unfortunately, this caused frame meta data to be copied between multiple
frames and it made it difficult to keep these frames in sync.
This commit separates these different physical layouts into `Views`.
A `Frame` now has one or more views which represent each layout.
Fragments have been moved from under the `Frame` to be contained
within the `View`.
There are two primary views:
- `standard`
- `inverse`
If a frame has a time quantum, then views are generated for these
each of the standard/inverse views. For example a time quantum
of `YMDH` for the date `2000-01-02T00:00:00Z` would create the
following views:
- `standard_2000`
- `inverse_2000`
- `standard_200001`
- `inverse_200001`
- `standard_20000102`
- `inverse_20000102`
From the user's perspective, nothing should change in PQL. Different
PQL statements will handle the appropriate view automatically. For
example, `Bitmap()` and `Profile()` will fetch using the `standard`
view or the `inverse` view, respectively. The `Range()` statement
will lookup the appropriate time-based views automatically.
Databases and frames now require explicit creation and have the
option of setting row & column labels. If no labels are provided
then the default `id` and `profileID` labels are used.
This commit changes the underlying storage for `Bitmap` from a
red-black tree to a roaring bitmap. It also removes bitmaps from
the cache and only stores the bitmap count.
This commit refactors the anti-entropy system to fetch data from
all replicated blocks and only set/clear bits which deviate from
the consensus between all blocks.
An example of this is if 3 nodes had the following bits set for
a single bitmap:
Node A: 1 2 3
Node B: 2 4
Node C: 1 2 4
Then only bits which are set on a majority will be set. In this
case bits 1, 2, & 4 are set but 3 only exists on a single node.
The node performing the merge would then determine the following
set/clear diffs for each node:
Node A: clear(3), set(4)
Node B: set(1)
Node C: none
Once the merge is performed and all nodes receive their diff
instructions then the nodes will be in sync:
Node A: 1 2 4
Node B: 1 2 4
Node C: 1 2 4
There still exists situations where bits can be reset. If Node A
is up and Node B & C are down then Node A's bits will be reset
once B & C come back online. We should add write consistency
settings for incoming writes so that we can ensure that a quorum
is written to before returning a success. This is outside the
scope of this commit though.
This commit changes the `pql.Query` so that it can accept one or more
top-level calls instead of only one.
The query request format change because a query with a single call is
still valid. However, the result format now returns a `results` field
that has one result for each top-level call. The `profiles` field is
still the same, however, it combines all profiles from all bitmap
responses into one return so that there's not duplicate attributes.
Fixes#59
This commit refactors the profile and bitmap attribute stores so
that they share the same code. There is a new `AttrStore` which
associates key/value pairs with a `uint64` identifier.
The previous JSON encoding has been fixed to use protobufs which
fixes encoding issues for int64.
This commit adds the ability of the Fragment to flush the cache
bitmap IDs to disk periodically. They can then be reloaded when
the fragment is reopened.
This commit adds the ability to set string, integer, and boolean
values on bitmaps and profiles within Pilosa. Bitmap attributes
are automatically returned when making a `Bitmap()` call. Profile
attributes must be requested by setting `profile=true` in the
URL.
The function names have also been renamed to initial caps so that
the PQL query language can support math operations in the future.
This commit refactors the pilosa codebase. It makes several major
changes:
* Removes bitmap handles
* Removes dispatch/hold/transport
* Removes etcd dependency
* Adds consistent hash ring for slice placement
* Refactors parser/lexer
* Adds strong typing to PQL AST
* Flattens package hierarchy
