Introduces new `pilosactl sort` to sort import files by bit
position so they can be inserted faster. Also optimizes container
scanning and adds a `-buffer-size` flag to `import`.
This commit makes several changes to optimize the TopN() query:
- Reduce highbits() back from 24-bits to 16-bits.
- Reduce MaxArraySize back from 2^20 to 4096.
- Optimize bitmap count invalidation.
- Parallelize TopN() across nodes.
- Parallelize TopN() across slices.
CSV exports can now be done with the pilosactl application:
$ pilosactl -d mydb -f myframe -o MYFILE.csv
If `-o` is not specified then the CSV is written to STDOUT. The
exporter combines all slices for the db/frame to into a single
concatenated CSV file.
This commit moves the cache flush to the `Index` and only serializes
a single fragment at a time.
Also included in this commit is the `inspect` command for the
`pilsoactl` binary. This provides insight into pilosa data files.
This commit refactors the anti-entropy system to fetch data from
all replicated blocks and only set/clear bits which deviate from
the consensus between all blocks.
An example of this is if 3 nodes had the following bits set for
a single bitmap:
Node A: 1 2 3
Node B: 2 4
Node C: 1 2 4
Then only bits which are set on a majority will be set. In this
case bits 1, 2, & 4 are set but 3 only exists on a single node.
The node performing the merge would then determine the following
set/clear diffs for each node:
Node A: clear(3), set(4)
Node B: set(1)
Node C: none
Once the merge is performed and all nodes receive their diff
instructions then the nodes will be in sync:
Node A: 1 2 4
Node B: 1 2 4
Node C: 1 2 4
There still exists situations where bits can be reset. If Node A
is up and Node B & C are down then Node A's bits will be reset
once B & C come back online. We should add write consistency
settings for incoming writes so that we can ensure that a quorum
is written to before returning a success. This is outside the
scope of this commit though.
This commit adds a simple benchmarking utility to the `pilosactl`
binary. It currently only supports individual `SetBit()` commands
but it's a good start towards making a generic benchmarking
framework at the integration level.
The subcommands and usage/help messages were also cleaned up to
output correctly.
This commit adds two commands:
pilosactl backup
pilosactl restore
The `Client` implementations have also been added to support these
commands:
func (c *Client) BackupTo(w io.Writer, db, frame string) error
func (c *Client) RestoreFrom(r io.Reader, db, frame string) error
Backups occur on a per-frame basis and all slices from the cluster
are packed into a single tar file. The backup tool attempts to
read from owner nodes in a random order and will retry against
the next owner if one fails.
During restore, the slices are restored to all owner nodes. This
allows users to backup a frame from one cluster and restore it
to a different one -- even if the topology of the new cluster is
different.
This commit fixes the `pilosactl import` command to import data files
across an entire cluster of machines.
A `pilosactl config` command is also added to provide a default config.
This commit refactors the pilosa codebase. It makes several major
changes:
* Removes bitmap handles
* Removes dispatch/hold/transport
* Removes etcd dependency
* Adds consistent hash ring for slice placement
* Refactors parser/lexer
* Adds strong typing to PQL AST
* Flattens package hierarchy
