A `StatsClient` for `expvar` is added so we can track Stats through
the `/debug/vars` endpoint. Tags are nested inside maps so that we
can see stats for db, frame & slice.
Also added a `MultiStatsClient` for chaining multiple `StatsClient`
implementations together (e.g. `expvar` and DataDog).
Introduces new `pilosactl sort` to sort import files by bit
position so they can be inserted faster. Also optimizes container
scanning and adds a `-buffer-size` flag to `import`.
This commit makes several changes to optimize the TopN() query:
- Reduce highbits() back from 24-bits to 16-bits.
- Reduce MaxArraySize back from 2^20 to 4096.
- Optimize bitmap count invalidation.
- Parallelize TopN() across nodes.
- Parallelize TopN() across slices.
CSV exports can now be done with the pilosactl application:
$ pilosactl -d mydb -f myframe -o MYFILE.csv
If `-o` is not specified then the CSV is written to STDOUT. The
exporter combines all slices for the db/frame to into a single
concatenated CSV file.
This commit moves the cache flush to the `Index` and only serializes
a single fragment at a time.
Also included in this commit is the `inspect` command for the
`pilsoactl` binary. This provides insight into pilosa data files.
This commit refactors the anti-entropy system to fetch data from
all replicated blocks and only set/clear bits which deviate from
the consensus between all blocks.
An example of this is if 3 nodes had the following bits set for
a single bitmap:
Node A: 1 2 3
Node B: 2 4
Node C: 1 2 4
Then only bits which are set on a majority will be set. In this
case bits 1, 2, & 4 are set but 3 only exists on a single node.
The node performing the merge would then determine the following
set/clear diffs for each node:
Node A: clear(3), set(4)
Node B: set(1)
Node C: none
Once the merge is performed and all nodes receive their diff
instructions then the nodes will be in sync:
Node A: 1 2 4
Node B: 1 2 4
Node C: 1 2 4
There still exists situations where bits can be reset. If Node A
is up and Node B & C are down then Node A's bits will be reset
once B & C come back online. We should add write consistency
settings for incoming writes so that we can ensure that a quorum
is written to before returning a success. This is outside the
scope of this commit though.
This commit adds a simple benchmarking utility to the `pilosactl`
binary. It currently only supports individual `SetBit()` commands
but it's a good start towards making a generic benchmarking
framework at the integration level.
The subcommands and usage/help messages were also cleaned up to
output correctly.
This commit adds an endpoint to restore an entire frame from
another cluster. Multiple hosts can use this endpoint to copy
and rebalance a cluster to a new cluster.
This commit changes the `pql.Query` so that it can accept one or more
top-level calls instead of only one.
The query request format change because a query with a single call is
still valid. However, the result format now returns a `results` field
that has one result for each top-level call. The `profiles` field is
still the same, however, it combines all profiles from all bitmap
responses into one return so that there's not duplicate attributes.
Fixes#59
This commit adds two commands:
pilosactl backup
pilosactl restore
The `Client` implementations have also been added to support these
commands:
func (c *Client) BackupTo(w io.Writer, db, frame string) error
func (c *Client) RestoreFrom(r io.Reader, db, frame string) error
Backups occur on a per-frame basis and all slices from the cluster
are packed into a single tar file. The backup tool attempts to
read from owner nodes in a random order and will retry against
the next owner if one fails.
During restore, the slices are restored to all owner nodes. This
allows users to backup a frame from one cluster and restore it
to a different one -- even if the topology of the new cluster is
different.
This commit adds the ability of the Fragment to flush the cache
bitmap IDs to disk periodically. They can then be reloaded when
the fragment is reopened.
This commit adds the ability to set string, integer, and boolean
values on bitmaps and profiles within Pilosa. Bitmap attributes
are automatically returned when making a `Bitmap()` call. Profile
attributes must be requested by setting `profile=true` in the
URL.
The function names have also been renamed to initial caps so that
the PQL query language can support math operations in the future.
This commit fixes the `pilosactl import` command to import data files
across an entire cluster of machines.
A `pilosactl config` command is also added to provide a default config.
This commit adds integration testing for the `cmd/pilosa` package.
It uses randomly generated `set()` data and verifies the data using `get()`
both immediately after the insert and after restart.
This commit refactors the pilosa codebase. It makes several major
changes:
* Removes bitmap handles
* Removes dispatch/hold/transport
* Removes etcd dependency
* Adds consistent hash ring for slice placement
* Refactors parser/lexer
* Adds strong typing to PQL AST
* Flattens package hierarchy
This commit moves dependency management to use godep and the
new experimental vendoring support in Go 1.5.
NOTE: You must set `GO15VENDOREXPERIMENT` in your bash environment
for vendoring to work properly! See the updated `README` for details.
This commit moves the index package to the root. Because pilosa
is an index at its core, it's redundant to have an index
subpackage. It also provides better naming such as `pilosa.Bitmap`
instead of `index.Bitmap`.
This commit refactors the config into a `main.Config` object instead
of a global singleton. The `core.Service` is also refactored into
the `main` package and individual pieces of the service are wired
together by the `pilosa` binary.
These two changes are required to begin to decouple packages from
one another and allow them to be individually unit tested. Previously
most top-level objects in the system could access any other top-level
object through the `core.Service` which effectively made `Service` a
singleton in the system. Each top-level object now has inline interfaces
for their dependencies so that can be set at runtime by the `main`
package or can be mocked by a test package.
