Commit graph

517 commits

Author SHA1 Message Date
tgruben
0e9e7f43a5 Merge pull request #45 from benbjohnson/topn
TopN()
2016-02-10 15:00:41 -06:00
Ben Johnson
f59e441b24 TopN()
This commit adds TopN() functionality to the fragment, executor
and handler.
2016-02-09 14:43:49 -07:00
tgruben
6a3296457a Merge pull request #44 from benbjohnson/pql
Add bitmap and profile attribute support
2016-02-01 15:16:31 -06:00
Ben Johnson
01afc5364d add bitmap and profile attribute support
This commit adds the ability to set string, integer, and boolean
values on bitmaps and profiles within Pilosa. Bitmap attributes
are automatically returned when making a `Bitmap()` call. Profile
attributes must be requested by setting `profile=true` in the
URL.

The function names have also been renamed to initial caps so that
the PQL query language can support math operations in the future.
2016-02-01 08:44:33 -07:00
tgruben
0bb3c480d6 Merge pull request #42 from benbjohnson/attributes
Bitmap Attributes Storage
2016-01-19 11:17:19 -06:00
Ben Johnson
411a7dbcd4 bitmap attributes
This commit adds storage for bitmap attributes and adds
a Frame type to contain them.
2016-01-15 11:49:09 -07:00
tgruben
880de7b9dd Merge pull request #38 from benbjohnson/roaring-array-append-fix
Fix roaring array append
2016-01-09 17:18:17 -06:00
Ben Johnson
83ec088cd9 fix roaring array append
This commit fixes a bug where appends to an array container did
not cause it to roll over to a bitmap container when it reached
the max array size.
2016-01-09 12:27:20 -07:00
Ben Johnson
24f713ffa3 Merge pull request #37 from benbjohnson/import-error
Add import error string
2016-01-08 14:23:23 -07:00
Ben Johnson
6dd371af46 add import error string 2016-01-08 14:22:54 -07:00
tgruben
b55b4ac89b Merge pull request #35 from benbjohnson/roaring-uint64
Support uint64 in roaring bitmaps
2016-01-07 16:22:42 -06:00
Ben Johnson
eb4de1843a support uint64 in roaring bitmaps
This commit refactors the roaring bitmaps to use uint64 values
instead of uint32 values. This is required for the size of
values we need in pilosa.

This change is not backwards compatible with the previous data
format so any existing data directories need to be removed before
using this code.
2016-01-07 13:43:57 -07:00
Ben Johnson
ad80bb1a2e Merge pull request #34 from umbel/testing
added support for period (.) in frame names
2016-01-06 14:28:36 -07:00
Todd Gruben
7d5cb406e2 added support for period (.) in frame names 2016-01-06 15:24:25 -06:00
tgruben
f1d33f75da Merge pull request #32 from benbjohnson/cluster-import
Cluster-wide Import
2016-01-06 09:31:26 -06:00
Ben Johnson
7c8c2251ac cluster-wide import
This commit fixes the `pilosactl import` command to import data files
across an entire cluster of machines.

A `pilosactl config` command is also added to provide a default config.
2016-01-05 20:09:17 -07:00
Ben Johnson
60b8e2521c Merge pull request #29 from benbjohnson/import
Bulk Importer
2016-01-02 15:14:41 -07:00
Ben Johnson
54db22c54e bulk importer
This commit adds the pilosactl binary with a single `import` command.
The import allows users to bulk import data from a CSV file.
2016-01-02 15:10:17 -07:00
tgruben
79520bc95b Merge pull request #28 from benbjohnson/intg
Integration testing
2015-12-22 17:55:07 -06:00
Ben Johnson
9ee6ef6436 integration testing
This commit adds integration testing for the `cmd/pilosa` package.
It uses randomly generated `set()` data and verifies the data using `get()`
both immediately after the insert and after restart.
2015-12-22 16:33:27 -07:00
Todd Gruben
c39e7656ed added benchmark for assembly vs native popcount 2015-12-22 09:34:26 -06:00
tgruben
29116ce5d0 Merge pull request #27 from benbjohnson/storage
Storage
2015-12-18 10:05:16 -06:00
Ben Johnson
19a1a90800 roaring storage
This commit attaches the mmap-backed roaring bitmaps as storage
to the fragments. Currently snapshotting is not supported from
Pilosa so the WAL will grow indefinitely, however, it is not a
difficult change to make.
2015-12-17 20:35:33 -07:00
Ben Johnson
dd8d1e0718 fix roaring bitmap mapping flag
This commit properly sets the `mapped` flag on a roaring bitmap
container so that it can be unmapped when it's changed.
2015-12-16 13:20:18 -07:00
tgruben
4fa0c0e025 Merge pull request #26 from benbjohnson/roaring
Delta roaring bitmap
2015-12-16 07:38:59 -06:00
Ben Johnson
b2f45a7256 add delta roaring bitmap
This commit adds an implementation of roaring bitmaps that can be
used with mmap files and also allow for delta changes that are
appended to the end of the file as a write-head log.
2015-12-15 20:07:46 -07:00
Todd Gruben
1701da8d5d updated the Count cache on set operations 2015-12-11 13:19:19 -06:00
Ben Johnson
761be23e33 main: merge host and addr 2015-12-04 15:38:13 -07:00
Ben Johnson
809843668d Merge pull request #23 from benbjohnson/set
Add set() execution
2015-12-04 12:11:52 -07:00
Ben Johnson
e13da6aebd add set execution
This commit adds `set()` execution to the Executor. When a set is
issued and the replication factor is greater than 1 then the
query will be forwarded to all other nodes that own the partition.

Currently there is not a way to recover from a `set()` that only
succeeds on some nodes.
2015-12-03 20:13:17 -07:00
Ben Johnson
ab636e6296 add proto testdata 2015-12-03 10:26:25 -07:00
Ben Johnson
8aa9cb7115 Merge pull request #22 from benbjohnson/refactor
Refactor
2015-12-02 15:41:06 -07:00
Ben Johnson
34fa1afad0 refactor
This commit refactors the pilosa codebase. It makes several major
changes:

* Removes bitmap handles
* Removes dispatch/hold/transport
* Removes etcd dependency
* Adds consistent hash ring for slice placement
* Refactors parser/lexer
* Adds strong typing to PQL AST
* Flattens package hierarchy
2015-12-02 15:34:49 -07:00
Ben Johnson
e600c22089 Merge pull request #12 from benbjohnson/commands
Replace command queue with mutex
2015-09-10 17:30:38 -06:00
Ben Johnson
26bc85d565 replace command queue with mutex
This commit simplifies Fragment by replacing the request/response channels
with a `sync.Mutex` to restrict access. This also simplifies the
`FragmentContainer` code which can largely be removed now since most of
the methods are simply wrappers around `GetFragment()` and `Fragment`.
2015-09-10 17:26:32 -06:00
Ben Johnson
7048aa519b clean up ids 2015-09-03 15:50:08 -06:00
Ben Johnson
d777a4ed8f Merge pull request #11 from benbjohnson/vendor
Flatten util into pilosa
2015-09-03 14:45:55 -06:00
Ben Johnson
45fd4fc992 flatten util into pilosa
This commit moves ID-related types and functions to the top
level package and moves remaining statsd wrapper into a `statsd`
package.
2015-09-03 14:43:34 -06:00
Ben Johnson
a65c605c4d Merge pull request #10 from benbjohnson/vendor
godep
2015-09-03 13:55:13 -06:00
Ben Johnson
1d1cb134ec godep
This commit moves dependency management to use godep and the
new experimental vendoring support in Go 1.5.

NOTE: You must set `GO15VENDOREXPERIMENT` in your bash environment
for vendoring to work properly! See the updated `README` for details.
2015-09-03 13:52:45 -06:00
Ben Johnson
6b2b2672d7 Merge pull request #9 from benbjohnson/flatten
Move index package to root package
2015-08-31 16:56:58 -06:00
Ben Johnson
3c832591df move index package to root package
This commit moves the index package to the root. Because pilosa
is an index at its core, it's redundant to have an index
subpackage. It also provides better naming such as `pilosa.Bitmap`
instead of `index.Bitmap`.
2015-08-31 16:54:53 -06:00
Ben Johnson
9b6b43c9c3 Merge pull request #8 from benbjohnson/storage
refactor storage
2015-08-30 16:51:21 -06:00
Ben Johnson
306249c293 refactor storage
This commit moves individual storage engines to their own packages.
They can be registered and instantiated by a string name.
2015-08-30 16:49:32 -06:00
Ben Johnson
6c77b64756 Merge pull request #7 from benbjohnson/refactor-index
Refactor index package
2015-08-24 16:33:05 -06:00
Ben Johnson
c38d023cc1 refactor index package
This commit does clean up and minor refactoring of the
index package.
2015-08-24 16:31:39 -06:00
Ben Johnson
afbdebaac8 Merge pull request #6 from benbjohnson/vet
go vet
2015-08-20 15:10:42 -06:00
Ben Johnson
879fea53d2 remove interfaces pkg 2015-08-20 15:07:37 -06:00
Ben Johnson
73631de9e8 vet 2015-08-20 13:46:39 -06:00
Ben Johnson
6fdd32d5ce remove core.Service 2015-08-20 13:15:27 -06:00