Commit graph

540 commits

Author SHA1 Message Date
tgruben
a12ec57eec Merge pull request #54 from umbel/todd
fixed issue with SetBit/ClearBit not correctly reporting bitchange
2016-02-26 12:13:41 -06:00
Todd Gruben
ab5f436e44 merge 2016-02-26 12:13:03 -06:00
tgruben
794dccc854 Merge pull request #55 from benbjohnson/backup-restore
Add backup/restore endpoints
2016-02-25 18:37:40 -06:00
Ben Johnson
acdcbacad7 add backup/restore endpoints
This commit adds io.ReaderFrom and io.WriterTo implementations to
the Fragment and also adds backup & restore endpoints to the Handler.
2016-02-25 17:29:45 -07:00
Todd Gruben
daf828d2a8 generated using go generate 2016-02-25 17:15:40 -06:00
Todd Gruben
c60773d33a issue 41 2016-02-25 15:43:52 -06:00
Todd Gruben
1aca83a546 align more with go idioms 2016-02-24 15:02:24 -06:00
Todd Gruben
a72edfdae8 merged 2016-02-24 14:19:44 -06:00
Todd Gruben
ed1890b3a2 adjust SetBit/ClearBit to releably report bit change 2016-02-24 14:00:41 -06:00
Ben Johnson
7ef7eee569 Merge pull request #52 from benbjohnson/range
Add Range() support
2016-02-23 14:59:03 -07:00
Ben Johnson
42ba865b6d add Range() support
This commit adds support for setting bits in time-based frames.
2016-02-23 14:53:03 -07:00
tgruben
faf6406791 Merge pull request #50 from umbel/todd
added support for returning SetBit effect
2016-02-22 11:36:22 -06:00
Todd Gruben
7d6a8846b5 cleaned up tests for SetBit 2016-02-22 11:30:06 -06:00
Todd Gruben
a74d826b41 removed debugging code' 2016-02-17 13:23:30 -06:00
Ben Johnson
e1eca36a9d fix int64 attr 2016-02-17 10:59:17 -07:00
Todd Gruben
6d3d10da2b changed protobuf to uint; doesn't work 2016-02-17 11:06:57 -06:00
Todd Gruben
7c8a27338a added support for returning SetBit effect 2016-02-16 17:38:56 -06:00
tgruben
00413a9e9a Merge pull request #49 from benbjohnson/attr
Refactor attribute stores
2016-02-16 16:27:22 -06:00
Ben Johnson
b873b4c084 refactor attribute stores
This commit refactors the profile and bitmap attribute stores so
that they share the same code. There is a new `AttrStore` which
associates key/value pairs with a `uint64` identifier.

The previous JSON encoding has been fixed to use protobufs which
fixes encoding issues for int64.
2016-02-16 13:41:37 -07:00
tgruben
2b5831cbf2 Merge pull request #47 from umbel/proto-testdata
Re-add proto/testdata
2016-02-15 10:26:35 -06:00
Ben Johnson
985613e6e1 add proto/testdata 2016-02-15 09:25:14 -07:00
tgruben
22c3ac5aa4 Merge pull request #46 from benbjohnson/cache-persistence
Add cache persistence
2016-02-15 09:29:57 -06:00
Ben Johnson
1d7e3b18fe add cache persistence
This commit adds the ability of the Fragment to flush the cache
bitmap IDs to disk periodically. They can then be reloaded when
the fragment is reopened.
2016-02-13 10:02:53 -07:00
tgruben
0e9e7f43a5 Merge pull request #45 from benbjohnson/topn
TopN()
2016-02-10 15:00:41 -06:00
Ben Johnson
f59e441b24 TopN()
This commit adds TopN() functionality to the fragment, executor
and handler.
2016-02-09 14:43:49 -07:00
tgruben
6a3296457a Merge pull request #44 from benbjohnson/pql
Add bitmap and profile attribute support
2016-02-01 15:16:31 -06:00
Ben Johnson
01afc5364d add bitmap and profile attribute support
This commit adds the ability to set string, integer, and boolean
values on bitmaps and profiles within Pilosa. Bitmap attributes
are automatically returned when making a `Bitmap()` call. Profile
attributes must be requested by setting `profile=true` in the
URL.

The function names have also been renamed to initial caps so that
the PQL query language can support math operations in the future.
2016-02-01 08:44:33 -07:00
tgruben
0bb3c480d6 Merge pull request #42 from benbjohnson/attributes
Bitmap Attributes Storage
2016-01-19 11:17:19 -06:00
Ben Johnson
411a7dbcd4 bitmap attributes
This commit adds storage for bitmap attributes and adds
a Frame type to contain them.
2016-01-15 11:49:09 -07:00
tgruben
880de7b9dd Merge pull request #38 from benbjohnson/roaring-array-append-fix
Fix roaring array append
2016-01-09 17:18:17 -06:00
Ben Johnson
83ec088cd9 fix roaring array append
This commit fixes a bug where appends to an array container did
not cause it to roll over to a bitmap container when it reached
the max array size.
2016-01-09 12:27:20 -07:00
Ben Johnson
24f713ffa3 Merge pull request #37 from benbjohnson/import-error
Add import error string
2016-01-08 14:23:23 -07:00
Ben Johnson
6dd371af46 add import error string 2016-01-08 14:22:54 -07:00
tgruben
b55b4ac89b Merge pull request #35 from benbjohnson/roaring-uint64
Support uint64 in roaring bitmaps
2016-01-07 16:22:42 -06:00
Ben Johnson
eb4de1843a support uint64 in roaring bitmaps
This commit refactors the roaring bitmaps to use uint64 values
instead of uint32 values. This is required for the size of
values we need in pilosa.

This change is not backwards compatible with the previous data
format so any existing data directories need to be removed before
using this code.
2016-01-07 13:43:57 -07:00
Ben Johnson
ad80bb1a2e Merge pull request #34 from umbel/testing
added support for period (.) in frame names
2016-01-06 14:28:36 -07:00
Todd Gruben
7d5cb406e2 added support for period (.) in frame names 2016-01-06 15:24:25 -06:00
tgruben
f1d33f75da Merge pull request #32 from benbjohnson/cluster-import
Cluster-wide Import
2016-01-06 09:31:26 -06:00
Ben Johnson
7c8c2251ac cluster-wide import
This commit fixes the `pilosactl import` command to import data files
across an entire cluster of machines.

A `pilosactl config` command is also added to provide a default config.
2016-01-05 20:09:17 -07:00
Ben Johnson
60b8e2521c Merge pull request #29 from benbjohnson/import
Bulk Importer
2016-01-02 15:14:41 -07:00
Ben Johnson
54db22c54e bulk importer
This commit adds the pilosactl binary with a single `import` command.
The import allows users to bulk import data from a CSV file.
2016-01-02 15:10:17 -07:00
tgruben
79520bc95b Merge pull request #28 from benbjohnson/intg
Integration testing
2015-12-22 17:55:07 -06:00
Ben Johnson
9ee6ef6436 integration testing
This commit adds integration testing for the `cmd/pilosa` package.
It uses randomly generated `set()` data and verifies the data using `get()`
both immediately after the insert and after restart.
2015-12-22 16:33:27 -07:00
Todd Gruben
c39e7656ed added benchmark for assembly vs native popcount 2015-12-22 09:34:26 -06:00
tgruben
29116ce5d0 Merge pull request #27 from benbjohnson/storage
Storage
2015-12-18 10:05:16 -06:00
Ben Johnson
19a1a90800 roaring storage
This commit attaches the mmap-backed roaring bitmaps as storage
to the fragments. Currently snapshotting is not supported from
Pilosa so the WAL will grow indefinitely, however, it is not a
difficult change to make.
2015-12-17 20:35:33 -07:00
Ben Johnson
dd8d1e0718 fix roaring bitmap mapping flag
This commit properly sets the `mapped` flag on a roaring bitmap
container so that it can be unmapped when it's changed.
2015-12-16 13:20:18 -07:00
tgruben
4fa0c0e025 Merge pull request #26 from benbjohnson/roaring
Delta roaring bitmap
2015-12-16 07:38:59 -06:00
Ben Johnson
b2f45a7256 add delta roaring bitmap
This commit adds an implementation of roaring bitmaps that can be
used with mmap files and also allow for delta changes that are
appended to the end of the file as a write-head log.
2015-12-15 20:07:46 -07:00
Todd Gruben
1701da8d5d updated the Count cache on set operations 2015-12-11 13:19:19 -06:00