Travis Turner
c66daabc81
convert the anti-entropy logic to use ImportRoaring instead of QueryNode
2018-12-10 21:05:08 -06:00
Ben Johnson
8e49332b25
Add distributed tracing.
2018-11-21 15:08:33 -06:00
Matt Jaffee
8bc1104585
fix fragment checksums race condition
2018-11-20 14:16:23 -06:00
Matt Jaffee
65f478470f
logging cleanup - start with lowercase unless reporting error or warning
2018-11-20 14:08:06 -06:00
Seebs
a203313143
move Logger and Stats to their own packages
...
I'd like to add stat tracking to Roaring, which means it
has to be able to import the stats package, which means
stats has to be a package rather than part of the pilosa
package. If stats stops being in pilosa, it still needs
a way to import logger, so logger also has to leave the
pilosa package. Then everything using them needs to import
them and use package selectors on their names.
This doesn't actually add the stats support to roaring,
it just makes it so there's a way to import the stats
code from something in the roaring package.
2018-11-15 15:10:44 -06:00
Yuce Tekol
70f85211d9
prevent panic in Bitmap.UnmarshalBinary when there is no data
2018-11-15 22:06:21 +03:00
Matt Jaffee
30590c83bd
Merge branch 'master' into new-rows-iterate
2018-10-25 08:14:58 -05:00
Travis Turner
5fb6ef224c
add clear support for ImportRoaring
2018-10-23 17:37:57 -05:00
Travis Turner
a7a15c64a2
support clear imports to int fields. fix bug in fragment.sum
2018-10-23 17:37:57 -05:00
Travis Turner
caf8e06712
add clear functional option for imports
2018-10-23 17:37:57 -05:00
Matt Jaffee
b6a953cade
cleanup groupby - more comments, remove panic, remove dup test
2018-10-16 19:56:54 -05:00
Matt Jaffee
fb706ab883
add GroupBy Rows(limit) test and fix bug
...
run all group by tests on two cluster sizes
2018-10-12 18:47:41 -05:00
Matt Jaffee
07d279a155
implement mergeGroupCounts w/o map, remove dead code
...
move rowFilters to fragment.go
new mergeGroupCounts implementation takes limit into account while merging,
exploits inherent order of group count results.
2018-10-11 19:06:46 -05:00
Matt Jaffee
9d896c5d2f
implement alternate groupByIterator using fragment rowIterator
...
doesn't re-intersect the same rows for every record
2018-10-11 18:01:44 -05:00
Matt Jaffee
36a539d24e
Merge branch 'master' into new-rows-iterate
2018-10-09 12:27:05 -05:00
Matt Jaffee
1474884f5d
use existing var instead of recalculating
...
silly mistake - thanks todd
2018-10-09 10:27:57 -05:00
Matt Jaffee
e2bbcb28e5
fix linter issues
2018-10-08 19:10:16 -05:00
Matt Jaffee
c172ca0680
combine fragment.rows and rowsForColumn with generalized filter
...
use filter funcs with closures for state instead of methods on structs. seems a
bit cleaner.
2018-10-08 16:16:39 -05:00
Travis Turner
619bc1bcd9
implements fragment.setRow(row, rowID)
2018-10-05 08:45:05 -05:00
Matt Jaffee
8cd82af2e7
remove extraneous fragment.rows* methods
...
variadic filters makes separate methods unnecessary
2018-10-02 09:31:13 -05:00
Matt Jaffee
4f3f2e1a49
remove noFilter and filterWithOffsetLimit
...
can use an empty list of filters and a list of offsetFilter followed by limit
filter respectively
2018-10-02 09:25:07 -05:00
Matt Jaffee
ed1b09a1cd
fix columnID<>shard checks in executor and fragment
...
fragment panics if rowsForColumn is called with a column id not in the
fragment's shard. The justification for this is that we're wasting resources if
we're sending requests for a specific column to any shard other than the one
which contains that column.
2018-09-28 10:17:53 -05:00
Matt Jaffee
78ff75690d
convert Rows to use previous/limit
...
pass previous+1 directly to fragment.rows so that the iterator can seek directly
to the start point. handle limit inside reduce so it can skip out early and
avoid extra allocation.
2018-09-27 16:14:40 -05:00
Travis Turner
4a411067a5
remove Set() (and Clear()) from vector interface, and add error to return
2018-09-24 16:16:22 -05:00
Travis Turner
b6e99734f0
implement ClearRow() query
2018-09-24 15:52:43 -05:00
Travis Turner
4f17dbdbf1
add support for Bool fields
...
prevent import of non-boolean row values to bool fields
2018-09-21 09:25:41 -05:00
Travis Turner
995a24d0af
ensure mutex imports unset previous columns
2018-09-20 11:25:43 -05:00
Matt Jaffee
54ce537327
copy non roaring-import code from Todds's row-iterate PR
...
tests passing
2018-09-17 16:01:51 -05:00
Matt Jaffee
b412309447
rename standard roaring to "official" throughout
2018-09-13 13:38:18 -05:00
Matt Jaffee
7b0d4d75b4
remove validators on import-roaring and test handler
...
validators are for query args, not url vars. Also some misc cleanup and error
handling in the handler.
2018-09-13 13:14:35 -05:00
Matt Jaffee
fd655c998e
rename all instances of roaringbytes
...
function names now importRoaring and byte slice args are just called data. Also
renamed http endpoint from /importroaring to /import-roaring
2018-09-13 12:27:23 -05:00
Matt Jaffee
9e7cfeedc9
readability changes in fragment.go
2018-09-13 11:41:46 -05:00
Matt Jaffee
af7d405b11
add explanatory comment and move comment to correct spot
2018-09-13 10:41:36 -05:00
Matt Jaffee
79c783eaf8
add nolint directive
...
there isn't much piont in making snapshot take a WriterTo, because the only
thing that's going to be written to a fragment is a roaring bitmap. making it
WriterTo just makes it slightly more annoying to jump into the Bitmap.WriterTo
implementation. since it's a private function, it should be straightforward to
change this in the future if the need arises.
2018-09-12 17:31:57 -05:00
Matt Jaffee
d0d0bdd722
simplify fragment.unprotectedRow (unparam)
2018-09-12 17:25:12 -05:00
Matt Jaffee
f8569102f5
update some fragment comments re: the cache
2018-09-12 17:10:28 -05:00
Matt Jaffee
3ea07ae3a7
use cache.Recalculate instead of Invalidate for imports
...
Invalidate does not always rebuild the cache - if the last rebuild is < 10s ago,
it does nothing. We always want to rebuild the cache after imports.
Also updated the comments around recalculate/invalidate to clarify.
2018-09-12 17:07:46 -05:00
Matt Jaffee
14a8362261
roaring import - add cache invalidate and broken test
2018-09-12 16:48:07 -05:00
Matt Jaffee
b5a14de1fd
roaring import: fix api doc, refactor/rename, test merge
2018-09-12 16:11:51 -05:00
Todd Gruben
156260e68e
roaring allowed import to merge with existing data
2018-09-12 13:20:07 -05:00
Matt Jaffee
8254d0fadd
improve import roaring docs, check errors
2018-09-12 12:08:57 -05:00
Matt Jaffee
d3717e2afe
make Bitmap.UnmarshalBinary work for roaring or pilosa format
2018-09-12 10:50:05 -05:00
Todd Gruben
de478870fb
found deadlock in new code
2018-09-05 10:13:15 -05:00
Todd Gruben
cc80e0b0e4
initial support for bulk importing standard roaring files per shard
2018-09-05 10:13:15 -05:00
Travis Turner
f1a460aca7
adds view parameter to sync logic for syncing time fields
2018-08-16 11:27:08 -05:00
Cody Soyland
f01d850b17
Fix linter issues: gosimple
2018-07-20 09:06:43 -05:00
Travis Turner
f95d0f93fd
remove mapVector implementation of the vector interface
2018-07-19 14:02:58 -05:00
Travis Turner
a733fd7f67
fix spelling mistake
2018-07-19 14:01:30 -05:00
Travis Turner
06f5d04f1c
swap out mapVector for rowsVector in mutex fields
2018-07-19 14:01:30 -05:00
Travis Turner
a07ed360ab
add mutex field type
2018-07-19 14:01:29 -05:00
