Matt Jaffee
6fe1ab45c2
fixup Fragment_Import benchmark
...
It was doing a fresh import on the first round and then importing the same data
into the fragment over and over.
2019-03-05 08:47:44 -06:00
Matt Jaffee
7af64e382c
comments to make import mutex less confusing
2019-03-04 21:38:09 -06:00
Matt Jaffee
8476fffaa7
always operate on storage in importPositions regardless of smallWrite
...
This greatly simplifies the code, and with the recent addition of DirectAddN and
DirectRemoveN should be as or more performant than doing the separate bitmap and
union (in most cases, unsorted data could still be slower). Perhaps more
importantly, it is also less allocation heavy than the union approach. Also
makes it trivial to get the counts of changed bits, so I've cleaned up the stats
to show number of bits we're importing/clearing and the number of bits that
actually changed.
2019-03-04 21:38:09 -06:00
Matt Jaffee
868342d338
fix benchmarks broken by import modifying its args in-place
2019-03-04 21:38:08 -06:00
Matt Jaffee
4c42069d5d
implements bitmap batch Direct* operations which are optimized for sorted data
...
also reset the data on AddN and RemoveN ops if the log write fails
2019-03-04 21:38:08 -06:00
Matt Jaffee
c32c8cda84
only write changed values to op log
2019-03-04 21:38:08 -06:00
Matt Jaffee
9b8a97ccb6
maintain column set in bulkImportMutex to guard against repeats
2019-03-04 21:38:08 -06:00
Matt Jaffee
4082ce655a
wip on adding mutex support to random import perf
2019-03-04 21:38:08 -06:00
Matt Jaffee
f06a9f0e6e
positionsForValue appends to existing slices rather than allocating small ones
2019-03-04 21:38:08 -06:00
Matt Jaffee
a58459cf0f
fix spelling of unnecessary
2019-03-04 21:38:08 -06:00
Matt Jaffee
d429d7c496
code review feedback: add Bitmap.Any and remove unecessary condition
2019-03-04 21:38:07 -06:00
Matt Jaffee
023faebd90
fix bug where opN wasn't getting set/cleared correctly
2019-03-04 21:38:07 -06:00
Matt Jaffee
7037ebf4b3
rename smallPath->smallWrite for consistency
2019-03-04 21:38:07 -06:00
Matt Jaffee
088d618040
increase default MaxOpN
2019-03-04 21:38:07 -06:00
Matt Jaffee
04957308ba
gofmt -s
2019-03-04 21:38:07 -06:00
Matt Jaffee
3cbcb238fb
aggregate small bsi imports into a single-write append
2019-03-04 21:38:07 -06:00
Matt Jaffee
ce656bbcda
factor out code to import/clear by positions
2019-03-04 21:38:07 -06:00
Matt Jaffee
537ae99fb9
add import support with aggregated op log writes
2019-03-04 21:38:07 -06:00
Matt Jaffee
dd4a8755ae
add importValue benchmark and move data building to per benchmark
2019-03-04 21:38:06 -06:00
Matt Jaffee
0de419c95e
add a SetBit/ClearBit path to bulkImport for small updates
...
also add benchmarks for this situation and set default MaxOpN higher which
benchmarks suggest is a good idea
2019-03-04 21:38:06 -06:00
Matthew Jaffee
5027a7a883
Merge pull request #1887 from jaffee/metric.service-fix
...
[fix] improve help strings for metrics options
2019-03-04 21:37:34 -06:00
Matt Jaffee
52fe460f84
improve help for metrics options
2019-03-04 17:36:01 -06:00
Yuce Tekol
3f5feeb83c
Merge pull request #1881 from yuce/1880-shardwidth-in-indexinfo
...
Adds shardWidth to index info in schema
2019-03-01 18:08:24 +03:00
Yuce Tekol
d8ba398dd8
added missing index.go changes
2019-03-01 15:02:49 +03:00
Yuce Tekol
767062ab7c
Adds shardWidth to index info in schema
2019-03-01 14:57:46 +03:00
Matthew Jaffee
784cadd1b1
Merge pull request #1876 from jaffee/union-in-place-unmap
...
[fix] make sure to unmap containers before modifying
2019-02-26 11:13:56 -06:00
Matt Jaffee
67e7281a55
make sure to unmap containers before modifying
2019-02-25 17:12:27 -06:00
seebs
5b43c90762
Merge pull request #1863 from seebs/seebs/flock
...
avoid probable race when creating fragments
2019-02-21 20:17:23 -06:00
Seebs
dde6954de4
view.go: deal with races in fragment creation
...
There existed a case where two goroutines would try to
CreateIfNotExists the same fragment, and the first would
create it, but not put it in the fragments table, then
drop the lock, try to broadcast a message, and if it
succeeded then populate the fragments table. The second
would come along during the broadcast, not find an
entry, try to create one, and fail because the file was
already locked.
Basic problem: At least one test in server/ will fail
if we don't delay to send out broadcast messages. Everything
will lock up if we can wait forever (or even just a very
long time) for the message broadcast. We don't ever want
to have an inconsistent state -- so we don't want to either
fail to get a fragment when one's been created, or get one
that's about to be deleted if the broadcast fails.
So, creation and stashing in the fragments table is
atomic and immediate. After that, we optimistically attempt
to broadcast. If we fail, we fail. We delay up to about
50ms for the broadcast to be done, but after that return
anyway. This way, if things are going well everything
works, and if there's unexpected delays, things work except
some nodes in a cluster may not know about available
shards on other nodes sometimes. But that would have
happened anyway. A proper fix is beyond the scope of this
patch.
2019-02-21 16:43:19 -06:00
tgruben
11fe06be85
Merge pull request #1865 from tgruben/roaring-import-opt
...
removed copy for pilosa roaring files
2019-02-19 11:21:51 -06:00
Todd Gruben
e3fe55522d
comment adjustments
2019-02-19 10:54:09 -06:00
Todd Gruben
02ed4568dd
gofmt
2019-02-19 10:25:34 -06:00
Todd Gruben
a13e5fafa4
updated comments and added error wrapping
2019-02-19 10:19:17 -06:00
Todd Gruben
ffe1a285bd
removed copy for pilosa roaring files
2019-02-18 16:28:37 -06:00
Cody Soyland
62b9697a6a
Merge pull request #1616 from codysoyland/go-module
...
Go module support
2019-02-07 12:09:38 -06:00
Cody Soyland
44f6b1d0e3
Regenerate go mod files
2019-02-07 11:46:30 -06:00
Cody Soyland
a9b8b352fc
Go mod should be enabled here
2019-02-07 10:49:40 -06:00
Cody Soyland
8ffc4537b7
Ensure Go modules disabled during installation of tools
2019-02-07 10:44:46 -06:00
Cody Soyland
9ebe428824
Merge branch 'master' into go-module
2019-02-06 14:12:17 -06:00
Matthew Jaffee
755c1f4238
Merge pull request #1858 from jaffee/missing-doc-heading
...
add Group By heading to query language docs
2019-02-06 13:27:36 -06:00
Matt Jaffee
75d91e9b63
add Group By heading to query language docs
2019-02-06 12:49:57 -06:00
Cody Soyland
30143cf44e
Fix config passing
2019-02-04 09:32:51 -06:00
Cody Soyland
4bef33eaa9
Address code review feedback
2019-02-04 08:56:32 -06:00
Travis Turner
8be84b7c76
Merge pull request #1851 from travisturner/rows-time-range
...
add from/to range arguments to Rows() call
2019-02-03 11:00:48 -06:00
Travis Turner
31e0371a55
update the rows range query docs to match the logic
2019-02-02 21:57:03 -06:00
Travis Turner
a0a641c541
remove extra nesting
2019-02-01 16:46:45 -06:00
Travis Turner
87b3438cb1
add logic to restrict time range to available views
2019-02-01 15:30:10 -06:00
Travis Turner
b544328647
parseTime() function to handle interface to time parsing
2019-01-31 18:29:32 -06:00
Travis Turner
a242b8cbaf
add from/to range arguments to Rows()
2019-01-30 16:27:07 -06:00
alanbernstein
839371711c
Merge pull request #1849 from alanbernstein/docs-fixes
...
Docs fixes
2019-01-30 15:49:09 -06:00
