Commit graph

405 commits

Author SHA1 Message Date
Matt Jaffee
226f15446b
lock MaxMapCount and fix unused var 2019-03-19 12:55:26 -05:00
Matt Jaffee
e469285fe3
add fragment mmap tracking and limiting
in the case that the map limit is reached, we'll fall back to reading the file
into memory normally.
2019-03-19 12:55:26 -05:00
Matt Jaffee
e33ca2d0ae
use UnionInPlace in import-roaring
get the count of the existing fragment and compare it to the incoming bits to
decide which should be unioned into the other. This should generally result in
far fewer allocations, though there is much work that needs to be done within
UnionInPlace to further improve things.

unrelatedly, I added a TODO to change the long-query-time option to move it out
of cluster. It should probably be happening at the API level so that different
handlers can reuse it, but if we're going to do that we'll want to make sure
that any potentially time intensive operations are pulled into api from
handler (e.g. protobuf decoding)
2019-03-11 17:43:55 -05:00
Yuce Tekol
767062ab7c
Adds shardWidth to index info in schema 2019-03-01 14:57:46 +03:00
Travis Turner
efb9f97e61
Advertise address and listen on 0.0.0.0
This commit adds support for advertise address by using a new config
option `advertise`, or by defaulting its value to that
specified in `bind`.

Also adds support for listening on 0.0.0.0 by trying to determine
the preferred outbound IP to use for the advertise address.
2019-01-21 22:36:55 -06:00
Matt Jaffee
6f4dee5e31
pass loggers around properly in gossip 2019-01-21 14:47:02 -06:00
Matt Jaffee
f1ecead069
fix failing tests due to staticcheck fixes 2019-01-21 14:38:58 -06:00
Yuce Tekol
dc735c17b8
Fixes #1805. Fixes Store call error messages 2019-01-16 20:52:33 +03:00
Yuce Tekol
65c30283d7
adds tests for GroupBy with keys; removes unused Bit message from proto 2019-01-03 15:20:02 +03:00
Matt Jaffee
31ab3d37b8
add stress tests
these produce an issue on master, but it is fixed on this branch
2018-12-19 17:21:57 -06:00
Travis Turner
d28170ddc6
Syncs AvailableShards when handling a ResizeInstruction.
There was a situation where availableShards on a new
node were not in sync with the cluster, so queries
following a resize were incorrect.
- Start a one-node cluster.
- Write data to shards 0 and 1
- Start a second node.
In the case where the hash algo was moving shard 0 to
node1, then node1 only knew about shard 0, so queries
to node1 would be incomplete.

This PR modifies the ResizeInstruction message to replace
`Schema` with `NodeStatus` (which contains both `Schema` and
`AvailableShards`). So now when a resize instruction is received,
the receiving node is able to sync its schema and availableShards.
2018-12-18 08:34:48 -06:00
Matt Jaffee
ef6db5cc1a
propogate updates to node details (not just additions and deletions)
to all nodes in cluster, not just coordinator
2018-12-04 14:03:14 -06:00
Cody Soyland
53dfccbde4
Merge branch 'master' into arm64-support 2018-11-30 11:05:04 -06:00
Cody Soyland
f0c6394c61 Use syscall.Dup3 on ARM64 as Dup2 is unsupported 2018-11-27 11:03:59 -06:00
Yuce Tekol
d543689868
updated translate keys test to include new keys 2018-11-26 17:52:53 +03:00
Yuce Tekol
eb9e609794
Merged with master 2018-11-26 17:03:38 +03:00
Ben Johnson
8e49332b25 Add distributed tracing. 2018-11-21 15:08:33 -06:00
Yuce Tekol
81354461a0
Merge branch 'master' into 1710-suppress-std-view-on-time-fields 2018-11-21 18:44:29 +03:00
Yuce Tekol
12112e0aef
Merged with master; check for empty roaring data 2018-11-21 18:22:40 +03:00
Yuce Tekol
ac91635628
Merged with master 2018-11-21 18:16:43 +03:00
Yuce Tekol
2c91bf69b3
Merge branch 'master' into translate-keys-endpoint 2018-11-21 16:38:23 +03:00
Yuce Tekol
9c05155db4
Added /internal/translate/keys endpoint 2018-11-21 16:35:50 +03:00
Yuce Tekol
1faa789b31
remove ImportRoaringRequestView type 2018-11-21 14:53:16 +03:00
Yuce Tekol
3d54f737cd
ditch OptFieldTypeTimeWithOptions 2018-11-20 23:21:09 +03:00
Matt Jaffee
77598c2cc6
dup log output onto stderr to catch panics in log file 2018-11-20 14:08:07 -06:00
Matt Jaffee
65f478470f
logging cleanup - start with lowercase unless reporting error or warning 2018-11-20 14:08:06 -06:00
Seebs
a203313143 move Logger and Stats to their own packages
I'd like to add stat tracking to Roaring, which means it
has to be able to import the stats package, which means
stats has to be a package rather than part of the pilosa
package. If stats stops being in pilosa, it still needs
a way to import logger, so logger also has to leave the
pilosa package. Then everything using them needs to import
them and use package selectors on their names.

This doesn't actually add the stats support to roaring,
it just makes it so there's a way to import the stats
code from something in the roaring package.
2018-11-15 15:10:44 -06:00
Yuce Tekol
0fe5d56f68
Merged with master 2018-11-15 21:19:58 +03:00
Matt Jaffee
90c5f64b19
filter memberlist debug and info logs, use t.Log instead of fmt in cluster tests 2018-11-12 14:01:57 -06:00
Yuce Tekol
1642e22872
fixed handler tests 2018-11-12 19:24:32 +03:00
Yuce Tekol
84c04900e1
Import roaring enpoint accepts a list of views 2018-11-12 18:43:59 +03:00
Matt Jaffee
deae8ce7c0
improvements to clustertests and fix cluster pause bug by state sharing 2018-11-09 11:28:19 -06:00
Yuce Tekol
7913a419ae
adds NoStandardView field option. Fixes #1710 2018-11-08 18:33:39 +03:00
Matt Jaffee
ec3982c0ff
pull out advertise URI changes 2018-11-02 16:10:39 -05:00
Matt Jaffee
d86e9ed3ea
add docker based cluster tests, remove proxy stuff, fix advertise 2018-11-02 14:41:42 -05:00
Matt Jaffee
62c1bfe90e
implement MustNewClusterWithProxy and drop/undrop for partitioning 2018-10-26 16:13:20 -05:00
Matt Jaffee
28591b4a91
WIP: figure out why config.Gossip.Port is 0 after cluster start 2018-10-24 16:56:32 -05:00
Matt Jaffee
8db7a78a93
unpushed WIP on cluster-tests 2018-10-23 11:10:24 -05:00
Travis Turner
42531f72fb
prevent closing os.Stdout or os.Stderr (used in verbose test logging) 2018-10-17 17:59:57 -05:00
Yuce Tekol
066c3a4853
Fixes #1660 2018-10-01 14:15:51 +03:00
Travis Turner
cdfcd6db5e
store remoteAvailableShards to file 2018-09-24 15:37:43 -05:00
Cody Soyland
e4a1281997 Merge branch 'master' into configurable-translate-map-size 2018-09-20 16:05:43 -05:00
Cody Soyland
52ba336461 Address code review feedback 2018-09-20 16:05:28 -05:00
Travis Turner
b86478c613
test to ensure views match UTC time 2018-09-19 16:17:09 -05:00
Cody Soyland
ebc4d7fc13 Set low map size for 32-bit 2018-09-19 13:37:42 -05:00
Cody Soyland
2394d36108 Adjust tests, fix 32-bit config 2018-09-19 13:30:22 -05:00
Cody Soyland
7a32745b28 Make translate map size configurable. 2018-09-19 12:35:00 -05:00
Matt Jaffee
7b0d4d75b4
remove validators on import-roaring and test handler
validators are for query args, not url vars. Also some misc cleanup and error
handling in the handler.
2018-09-13 13:14:35 -05:00
Matt Jaffee
a3243f99e1
linter fixes - reorder return vals, remove unused const 2018-09-12 13:24:59 -05:00
Travis Turner
d50a0e853c
change NewTranslateStore() to take an interface for backward compatibility 2018-08-13 12:04:22 -05:00