Commit graph

459 commits

Author SHA1 Message Date
Matt Jaffee
9ad1106647
implement transaction API layer and intra-cluster messaging
also adds a "noSleep" option to the server command to avoid the 5
second sleep we introduced on startup for non-coordinator cluster
nodes. The sleep doesn't seem to be needed in the tests and makes them
much slower.
2020-04-20 13:30:48 -05:00
Cody Soyland
5e1c72e6f2 Copy reopen.FileWriter into pilosa 2020-04-20 11:58:09 -05:00
Travis
7de8399b17 move gRPC listener creation outside of grpcServer
For tests, we need to create the grpc listener with port 0 in order to
automatically assign a port. This PR moves the lister creation outside
of the grcpServer itself so that we can access that auto-created port.
2020-04-18 11:38:15 -05:00
Cody Soyland
1058440cfe Do not reuse error object (data race) 2020-04-16 17:08:50 -05:00
Cody Soyland
2155d6cab8 Move some things around, fix linter warnings. 2020-04-16 17:08:50 -05:00
Cody Soyland
a66ee26a6b Reopen log file on SIGHUP 2020-04-16 17:08:50 -05:00
Cody Soyland
08fa90e0f3 Reduce duplication of setupLogger arch-specific code 2020-04-16 17:08:50 -05:00
Travis
c45a4bf3dc ToTable and ToRows interface for gRPC 2020-04-16 14:16:54 -05:00
Alan Bernstein
71b9762501
Address review feedback again 2020-04-10 20:59:20 -05:00
Alan Bernstein
9947c92e8e
Add stats labels and slow-query log in GRPC endpoints 2020-04-10 20:59:20 -05:00
Alan Bernstein
eaf21eb19b
Add metrics for GRPC request timing 2020-04-10 20:59:19 -05:00
Alan Bernstein
8b405c226c
Add 'prometheus' option in other help text/comments 2020-04-10 20:59:18 -05:00
Alan Bernstein
84e6a25bad
Update help message 2020-04-10 20:59:17 -05:00
Todd Gruben
e7bbc3a0a8 Added delay to allow cooridinator a head start in launch on multi node clusters.
considered using Cluster.Disabled to identify if pilosa was stand alone but settled
on using an empty gossip seeds list
2020-04-09 14:20:01 -05:00
Kuba Podgórski
1217deee1c Rebase 2020-04-08 01:06:18 +02:00
Travis
0e2bb550db add deleted (rebalanced) shards to remoteAvailableShards 2020-04-05 18:37:11 -05:00
Todd Gruben
40a3dce93c Co-authored-by: Travis Turner <github@calfrope.com> 2020-04-05 18:34:43 -05:00
Travis
182c1d3c42 alter tests to allow for shardwidth22
also, reset BitDepth on field and bsiGroup during
importRoaringOverwrite
2020-04-02 17:32:49 -05:00
Travis
4d985653ae convert grpc response to use pql.Decimal 2020-04-01 23:59:50 -05:00
Travis
22cca67d6a Revert "back out the pql.Decimal changes"
This reverts commit 741ba9b268.
2020-04-01 17:46:46 -05:00
Travis
741ba9b268 back out the pql.Decimal changes 2020-04-01 11:10:33 -05:00
Cody Soyland
3a7f385a01 Add updated CircleCI config
- Remove YAML magic
- Remove a lot of duplication
- Update linter
- Use parameterized jobs and matrix build
- Update Docker Hub CD to produce versioned and "latest" images
- Add custom shard width test to workflow
2020-03-31 15:17:11 -05:00
Travis
2724ecfd5f
WIP: include translate partitions in cluster resize instructions
This commit adds `TranslationSources` to the cluster
`ResizeInstruction`. These are the sources of translation
partitions which the receiving node needs in order to support
partition distribution in the new, resized cluster.

This also fixes a bug where index options were not being
encode in the proto Index object. That meant that the schema
transferred via protobuf was not correct. The reason why
things normally worked is because index creation typically
happens on the CreateIndex message, which does include the
options.

TODO:

- [ ] implement the TranslateStore interface for `InMemTranslateStore`
and `mock.TranslateStore`
- [ ] surely need some more tests around the `ReadFrom` and `WriteTo`
2020-03-30 21:24:16 -05:00
Travis
80da129861 require scale argument for decimal fields 2020-03-30 13:38:50 -05:00
Travis
1da7cf09bb support pql.Decimal for decimal field min/max arguments 2020-03-27 16:02:47 -05:00
Cory LaNou
391fda9849
Merge branch 'enterprise' into test-helpers 2020-03-24 15:51:01 -05:00
corylanou
f84185230d
make use of t.Helper 2020-03-24 15:00:32 -05:00
Cody Soyland
1aa1ec51aa Add QueryPQLUnary gRPC call 2020-03-23 16:24:31 -05:00
Travis
cbf80370cb support fragment sync for int and decimal fields
This PR adds support for anti-entropy syncing for integer
and decimal fields. It differs from the logic for other
field types in that it does not rely on a consensus to determine
what the value should be; instead, it considers the correct
values to be those of the primary replica. From there, data
is pushed to all non-primary replicas.
2020-03-13 12:15:54 -05:00
Travis
7aea54936e remove errant println from test 2020-03-03 11:27:36 -06:00
Seebs
cb686dcad0
make mmap test experiment with different amounts of mapping
This is sort of prototype-ish, but the idea is that we use SetMaxMapCount
from syswrap, which already exists, to let us test edge cases like
"what happens if you only sometimes have mapped data".
2020-02-21 16:38:35 -06:00
Matt Jaffee
fe46c84d19
also fix Sum query, but don't convert to float until the last step
this avoids compounding floating point errors while summing up the
numbers, and means less logic needs to change. Should probably convert
min and max to use this approach as well, though they don't suffer
from the compounding error issue, it is simpler.
2020-02-21 14:11:30 -06:00
Matt Jaffee
7321f9427c
min and max should properly scale their output for decimal fields
this involved adding an optional float value to the ValCount struct
which complicated result types, necessitated grpc changes, and needed
quite a few tests at different layers.
2020-02-21 14:11:29 -06:00
Travis Turner
52d450cbaf
Merge branch 'enterprise' into time-panic 2020-02-21 10:31:24 -06:00
Travis
13a1b3a2c1 send nil time value to prevent downstream panic 2020-02-18 15:55:40 -06:00
Kuba Podgórski
f3f11f4a44
Let translate keys as empty strings (#120) 2020-02-18 12:54:24 +01:00
Ben Johnson
c8cefea897
Fix test performance 2020-02-12 10:25:26 -06:00
Travis
d9ef4c0986
use OpenInMemTranslateStore by default in tests 2020-02-12 10:25:25 -06:00
Travis
49c8bf01a0
WIP: Thread OpenTranslateStore through Holder to Index 2020-02-12 10:25:25 -06:00
Travis
e40400b130 Ensure ForeignIndex key translation happens in API.
For Fields with ForeignIndex (which have keys), the API was missing
the logic to do that translation against the translateStore of
the foreign index. This commit adds that logic, as well as some
missing translateStore-related logic in the gRPC code.
2020-01-28 22:46:12 -06:00
Travis
df51f07f96 Merge branch 'enterprise' into translation-sharding 2020-01-14 20:05:12 -06:00
Matt Jaffee
ff9ad9a4fe
mutex field data type should be string not []string 2020-01-12 17:38:43 -06:00
Travis
3d3286a9ca fix an issue caused by empty column list defaulting to IDs 2020-01-10 12:28:28 -06:00
Travis
f79fde43e3 exclude internal fields (i.e. _exists) from Inspect output 2020-01-10 12:28:28 -06:00
Travis
742135dc10 Get ForeignIndex string value when reading BSI field.
In the `Inspect` function in `server/grpc.go`, getting
the value of an `int` field with a foreign index to
an index with `Keys()`, we need to return the string
key value instead of the BSI int value for the field.

This commit also changes the method `Field.keys()` to be
exported as `Field.Keys()` so that it's accessible in
the server package.
2020-01-10 12:28:28 -06:00
Ben Johnson
2f76283f03 fix replication errors & test races 2020-01-08 09:47:43 -07:00
Ben Johnson
b3e86e8394 refactoring stores back into index/field 2020-01-08 09:47:43 -07:00
Ben Johnson
7215bfd16c Implement translator store sharding 2020-01-08 09:47:43 -07:00
Travis Turner
98df5672e9
Merge branch 'enterprise' into inspect-field-output 2019-12-27 13:26:38 -06:00
Travis
3be4141382 Return the correct data type label in grpc header
Based on the pilosa field type, return the correct data type
label in the gRPC column header.
2019-12-27 12:43:45 -06:00