Commit graph

8997 commits

Author SHA1 Message Date
Antonio Navarro Perez
d20b831084 Add port wrapper POC
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
2021-01-13 20:14:33 +01:00
Kuba Podgórski
61edff3eee Apply closed channel fix
45619a5b7b
2021-01-13 15:37:13 +01:00
Kuba Podgórski
4afb0ecc51 Close TCP listeneer on port mapper 2021-01-13 13:53:27 +01:00
Travis
877af6dad9
replace a MustNewCluster with MustRunCluster 2021-01-12 23:30:10 -06:00
Travis
ef8d0759d5
add retry to pg test ServerTLS() 2021-01-12 23:04:49 -06:00
Travis
f2234929d8
add retry to MustRunCluster 2021-01-12 22:09:09 -06:00
Travis
bc13834343
disco/etcd work: fix lots of races, start all cluster nodes at once.
port mapper gives out ports from 63000-65000 for the tests

fix another race

http test uses port.MustGetPort

rbf: remove :0 port request

ocd happy

test fix for grpc listener address already in use

test/disco allocates BindGRPC port from the port mapper

dump stack on each GetPort

verify each port is usable right away

server/config.go has Config.Validate() now

panic if gossip port is 0. validate server.Config

fix another gossip port 0

builds

quiet, don't dump stack on each port alloc

builds

happy linter

even gossip fallback should not be zero but rather use the port mapper
2021-01-12 21:06:12 -06:00
Ben Johnson
fbf546f131
Merge pull request #1319 from molecula/pilosa-bench 2021-01-12 16:51:27 -07:00
Ben Johnson
b7973e2612 Add read benchmarks 2021-01-12 13:48:09 -07:00
Matthew Jaffee
701d6448cc
Merge pull request #1307 from jaffee/1292-panic-groupby-int-distinct-aggregate
Fix potential panic when grouping on int field with Distinct aggregate
2021-01-08 16:38:00 -06:00
Matt Jaffee
48552553dc
guard against NPE when setting precomputed data
If a precomputed call returns a nil Row result somehow, that could
cause a nil pointer exception when handling the result in
handlePreCall.

In this particular case, A Distinct call on a BSI field with a filter
which returned no results could return a SignedRow{} with nil *Rows
inside of it. This only manifested if there was data in a single shard
as otherwise the reduce logic created a SignedRow with empty *Row
objects rather than nil ones. Isn't that fun?

Extra fun: the reason the filter was returning no results was not
because it was actually empty, but because of another bug where
constructing the Distinct calls to compute the aggregate of a GroupBy
doesn't take into account that the group might include an integer
field which means that the call needs to be constructed
differently. That bug is not fixed in this commit, hence the tests are
still failing, but not panicking.
2021-01-08 15:03:49 -06:00
Matthew Jaffee
616ba408f8
Merge pull request #1308 from codysoyland/groupby-distinct-int
Fix GroupBy Distinct aggregate on int field
2021-01-08 13:55:38 -06:00
Cody Soyland
83ce30f1c4
Fix GroupBy Distinct aggregate on int field 2021-01-08 13:35:58 -06:00
Ben Johnson
55fad52c59
Merge pull request #1306 from molecula/rbf-remove-cursor-arena
Remove RBF cursor arena.
2021-01-08 10:13:52 -07:00
Ben Johnson
489f33a173 Remove RBF cursor arena.
Previously there were two implementations of cursor reuse: sync pool &
an arena. This commit removes the arena in favor of the global pool.
2021-01-08 07:59:10 -07:00
Ben Johnson
1865de8248
Merge pull request #1301 from molecula/fix-rbf-cursor-close 2021-01-07 19:22:34 -07:00
jaten-molecula
b3854e867b
Merge branch 'master' into fix-rbf-cursor-close 2021-01-07 20:14:59 -06:00
Jason E. Aten
ecada682ae cluster_internal_tests use getport 2021-01-07 22:30:04 +00:00
jaten-molecula
0c1f2f06ba
Merge pull request #1304 from jaten-molecula/portmap
GlobalPortMapper avoids many races in port allocation for cluster setup
2021-01-07 16:24:02 -06:00
Jason E. Aten
1aabcb3d14 GlobalPortMapper avoids many races in port allocation for cluster setup 2021-01-07 22:20:49 +00:00
Cody Soyland
476767f150
Merge pull request #1285 from codysoyland/grpc-crd
Add create, read, and delete index methods to gRPC interface
2021-01-07 15:24:13 -06:00
Cody Soyland
4f57b4d07b
Remove TrackExistence configuration from gRPC CreateIndex 2021-01-07 15:19:08 -06:00
Cody Soyland
d8ebfda1bd
Undo error cause changes due to broken logic in other places, check for ConflictError explicitly 2021-01-07 14:49:18 -06:00
Travis Turner
4eb36a34f6
Merge pull request #1294 from travisturner/disco-cleanup
Disco cleanup
2021-01-07 14:08:55 -06:00
Cody Soyland
219714a18e
Fix linter problems 2021-01-07 13:50:59 -06:00
Travis
8dbfae1d86
temporarily have cluster implement Noder 2021-01-07 13:45:46 -06:00
Cody Soyland
daa8c9bd8b
Add tests for new gRPC create/get/delete calls 2021-01-07 13:05:48 -06:00
Cody Soyland
fdf5818fc2
Add create, read, and delete index methods to gRPC interface 2021-01-07 13:05:48 -06:00
Ben Johnson
93f06e0f9d Fix rbf.Cursor.Close() panic 2021-01-07 11:28:46 -07:00
Maxton Huff
17e010cca2
Merge pull request #1293 from Maxtonian/verbose-message
Fix formatting bug in verbose log message
2021-01-07 09:38:29 -06:00
Travis
134abda51b
Implement snap := ClusterSnapshot()
Below is the list of instance of `ClusterSnapshot()` in the latest
`with-etcd` code. Some of these may not yet exist in the `disco` branch,
but this commit is implementing any that currently apply.

==========================
Done:
==========================
index.go
930:	snap := NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)
1072:	snap := NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)

cmd/pilosa-fsck/fsck.go
786:	snap := pilosa.NewClusterSnapshot(cfg.topo, cfg.topo.Hasher, cfg.topo.ReplicaN)

boltdb/translate.go
558:	snap := pilosa.NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)
1264:	snap := pilosa.NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)

fragment.go
3448:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
3568:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
3620:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)

==========================
Remaining:
==========================

cluster.go
371:	snap := NewClusterSnapshot(NewLocalNoder(nodes), c.Hasher, c.ReplicaN)
474:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
639:	fSnap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
640:	toSnap := NewClusterSnapshot(to.noder, c.Hasher, to.ReplicaN)
703:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1475:		snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1502:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1941:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1986:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
2049:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
2126:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)

api.go
475:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
604:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
690:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
1684:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
1946:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)

executor.go
3781:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4157:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4200:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4243:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4517:	snap := NewClusterSnapshot(NewLocalNoder(e.Cluster.Nodes()), e.Cluster.Hasher, e.Cluster.ReplicaN)

holder.go
1465:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1668:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1889:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1963:	snap := NewClusterSnapshot(c.Cluster.noder, c.Cluster.Hasher, c.Cluster.ReplicaN)
2021-01-06 22:53:20 -06:00
Travis
3f26d667b4
remove pilosa.DefaultPartitionN 2021-01-06 22:45:44 -06:00
jaten-molecula
a6bf52d753
Merge pull request #1290 from travisturner/disco-subpackages
add subpackages: topology, net
2021-01-06 17:23:56 -06:00
Jason E. Aten
6a845f1de1 use bbolt v1.3.5 that has fixed the checkptr bugs 2021-01-06 23:19:53 +00:00
Travis
20816ffa20
remove pilosa.URI 2021-01-06 16:19:14 -06:00
Travis
4515a24e48
change all references to use subpackages: topology, net 2021-01-06 16:09:24 -06:00
Maxton Huff
d10e45648b change f.path to f.path() 2021-01-06 15:58:01 -06:00
Travis
f8e6115c0e
fix linter and go.mod issues 2021-01-06 15:01:50 -06:00
Travis
2c1a019c9e
add licence headers 2021-01-06 14:42:28 -06:00
Travis
bd989f464a
change bbolt version back to 1.3.3 2021-01-06 13:26:35 -06:00
Travis
0da35fb72b
add subpackages: topology, net 2021-01-06 13:21:36 -06:00
Matthew Jaffee
d6cba17a01
Merge pull request #1274 from jaffee/generalized-groupby-sort-2
Add ability to sort on count or aggregate in GroupBy. Fix bug with offset being unsupported. Fix bugs with limit interacting poorly with other arguments.
2021-01-05 09:41:18 -07:00
Matt Jaffee
216e28a77e
add getSorter tests, fix bugs 2021-01-05 09:54:46 -06:00
Matt Jaffee
6dccb3d6be
remove (unused) sorting code related to fields, add comments 2021-01-03 08:34:53 -06:00
Matt Jaffee
6094663e7a
fix bug with "having" and "limit" in GroupBy
the limit could get applied before "having" in some cases which could
result in results being discarded which met the having condition while
results were kept which did not, ultimately resulting in GroupBy
falsely reporting fewer results than actually existed.
2021-01-01 21:56:59 -06:00
Matt Jaffee
ea539d8241
simplify groupby sorting and fix bugs
Back out support for sorting on fields (only count and aggregate
supported for now).

Fix bug where default return of "true" caused sort to be unstable. (If
they are equal, Less should return false)

Fix bug where limit was being applied before sorting.

Fix bug where offset was not actually allowed to be an argument to
GroupBy (weird! guess we weren't testing that very well)

Apply "having" after calculating Count(Distinct) aggregate so that
having can apply to that.

Switch to stable sort to make testing easier.
2020-12-31 14:42:03 -06:00
Matt Jaffee
5fdae74812
draft of sorting groupby results 2020-12-31 14:41:52 -06:00
Matthew Jaffee
46bbff786e
Merge pull request #1245 from codysoyland/groupby-aggregate-distinct
Add Distinct call as GroupBy aggregate
2020-12-30 10:05:37 -07:00
Matt Jaffee
16fd6a7edd
add tests for GroupBy(Distinct), fix various problems
We execute the aggregate Distinct calls after the GroupBy is complete,
and we need these to act like non-remote calls in that they forward to
all nodes, but like remote calls in that they bypass key
translation. Added a "PreTranslated" flag to the QueryRequest to
achieve this.

Discovered an issue where a nil *Row in EmbeddedData would cause a
panic in the protobuf serialization. Changed the encoding code we
control to never pass a nil *Row.

Got fed up with lack of context on errors and added wrapping to all
calls under executor.executeCall as well as a few other places.

Handled a situation where not having data on a shard for a particular
field could cause a query to error instead of just treating that
fragment as being empty. (see the switch in executeDistinctShardSet)

Stopped GroupBy from executing the Count(Distinct) aggregate on Remote
calls.

Fixed a longstanding issue where errors retrieved from remote query
calls had a garbage character at the front due to treating a protobuf
payload as an error message instead of decoding it. (see
http/client.go)
2020-12-30 08:13:41 -06:00
Cody Soyland
fd7417a49b
Use shardwidth instead of hardcoded value 2020-12-30 08:12:09 -06:00