scriptoriumadmin/featurebase
1
0
Fork 0
mirror of https://github.com/featurebasedb/featurebase.git synced 2026-09-05 16:15:56 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
6644 commits 45 branches 174 tags 85 MiB
Commit graph

6644 commits

This branch
This branch
All branches
Author SHA1 Message Date
Kuba Podgórski
a4f9aee28e Set etcd log level to error 2021-01-15 14:11:48 +01:00
Kuba Podgórski
6cd8f6a970 Less TestMain_Set_Quick parallel tests 2021-01-15 13:24:41 +01:00
Kuba Podgórski
17b1eeb0d0 Increase timeout (30s) for cluster NORMAL state 2021-01-15 12:55:03 +01:00
Kuba Podgórski
64425736b0 Create disco dir outside pilosa 2021-01-15 12:30:57 +01:00
Travis
684b20edb3
disco open/close debugging 2021-01-14 23:02:58 -06:00
Kuba Podgórski
4afaaa5e25 remove zap 2021-01-14 20:59:17 +01:00
Kuba Podgórski
9a0facaaa3 set even more ports for Node Command 2021-01-14 20:20:23 +01:00
Kuba Podgórski
201e851511 set more ports for Node Command 2021-01-14 20:04:09 +01:00
Kuba Podgórski
33c4c77495 revert bind port to 0 2021-01-14 18:33:22 +01:00
Kuba Podgórski
36f17eee1d global mutex on GetPorts 2021-01-14 18:04:16 +01:00
Kuba Podgórski
a100a38b4a don't close disco on Server.Close 2021-01-14 17:47:06 +01:00
Kuba Podgórski
0e4a7a29fb close disco before holder 2021-01-14 17:35:30 +01:00
Kuba Podgórski
2713be9329 check error as string 2021-01-14 17:12:52 +01:00
Kuba Podgórski
1f6ed4feca bangbang theory 2021-01-14 16:25:44 +01:00
Antonio Navarro Perez
1a5ab4b155 Fix some more problems 
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
 2021-01-14 13:10:51 +01:00
Antonio Navarro Perez
052aadb3b0 Wrap some missing constructors using ports. 
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
 2021-01-14 10:48:46 +01:00
Travis
27614c42f7
Finish implementing port wrapper 2021-01-13 22:56:54 -06:00
Antonio Navarro Perez
d20b831084 Add port wrapper POC 
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
 2021-01-13 20:14:33 +01:00
Kuba Podgórski
61edff3eee Apply closed channel fix 
45619a5b7b
 2021-01-13 15:37:13 +01:00
Kuba Podgórski
4afb0ecc51 Close TCP listeneer on port mapper 2021-01-13 13:53:27 +01:00
Travis
877af6dad9
replace a MustNewCluster with MustRunCluster 2021-01-12 23:30:10 -06:00
Travis
ef8d0759d5
add retry to pg test ServerTLS() 2021-01-12 23:04:49 -06:00
Travis
f2234929d8
add retry to MustRunCluster 2021-01-12 22:09:09 -06:00
Travis
bc13834343
disco/etcd work: fix lots of races, start all cluster nodes at once. 
port mapper gives out ports from 63000-65000 for the tests

fix another race

http test uses port.MustGetPort

rbf: remove :0 port request

ocd happy

test fix for grpc listener address already in use

test/disco allocates BindGRPC port from the port mapper

dump stack on each GetPort

verify each port is usable right away

server/config.go has Config.Validate() now

panic if gossip port is 0. validate server.Config

fix another gossip port 0

builds

quiet, don't dump stack on each port alloc

builds

happy linter

even gossip fallback should not be zero but rather use the port mapper
 2021-01-12 21:06:12 -06:00
Jason E. Aten
ecada682ae cluster_internal_tests use getport 2021-01-07 22:30:04 +00:00
jaten-molecula
0c1f2f06ba
Merge pull request #1304 from jaten-molecula/portmap 
GlobalPortMapper avoids many races in port allocation for cluster setup
 2021-01-07 16:24:02 -06:00
Jason E. Aten
1aabcb3d14 GlobalPortMapper avoids many races in port allocation for cluster setup 2021-01-07 22:20:49 +00:00
Travis Turner
4eb36a34f6
Merge pull request #1294 from travisturner/disco-cleanup 
Disco cleanup
 2021-01-07 14:08:55 -06:00
Travis
8dbfae1d86
temporarily have cluster implement Noder 2021-01-07 13:45:46 -06:00
Travis
134abda51b
Implement snap := ClusterSnapshot() 
Below is the list of instance of `ClusterSnapshot()` in the latest
`with-etcd` code. Some of these may not yet exist in the `disco` branch,
but this commit is implementing any that currently apply.

==========================
Done:
==========================
index.go
930:	snap := NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)
1072:	snap := NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)

cmd/pilosa-fsck/fsck.go
786:	snap := pilosa.NewClusterSnapshot(cfg.topo, cfg.topo.Hasher, cfg.topo.ReplicaN)

boltdb/translate.go
558:	snap := pilosa.NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)
1264:	snap := pilosa.NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)

fragment.go
3448:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
3568:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
3620:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)

==========================
Remaining:
==========================

cluster.go
371:	snap := NewClusterSnapshot(NewLocalNoder(nodes), c.Hasher, c.ReplicaN)
474:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
639:	fSnap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
640:	toSnap := NewClusterSnapshot(to.noder, c.Hasher, to.ReplicaN)
703:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1475:		snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1502:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1941:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1986:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
2049:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
2126:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)

api.go
475:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
604:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
690:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
1684:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
1946:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)

executor.go
3781:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4157:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4200:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4243:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4517:	snap := NewClusterSnapshot(NewLocalNoder(e.Cluster.Nodes()), e.Cluster.Hasher, e.Cluster.ReplicaN)

holder.go
1465:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1668:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1889:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1963:	snap := NewClusterSnapshot(c.Cluster.noder, c.Cluster.Hasher, c.Cluster.ReplicaN)
 2021-01-06 22:53:20 -06:00
Travis
3f26d667b4
remove pilosa.DefaultPartitionN 2021-01-06 22:45:44 -06:00
jaten-molecula
a6bf52d753
Merge pull request #1290 from travisturner/disco-subpackages 
add subpackages: topology, net
 2021-01-06 17:23:56 -06:00
Jason E. Aten
6a845f1de1 use bbolt v1.3.5 that has fixed the checkptr bugs 2021-01-06 23:19:53 +00:00
Travis
20816ffa20
remove pilosa.URI 2021-01-06 16:19:14 -06:00
Travis
4515a24e48
change all references to use subpackages: topology, net 2021-01-06 16:09:24 -06:00
Travis
f8e6115c0e
fix linter and go.mod issues 2021-01-06 15:01:50 -06:00
Travis
2c1a019c9e
add licence headers 2021-01-06 14:42:28 -06:00
Travis
bd989f464a
change bbolt version back to 1.3.3 2021-01-06 13:26:35 -06:00
Travis
0da35fb72b
add subpackages: topology, net 2021-01-06 13:21:36 -06:00
Matthew Jaffee
d6cba17a01
Merge pull request #1274 from jaffee/generalized-groupby-sort-2 
Add ability to sort on count or aggregate in GroupBy. Fix bug with offset being unsupported. Fix bugs with limit interacting poorly with other arguments.
 2021-01-05 09:41:18 -07:00
Matt Jaffee
216e28a77e
add getSorter tests, fix bugs 2021-01-05 09:54:46 -06:00
Matt Jaffee
6dccb3d6be
remove (unused) sorting code related to fields, add comments 2021-01-03 08:34:53 -06:00
Matt Jaffee
6094663e7a
fix bug with "having" and "limit" in GroupBy 
the limit could get applied before "having" in some cases which could
result in results being discarded which met the having condition while
results were kept which did not, ultimately resulting in GroupBy
falsely reporting fewer results than actually existed.
 2021-01-01 21:56:59 -06:00
Matt Jaffee
ea539d8241
simplify groupby sorting and fix bugs 
Back out support for sorting on fields (only count and aggregate
supported for now).

Fix bug where default return of "true" caused sort to be unstable. (If
they are equal, Less should return false)

Fix bug where limit was being applied before sorting.

Fix bug where offset was not actually allowed to be an argument to
GroupBy (weird! guess we weren't testing that very well)

Apply "having" after calculating Count(Distinct) aggregate so that
having can apply to that.

Switch to stable sort to make testing easier.
 2020-12-31 14:42:03 -06:00
Matt Jaffee
5fdae74812
draft of sorting groupby results 2020-12-31 14:41:52 -06:00
Matthew Jaffee
46bbff786e
Merge pull request #1245 from codysoyland/groupby-aggregate-distinct 
Add Distinct call as GroupBy aggregate
 2020-12-30 10:05:37 -07:00
Matt Jaffee
16fd6a7edd
add tests for GroupBy(Distinct), fix various problems 
We execute the aggregate Distinct calls after the GroupBy is complete,
and we need these to act like non-remote calls in that they forward to
all nodes, but like remote calls in that they bypass key
translation. Added a "PreTranslated" flag to the QueryRequest to
achieve this.

Discovered an issue where a nil *Row in EmbeddedData would cause a
panic in the protobuf serialization. Changed the encoding code we
control to never pass a nil *Row.

Got fed up with lack of context on errors and added wrapping to all
calls under executor.executeCall as well as a few other places.

Handled a situation where not having data on a shard for a particular
field could cause a query to error instead of just treating that
fragment as being empty. (see the switch in executeDistinctShardSet)

Stopped GroupBy from executing the Count(Distinct) aggregate on Remote
calls.

Fixed a longstanding issue where errors retrieved from remote query
calls had a garbage character at the front due to treating a protobuf
payload as an error message instead of decoding it. (see
http/client.go)
 2020-12-30 08:13:41 -06:00
Cody Soyland
fd7417a49b
Use shardwidth instead of hardcoded value 2020-12-30 08:12:09 -06:00
Cody Soyland
f099a90264
Modify aggregate distinct logic and add tests 
Use execute instead of directly using executeCount
Address code review feedback (add additional filters if provided)
Add basic tests
 2020-12-30 08:12:09 -06:00
Cody Soyland
b435e9d793
Add Distinct call as GroupBy aggregate 2020-12-30 08:12:08 -06:00