Commit graph

410 commits

Author SHA1 Message Date
Matt Jaffee
08d4511e6c
fix issue where restarting a cluster logged "invalid pilosa.Message"
we were sending pilosa.Message objects from a spool, but actually
passing a pointer to them rather than the Message itself. I'm
concerned this wasn't caught in any test, and also curious if that
needed to be a pointer for some reason or if it's a typo.

Definitely need to write a test still.
2021-03-05 12:51:39 -06:00
Nia Weiss
3fbf6993d0
defer cluster messages until startup 2021-02-25 16:01:49 -05:00
Travis
2bbe1fdde0
remove remaining references to "coordinator" 2021-02-23 17:23:09 -06:00
Antonio Navarro Perez
f0a5ca5d3a Change coordinator error to primary
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
2021-02-23 09:58:22 +01:00
Kuba Podgórski
bfc24a1745 Check state in shardsByNode once stator is implemented 2021-02-17 16:02:28 +01:00
Travis
8f0270acda
adjust openExistenceField() to check on disk first 2021-02-12 20:35:36 -06:00
Travis
7a5192aba6
handle error on LoadSchema() message 2021-02-10 12:00:36 -06:00
Travis
1c3b0f364d
implement ApplySchema, LoadSchema, and LoadSchemaMessage 2021-02-10 11:51:52 -06:00
Travis
8fe1b9a37c
store views in etcd via Schemator 2021-02-10 11:47:26 -06:00
Travis
3542134100
WIP: implement Schemator 2021-02-08 10:42:55 -06:00
Antonio Navarro Perez
b1ff8e55cd
Unify state
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
2021-02-06 16:53:42 -06:00
Travis
30d4687a99
remove type Topology 2021-02-05 16:13:43 -06:00
Kuba Podgórski
ab37bf5c7b Apply resizer interface (remove and add node) 2021-02-04 20:26:41 +01:00
Travis
afc53e1163
remove ReceiveEvent 2021-02-03 23:31:38 -06:00
Travis
652014539c
remove temporary Gossiper interface 2021-02-03 23:06:51 -06:00
Travis
6e4ea21ce5
remove gossip listenForJoins 2021-02-03 22:34:45 -06:00
Travis
f9661b7b81
add PrimaryNodeID() method to Noder interface 2021-02-03 15:11:24 -06:00
Kuba Podgórski
6601835ba1 Remove public mutex from Node 2021-02-03 19:37:54 +01:00
Kuba Podgórski
6826997852 Remove state member from cluster.
Remove all function SetState like. Stop broadcasting cluster state.
2021-02-03 15:16:59 +01:00
Antonio Navarro Perez
c45e21640c
Change coordinator to primary
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
2021-02-02 15:56:12 -06:00
Travis
91c0df29a1
remove disco debugging printlns 2021-02-01 22:28:43 -06:00
Travis
26176c15eb
fix linter issues (wrap all ClusterStates in string() until we update the type) 2021-02-01 16:57:35 -06:00
Travis
855e1b35f5
more use of noder; remove c.nodes
disable some of the gossip logic

implement some of the stator logic
2021-01-31 23:42:49 -06:00
Travis
97eaff5c82
use Etcd Noder; actually use EtcdWithCache 2021-01-31 23:42:44 -06:00
Travis
7ed1417893
set node metadata in server.Open() 2021-01-30 09:12:07 -06:00
Travis
457194f6a8
update config to support etcd arguments 2021-01-29 19:43:48 -06:00
Kuba Podgórski
cef6925e7b Fix server tests 2021-01-28 17:39:58 +01:00
Travis
a196e1e74c
use etcd for node.ID
this commit adds a temporation interface for starting gossip.
we needed this so we can start gossip AFTER setting up the node,
but before waitingForJoins.
2021-01-23 19:52:58 -06:00
Travis
08fae2be4c
introduce storage.Config 2021-01-20 22:05:38 -06:00
Travis
1d55e671a2
go mod tidy and linter
fix race

cleanup
2021-01-15 17:48:44 -06:00
Travis
684b20edb3
disco open/close debugging 2021-01-14 23:02:58 -06:00
Kuba Podgórski
36f17eee1d global mutex on GetPorts 2021-01-14 18:04:16 +01:00
Kuba Podgórski
a100a38b4a don't close disco on Server.Close 2021-01-14 17:47:06 +01:00
Kuba Podgórski
0e4a7a29fb close disco before holder 2021-01-14 17:35:30 +01:00
Travis
bc13834343
disco/etcd work: fix lots of races, start all cluster nodes at once.
port mapper gives out ports from 63000-65000 for the tests

fix another race

http test uses port.MustGetPort

rbf: remove :0 port request

ocd happy

test fix for grpc listener address already in use

test/disco allocates BindGRPC port from the port mapper

dump stack on each GetPort

verify each port is usable right away

server/config.go has Config.Validate() now

panic if gossip port is 0. validate server.Config

fix another gossip port 0

builds

quiet, don't dump stack on each port alloc

builds

happy linter

even gossip fallback should not be zero but rather use the port mapper
2021-01-12 21:06:12 -06:00
Travis
4515a24e48
change all references to use subpackages: topology, net 2021-01-06 16:09:24 -06:00
Matt Jaffee
a3b07ff519
re-add log line which has more utility than I thought
From Nia:
While debugging the Q2 bugs this was somewhat useful in analyzing cluster events. As for the spammy part. . . that seems to be more of an issue with spamming our resets than an issue with the log itself.
2020-12-28 15:30:38 -06:00
Matt Jaffee
446950979a
fix potential nil dereference in SignedRow.ToRows
This used to be possible to hit, but I think now that Distinct on a
set field returns a *Row rather than a SignedRow it isn't an issue. (I
wasn't able to trigger it in the tests). Adding the fix anyway as it
seems safer than not.

The rest of the changes are test infrastructure to make it easy to
call GRPC queries and verify the results as CSV.
2020-12-28 11:00:46 -06:00
Matt Jaffee
1372bafe02
fix bugs where row index and field weren't always being propagated
I used a "paranoia" check to find these, but then realized the check
had a ton of false positives and doing it properly wasn't going to be
straightforward. I'm leaving the paranoia stuff in unless there are
objections, because I've wanted it before and not had it.

I also removed a log line that is very verbose and I don't think helps
anyone.
2020-12-23 19:08:10 -06:00
Nia Weiss
aaf94c6aeb
fix the automatic snapshot queue enable check
The "needs snapshot queue" check was broken, as it only checked inside a loop over indices.
If there are no indexes yet (or more likely if the indices have not yet been loaded off of disk), then this would never use the snapshot queue on roaring.
2020-12-16 08:15:17 -05:00
Maxton Huff
dee700741a deprecate cluster.long-query-time and create long-query-time
moved lonquerytime from cluster into server and moved cluster.longquerytime into top level config
kept cluster.longquerytime for backwards compatibility, favored if both longquerytime options are present
2020-12-10 10:51:31 -06:00
Jason E. Aten
03a54c6d5f rowcache off by default. pilosa server --rowcache-on turns it back on. 2020-12-08 23:18:45 +00:00
Nia
2a863e5c13
clarify OptServerOpenIDAllocator
Co-authored-by: Matthew Jaffee <matthew.jaffee@gmail.com>
2020-12-07 08:08:51 -05:00
Nia Weiss
7ea2e4b7e3
add ID auto-generation on the coordinator 2020-12-03 13:04:14 -05:00
Alan Bernstein
2bc2a32263 Make query history length configurable 2020-11-23 20:23:27 -06:00
Jason E. Aten
509348260f better migration logging 2020-10-28 22:54:58 +00:00
Jason E. Aten
957cba1768 performance tuning: rbfcfg package, binary search for wal segment
- rbfcfg package holds Config for --rbf- command line flags
- wal.go: replace linear search with bisection for wal segment
2020-10-27 00:49:56 +00:00
Jason E. Aten
2503fe5b66 pilosa: avoid re-scanning shards during Holder.Open()
- view.openFragmentInTx was forcing a directory scan
   for shards on every open fragment during Holder.Open().
   Seen by pprof profile having excessive allocations
   from dbshard.go listDirUnderDir().
2020-10-23 23:50:22 +00:00
Jason E. Aten
8d2ad048fa pilosa server --norowcache disables the row cache
- this can lessen memory pressure
- certain backends may not need it
- enables performance benchmarking and tuning
2020-10-22 22:20:48 +00:00
Jason E. Aten
fe425a84c0 pilosa-fsck: scan and repair of pilosa backups
- the -fix flag repairs replication errors by copying from the primary.
 - the -fixkeys flag repairs any string key translation issues.
 - make pilosa-fsck installs pilosa-fsck and builds release-pilosa-fsck.COMMIT.GOOS.tar.gz release tarbar
2020-10-02 16:47:56 -05:00