We execute the aggregate Distinct calls after the GroupBy is complete,
and we need these to act like non-remote calls in that they forward to
all nodes, but like remote calls in that they bypass key
translation. Added a "PreTranslated" flag to the QueryRequest to
achieve this.
Discovered an issue where a nil *Row in EmbeddedData would cause a
panic in the protobuf serialization. Changed the encoding code we
control to never pass a nil *Row.
Got fed up with lack of context on errors and added wrapping to all
calls under executor.executeCall as well as a few other places.
Handled a situation where not having data on a shard for a particular
field could cause a query to error instead of just treating that
fragment as being empty. (see the switch in executeDistinctShardSet)
Stopped GroupBy from executing the Count(Distinct) aggregate on Remote
calls.
Fixed a longstanding issue where errors retrieved from remote query
calls had a garbage character at the front due to treating a protobuf
payload as an error message instead of decoding it. (see
http/client.go)
moved lonquerytime from cluster into server and moved cluster.longquerytime into top level config
kept cluster.longquerytime for backwards compatibility, favored if both longquerytime options are present
- the -fix flag repairs replication errors by copying from the primary.
- the -fixkeys flag repairs any string key translation issues.
- make pilosa-fsck installs pilosa-fsck and builds release-pilosa-fsck.COMMIT.GOOS.tar.gz release tarbar
At some point the cluster code was modified to do 120 tries to confirm
if a node was down which is a bit excessive for production. My
understanding is that this was done to help trigger or fix a problem
during testing which is hopefully no longer relevant.
Also changed the tournament script to use what I think are more
standard bash-isms that work on mac. Please confirm this still works
on Linux as well.
- blue_green for doing migration. Called before Holder.Open finishes.
- holdbkg.go added for index lookup. Less wedging between a deadlock and a race.
- fix fault under read-only map under lmdb at
TestExecutor_Execute_Row_Range/RowIDColumnID by doing cow in roaring.
- roaring -tags gofuzz builds again
- roaringparanoia build tag added to make test targets in Makefile
- add rbf.NewDBWithAllocZero for out-of-bounds memory checks
- .circleci/config.yml test-shardwidth-22 with large run container, kept OOM-ing we suspect.
Fixes#819
- introduce Query Context (Qcx) for managing database-per-shard.
- replaces the MultiTx, so mtx.go is retired and removed.
- introduces the HolderConfig struct and all Holders now have
a path from birth.
- rbf speedups on bitwise writes
- badgerdb is removed due to unresolvable write conflicts.
fixes#703#676
a) All tests green under -race for both PILOSA_TXSRC=roaring and PILOSA_TXSRC=badger.
b) Distinct is merged back into mainline pilosa.
Seebs notes on the Distinct work:
merge Distinct plugin back into main source tree, convert to Tx
We drop all references to the Preemptively Deprecated Don't You Dare
Use This extension interface, and move the one and only extension we had
(Distinct) into the main executor.
Also this fixes an arguable bug, which is that Container.AsBitmap()
would panic on a nil parameter, but it should have returned an empty
bitmap, because a nil *Ccontainer is a valid empty container. This
simplifies logic significantly in Distinct.
Fixes#569#570#571#572#573#584#585
The new logic to send resize instructions more makes it easier
to hit this, but it's probably always been a theoretically possible
bug to hit: If you are shutting a cluster down, then you stop accepting
connections, which means that if you have an existing resize job, you
can't get responses for it. Which means that the other nodes will
fail to notify you of the success or failure of resize instructions,
so the code waiting on the resize job's status waits forever.
When closing, we bail immediately on that; we don't need to wait for
those notifications. We still have a buffer, and a reasonable confidence
that we'll never write more than one result status, so if one of them
*does* somehow show up and cause the job to have a status,
writing the status won't block.
Some cluster tests failed sporadically. In order to fix them, I
introduced some debugging-related functionality, which revealed
several new bugs that were actually existing bugs we just happened
not to hit in testing. This combines various fixes.
We start with "make the nodes used in testing have distinct names
based on the test case name", which lets us discover that we are
leaking clusters, which continue to sit around talking with each
other. That in turn causes significantly higher load on access to
ephemeral ports, which causes sporadic failures when we shut a
node down and try to restart it, but something else has gotten assigned
its ephemeral port number since then.
Part of the fix is to try to rebind on port 0 if an attempt to
bind to a specified port over 32k fails. This is a guess; the
actual ephemeral port range could be 16k+, 32k+, or 48k+, or just
about anything else really, but it seems reasonable in
practice.
There were bugs in the oft-repeated loops to await the cluster
achieving a given state, and it could hang forever if it didn't,
so we add a timeout and a standard function on the test.Cluster
type to handle that. Note that the timeout seems irrelevant; in
every case I've tried, a timeout of 0 is fine because the node
start doesn't complete until the cluster state has changed.
Add a method to test.Command to run a query, expecting a specific
result. Also clean up some of the formatting and generation of
queries, and allow parameterized (badly) queries. This lets us fix
a subtle bug, which is that test cases were depending on assumptions
about shardwidths. Also improve the diagnostic output from some of
these functions so test failures are more comprehensible.
But actually that dependency on shardwidths was ALSO revealing a
genuine underlying bug, which is that a node resize did not correctly
propagate the schema to a new node if there was no data present
on shards that node would own. We now also have a test case that
hits that (or would, if we hadn't fixed it).
Add comments explaining the server options parameters for MustNewCluster
and MustRunCluster.
Also, we implement the ReadFrom and WriteTo behaviors for
InMemTranslateStore, without which some of the cluster resize tests
fail. Props to the comment for specifically stating that they wouldn't
work if that happened, which probably saved me several hours of
debugging. The implementations may not be robust, but
InMemTranslateStore is intended to be used only in lightweight
and transient testing.
There's no reason to have 10-20 seconds of delays for testing this,
because in testing, we're running things on the local machine and don't
need to worry about significant network lag. Make retry count and delay
settable options, and set them lower. Moves the Replica2 test in
server/server_test.go from ~21s to ~2s.
I think this will improve the transaction response messages Kuba
mentioned where it was an empty transaction instead of a nil or not
there... if not it should make it easier to do that anyhow.
also adds a "noSleep" option to the server command to avoid the 5
second sleep we introduced on startup for non-coordinator cluster
nodes. The sleep doesn't seem to be needed in the tests and makes them
much slower.
For tests, we need to create the grpc listener with port 0 in order to
automatically assign a port. This PR moves the lister creation outside
of the grcpServer itself so that we can access that auto-created port.
This commit adds `TranslationSources` to the cluster
`ResizeInstruction`. These are the sources of translation
partitions which the receiving node needs in order to support
partition distribution in the new, resized cluster.
This also fixes a bug where index options were not being
encode in the proto Index object. That meant that the schema
transferred via protobuf was not correct. The reason why
things normally worked is because index creation typically
happens on the CreateIndex message, which does include the
options.
TODO:
- [ ] implement the TranslateStore interface for `InMemTranslateStore`
and `mock.TranslateStore`
- [ ] surely need some more tests around the `ReadFrom` and `WriteTo`
