moved lonquerytime from cluster into server and moved cluster.longquerytime into top level config
kept cluster.longquerytime for backwards compatibility, favored if both longquerytime options are present
- the -fix flag repairs replication errors by copying from the primary.
- the -fixkeys flag repairs any string key translation issues.
- make pilosa-fsck installs pilosa-fsck and builds release-pilosa-fsck.COMMIT.GOOS.tar.gz release tarbar
- blue_green for doing migration. Called before Holder.Open finishes.
- holdbkg.go added for index lookup. Less wedging between a deadlock and a race.
- fix fault under read-only map under lmdb at
TestExecutor_Execute_Row_Range/RowIDColumnID by doing cow in roaring.
- roaring -tags gofuzz builds again
- roaringparanoia build tag added to make test targets in Makefile
- add rbf.NewDBWithAllocZero for out-of-bounds memory checks
- .circleci/config.yml test-shardwidth-22 with large run container, kept OOM-ing we suspect.
Fixes#819
The testhook/ package provides an easy way to set up multiple
hooks to run before/after tests are run.
The audit hooks track open and closes of storage backends,
files, indexes, and holders, for example. A tempdir wrapper
creates temporary directories which are automatically cleaned up
when the test ends. Any kind of resource creation that
should be closed at test conclusion can be tracked. We
will complain at the end of the TestMain if resources are
leaking.
Leaks under go1.13:
We use a wrapper function which is a no-op for go 1.13, but actually
calls testing.TB.Cleanup in go1.14, so we can still build with 1.13 even though
tests will leak files all over the place there. Because of this,
don't run the testhook tests when using 1.13, as they'll always fail.
- the test/pilosa.go http client now times out after 10 seconds
to help diagnose hung server situations.
- Makefile targets added to get better progress reports.
1. Tests can choose the Tx engine desired by setting the PILOSA_TXSRC
env variable. For example:
PILOSA_TXSRC=badger go test -v -run TestImportClearRestart
2. pilosa server --tx is enabled now.
Examples:
pilosa server --tx roaring # gives the legacy approach.
pilosa server --tx rbf # will activate RBF
pilosa server --tx badger # will activate BadgerDB
pilosa server --tx badger_rbf # will run Blue-Green badger to RBF comparisons.
and so forth. See pilosa server -h or txfactory.go for all valid --tx choices.
3. Mechanism that makes both tests(1) and pilosa server(2) work at once:
pilosa/server/server.go injects PILOSA_TXSRC into env to
communicate with NewIndex in pilosa/index.go.
Some cluster tests failed sporadically. In order to fix them, I
introduced some debugging-related functionality, which revealed
several new bugs that were actually existing bugs we just happened
not to hit in testing. This combines various fixes.
We start with "make the nodes used in testing have distinct names
based on the test case name", which lets us discover that we are
leaking clusters, which continue to sit around talking with each
other. That in turn causes significantly higher load on access to
ephemeral ports, which causes sporadic failures when we shut a
node down and try to restart it, but something else has gotten assigned
its ephemeral port number since then.
Part of the fix is to try to rebind on port 0 if an attempt to
bind to a specified port over 32k fails. This is a guess; the
actual ephemeral port range could be 16k+, 32k+, or 48k+, or just
about anything else really, but it seems reasonable in
practice.
There were bugs in the oft-repeated loops to await the cluster
achieving a given state, and it could hang forever if it didn't,
so we add a timeout and a standard function on the test.Cluster
type to handle that. Note that the timeout seems irrelevant; in
every case I've tried, a timeout of 0 is fine because the node
start doesn't complete until the cluster state has changed.
Add a method to test.Command to run a query, expecting a specific
result. Also clean up some of the formatting and generation of
queries, and allow parameterized (badly) queries. This lets us fix
a subtle bug, which is that test cases were depending on assumptions
about shardwidths. Also improve the diagnostic output from some of
these functions so test failures are more comprehensible.
But actually that dependency on shardwidths was ALSO revealing a
genuine underlying bug, which is that a node resize did not correctly
propagate the schema to a new node if there was no data present
on shards that node would own. We now also have a test case that
hits that (or would, if we hadn't fixed it).
Add comments explaining the server options parameters for MustNewCluster
and MustRunCluster.
Also, we implement the ReadFrom and WriteTo behaviors for
InMemTranslateStore, without which some of the cluster resize tests
fail. Props to the comment for specifically stating that they wouldn't
work if that happened, which probably saved me several hours of
debugging. The implementations may not be robust, but
InMemTranslateStore is intended to be used only in lightweight
and transient testing.
also adds a "noSleep" option to the server command to avoid the 5
second sleep we introduced on startup for non-coordinator cluster
nodes. The sleep doesn't seem to be needed in the tests and makes them
much slower.
For tests, we need to create the grpc listener with port 0 in order to
automatically assign a port. This PR moves the lister creation outside
of the grcpServer itself so that we can access that auto-created port.
This is sort of prototype-ish, but the idea is that we use SetMaxMapCount
from syswrap, which already exists, to let us test edge cases like
"what happens if you only sometimes have mapped data".
add makeRows() tests
register the gRPC server
use api.Index() instead of api.Schema()
support most field types in Inspect() query
currently, there's no support for `time` fields.
those will be dependent upon the output format
and the ability to materialize the timestamp from
the time views.
this commit also changes the response type of the
`Inspect()` query to be a tabular `RowResponse`.
This change should have been adding the "GetClientCertificate"
function in server/tlsconfig.go. This is in addition to the
GetCertificate func which is only used by servers. It ended up being
much more involved for a few reasons:
1. We had no way of passing a configured HTTP client into the
translate store stuff.
2. Our cluster tests assumed http, not HTTPS, and didn't have any way
to pass the necessary configuration in.
3. I encountered what turned out to be an unrelated bug in
cmd/server_test.go which is why I moved "close(m.Started)" in
server/server.go. Basically, I was running something on port 10111
which caused the test to fail (because it was trying to bind to that),
but the failure was not immediately caught during server startup
because the m.Started channel got closed which allowed the test code
to fall through to where it called m.Close() which then got a nil
pointer exception because m.Handler had never been set up.
4. Our test code was assuming that it could create clients that
ignored the config, which meant they didn't do TLS. I added an
InternalClient() method to pilosa.Server to expose the configured
client.
