etcd runs a LOT more goroutines during server startup. Fix a
goroutine/for loop bug causing us to run four 7-node clusters
instead of 1/3/4/7-node clusters, also have the test/cluster
code reduce import workers. We can't do much about the spamminess
of the Raft stuff, but this should tone it down some.
Long story short: Once we create a server and start it, we can't start
it again. We can't close it and restart it, and we can't just start
it without closing it.
Unfortunately, if the server's config needs to change, we have a Problem
here.
This ultimately means that the retry logic for GetListeners can't actually
retry successfully; if we fail on the first attempt, we necessarily fail
on any later attempts also, and if we try to fix that, we get panics.
But!
We don't actually NEED to retry. We just need to ensure that we can
open a :0 port, extract the actual port number, and use that in places
where the port number mattered, without having to rebind it.
The only actual place we needed to rebind things was opening gRPC
servers, so we introduce a gRPC Listener that can be used instead of
trying to bind to a specified port.
In a bunch of other cases where we had similar logic to try to allocate
and then use a port, we can switch to just using a provided listener.
For instance, net/http has `Serve(net.Listener, handler)`, not just
ListenAndServe(addr, handler).
This should eliminate the weird CI failures from eaddrinuse.
NOT fixed: server/cluster_test.go/TestClusterResize_AddNode isn't working
right now. The new node isn't actually being added to the existing cluster.
I attempted this but was outsmarted by it, and I think fixing the
rest of this is worth it as a separate thing.
port mapper gives out ports from 63000-65000 for the tests
fix another race
http test uses port.MustGetPort
rbf: remove :0 port request
ocd happy
test fix for grpc listener address already in use
test/disco allocates BindGRPC port from the port mapper
dump stack on each GetPort
verify each port is usable right away
server/config.go has Config.Validate() now
panic if gossip port is 0. validate server.Config
fix another gossip port 0
builds
quiet, don't dump stack on each port alloc
builds
happy linter
even gossip fallback should not be zero but rather use the port mapper
If a precomputed call returns a nil Row result somehow, that could
cause a nil pointer exception when handling the result in
handlePreCall.
In this particular case, A Distinct call on a BSI field with a filter
which returned no results could return a SignedRow{} with nil *Rows
inside of it. This only manifested if there was data in a single shard
as otherwise the reduce logic created a SignedRow with empty *Row
objects rather than nil ones. Isn't that fun?
Extra fun: the reason the filter was returning no results was not
because it was actually empty, but because of another bug where
constructing the Distinct calls to compute the aggregate of a GroupBy
doesn't take into account that the group might include an integer
field which means that the call needs to be constructed
differently. That bug is not fixed in this commit, hence the tests are
still failing, but not panicking.
This used to be possible to hit, but I think now that Distinct on a
set field returns a *Row rather than a SignedRow it isn't an issue. (I
wasn't able to trigger it in the tests). Adding the fix anyway as it
seems safer than not.
The rest of the changes are test infrastructure to make it easy to
call GRPC queries and verify the results as CSV.
add Distinct test with integer data, and because one of the records
had a null value (and was in a shard by itself), it uncovered this
issue. I added a special error type if a view or fragment is not found
when so that we can match against it and ignore it when calculating
the results for a query.
I also added an implementation within executeCount to handle the
SignedRow case, but discovered that handlePrecalls always dumps the
negative data and that will be a bigger thing to fix
the Distinct call would get precomputed correctly, but then the
executeCount would happen in the available shards context of the
index. So if the index only had records in (e.g.) shards 10,12,18,22,
and all the values of the Distinct call were in shard 0, you'd see 0
results.
The fix skips the whole map/reduce step of executeCount (which was
basically fake anyway when the argument is precomputed), and just adds
up all counts of all the precomputed segments.
This currently won't properly count Distinct values from an int field
which contains negative numbers... going to add a test and fix for
that next.
There is also still a key translation bug which is why the one test
case is commented out... fix coming for that soon as well.
- the -fix flag repairs replication errors by copying from the primary.
- the -fixkeys flag repairs any string key translation issues.
- make pilosa-fsck installs pilosa-fsck and builds release-pilosa-fsck.COMMIT.GOOS.tar.gz release tarbar
- introduce Query Context (Qcx) for managing database-per-shard.
- replaces the MultiTx, so mtx.go is retired and removed.
- introduces the HolderConfig struct and all Holders now have
a path from birth.
- rbf speedups on bitwise writes
- badgerdb is removed due to unresolvable write conflicts.
fixes#703#676
The testhook/ package provides an easy way to set up multiple
hooks to run before/after tests are run.
The audit hooks track open and closes of storage backends,
files, indexes, and holders, for example. A tempdir wrapper
creates temporary directories which are automatically cleaned up
when the test ends. Any kind of resource creation that
should be closed at test conclusion can be tracked. We
will complain at the end of the TestMain if resources are
leaking.
Leaks under go1.13:
We use a wrapper function which is a no-op for go 1.13, but actually
calls testing.TB.Cleanup in go1.14, so we can still build with 1.13 even though
tests will leak files all over the place there. Because of this,
don't run the testhook tests when using 1.13, as they'll always fail.
- the test/pilosa.go http client now times out after 10 seconds
to help diagnose hung server situations.
- Makefile targets added to get better progress reports.
