stdout/stderr around
A lot of functions in the cmd and ctl packages were passing these
around and barely using them. Replaced them with a logger for most
functions. Some functions get an io.Writer instead so that their
tests can find the output they're looking for.
More cleanup on fb-1766: reworked the tests that were using io.Pipe
or os.Pipe to check their results so they now use a bytes.Buffer.
Unexported some variables that didn't need to be exported.
Fixed NewConfigCommand to use the provided stderr, not os.Stderr.
Added tests for rbf_dump, rbf_page, and keygen, since those weren't
being tested at all.
Added chksum_test, final cleanup.
(cherry picked from commit f627199acb)
Cobra automatically displays usage messages, and also a gratuitous
"Error: [...]" line in some cases, when any error at all occurs
running a command. To suppress the usage message, you have to set
cmd.SilenceUsage to true. But the code that would do this doesn't
have access to it. To address this, we introduce a category of
"usage error", implemented with stdlib error wrapping (%w) and
use errors.Is to check for it. There's also utility functions
to do this checking automatically, or indeed, to handle wrapping
of the ctl.SomethingCommand and handle running it with a suitable
context and everything.
In fact, several of the places we're checking for usage errors,
we can never actually report one, but we're checking consistently
so that if we want to report usage errors, we can.
For instance, server.Start and (dax)server.Start don't ever
return usage errors, right now, but we're checking their responses
anyway.
(cherry picked from commit c681642734)
This fixes the OriginalIP and RequestUserID in the main featurebase
package, and the Access and Refresh tokens, the UserInfo, and the
[]string of Indexes passed with context.Context(s) in the authn package.
An empty struct was used for all of these keys (and relevant helper
functions we added) to avoid allocations where possible while still
using the context functionality.
Some of the logic in the server.GetIndexes function was fixed.
(cherry picked from commit 0f5a56c958)
When doing tests, we create a ton of one-off clusters. This
turns out to be expensive and slow. Fixing it is surprisingly hard.
Fundamentally: If we're sharing clusters, we need to use different
indexes for each test, to avoid clashes. This changes index names.
As a side-effect, this reorders many partition-based things, like
the order keys are returned in. Thus, to fix this, we change a lot
of tests to no longer depend on the *order* in which strings are
returned.
Having done that, we can also discard the ModHasher behavior, since
that only existed to allow us to reliably predict partitioning.
The basic design is as follows: Instead of a cluster being a
[]*Command, a "shareable" cluster is now a []*Command plus some
flags, and a "cluster" is a pointer to a possibly-shared cluster,
plus a link to the specific test using this specific cluster,
and correspondingly, its test name suitably coerced to be a valid
index name prefix.
The "test.Cluster" object now has methods to allow retrieving an
index name, and also implemnts fmt.Formatter to let you use,
e.g., `%i` with it in Sprintf to get "the index name, plus an i".
(This works for everything but %p and %T.)
This allows us to consistently rework all the many things that
use index names in a persistent way.
We also have `MustUnshared` and `MustRunUnsharedCluster` methods
which allow us to specify that a given test needs its own cluster
for some reason. For instance, the tests that want to run backups
need their own isolated cluster, and the tests that want to close
or reopen nodes need their own cluster because a reopened cluster
won't have working GRPC for some reason.
On "closing" a shared cluster (actually the test-specific wrapper
that reflects a given sharing), we delete any indexes starting with
that test's index name prefix. Otherwise, the huge pile of open
indexes prevents `go test -race` from working on MacOS, where we
run out of address space too quickly.
This is fairly enormous but most of the individual changes are
fairly trivial things like replacing the string "i" with "c.Idx()".
We also tweaked a test that failed for me a couple of times to
not depend on sort order.
This test used to be large, because it was testing some features that
were refactored out in October of 2019. Since we no longer have the
"buffer growth" to check, let's check a much smaller file.
* removes unused filesize function
* removes ioutil usage
* updates ioutil.ReadAll to io.ReadAll
* updates ioutil.TempFile to os.CreateTemp
* updates ioutil.TempDir to os.MkdirTemp
* updates ioutil.ReadAll to os.ReadAll
* update ioutil.WriteFile to os.WriteFile
* updates ioutil.Discard to io.Discard
* updates ioutil.ReadDir to os.ReadDir where applicable
* removes unused code in idk
* creates type to use for context value keys
* replaces assert.Nil with assert.NoError for error checks
also fix a *bunch* of tests that weren't closing the clusters they
created. Cleaned up one test to use t.Run instead of just checking
everything in a loop
So with the switch to a new linter, we get a lot of new warnings,
and the majority of them are harmless probably, but a few might be
real. Variously just use _ to suppress warnings, or report errors.
There's probably things here that deserve better fixes, but we can
always revisit it.
the RangeEnabled option now has no effect, but it still exists in the API. A few
tests still use it to ensure this. This would only be considered a breaking
change if someone was relying on Pilosa to enforce the RangeEnabled: false
option to prevent fields being created in certain frames. This seems unlikely.
these were occaisionally referenced elsewhere in the codebase - in all but one
case, there were workarounds that are actually better I think.
In the one case there wasn't I created a single top level DefaultConfig object
which is instantiated with all the default values and can be referred to if
necessary.
There was a bug in fragment.go with the way MaxWritesPerRequest was treated if
it was 0. Elsewhere, 0 meant no limit, but here, it would have caused a division
by 0.
Changed the default metrics provider from "nop" to "none", although "nop" will
still work. Previously, any value other than "statsd" or "expvar" was treated as
"nop", but I've changed this behavior to return an error if an invalid string is
provided. I think this is better behavior, because in the case that someone
bothered to change the default, they were probably interested in actually
getting stats, and might be annoyed when it silently failed.
