This commit fixes an issue where translation `LogEntry` must be
read in its entirety, however, large entries can exceed the buffer
size. This has been changed so that partial entries reads are allowed.
The `LogEntry.ReadFrom()` may still generate large byte slices
during reads of large individual fields or keys.
This commit implements BSI with variable bit depth using a
sign magnitudeto indicate whether a value is positive or negative.
This also rearranges the existence bit to be the first bit instead
of the last bit.
also fix a *bunch* of tests that weren't closing the clusters they
created. Cleaned up one test to use t.Run instead of just checking
everything in a loop
So with the switch to a new linter, we get a lot of new warnings,
and the majority of them are harmless probably, but a few might be
real. Variously just use _ to suppress warnings, or report errors.
There's probably things here that deserve better fixes, but we can
always revisit it.
In the case where a translate log entry contained
many key/id pairs, it was possible for the read
buffer (which was allocated at 65536 bytes) to
fail to handle it. This happened when the serialized
LogEntry was larger than 65536 bytes.
This PR adds logic which returns a custom error called
ErrTranslateReadTargetUndersized notifying the reader
to reallocate a larger read buffer and try the read
again.
TODO:
- [ ] Add a max buffer size check to prevent this from doubling the
buffer size with no limit.
- [ ] Add tests.
suspect that this is somehow causing "cannot assign requested address" bugs for
some users. removing since it wasn't a necessary part of the deadlock fix, but
just seemed like a prudent thing to have.
I'd like to add stat tracking to Roaring, which means it
has to be able to import the stats package, which means
stats has to be a package rather than part of the pilosa
package. If stats stops being in pilosa, it still needs
a way to import logger, so logger also has to leave the
pilosa package. Then everything using them needs to import
them and use package selectors on their names.
This doesn't actually add the stats support to roaring,
it just makes it so there's a way to import the stats
code from something in the roaring package.
Long ago, the maintainers of gorilla/mux concluded that
it was a "wontfix" to return StatusMethodNotAllowed instead
of StatusNotFound for a method mismatch. Pilosa had a
workaround for this for the most common case (GET requests
to /index/{index}/query), and a TODO to address the other
cases.
While browsing the go-pilosa client code, I noticed that there
is a test for roaring import support which relies on getting
StatusMethodNotAllowed. But how can this work, if gorilla/mux
doesn't do that?
Answer: They started doing it in mid-2017, apparently:
https://github.com/gorilla/mux/issues/271
Dropping this code changes the exact text of the message
produced for that one case, but not the status code,
and makes the behavior less confusing.
This commit adds the ability to remove an 'available shard'
from the shard cache. This does not affect shards known to be
available because of local data.
