needed an unlocked version of sendsync for use within the cluster, so also
implemented that. Added a number of tests trying to reproduce the issue, but was
not able to. Not sure it's worth keeping the new tests.
Explained in a comment:
We default these Max File/Map counts very high. This is basically a
backwards compatibility thing where we don't want to cause different
behavior for those who had previously set their system limits high,
and weren't experiencing any bad behavior. Ideally you want these set
a bit below your system limits.
report the approximate hardware specs (CPU speed, cores, memory)
of the server in the /info endpoint. This may be useful when
benchmarking.
We do some workarounds because gopsutil's core count output is
confusingly different between Linux and Darwin, and the MHz output
is usually wrong on Linux. Intel's app notes say to just parse
the model string. Whyyyyyyy.
set sane defaults. The performance overhead seems to be negligible, and this will allow us to obtain mutex and blocking profiles from running Pilosas by default.
close files after using them if global max is passed.
I originally implemented this without the global count—just always closing files
when done with them, and reopening for new writes. This was crazy slow for that
one test that uses mustSetBits in a big loop. I modified the test to use
importRoaring and everything worked better (though much more slowly).
After adding the global counter, I ran the tests with that one test using
mustSetBits again, and the performance was similar to master. After completing
this PR, I ran the tests with the max limit set to 5—they still passed but were
much slower.
get the count of the existing fragment and compare it to the incoming bits to
decide which should be unioned into the other. This should generally result in
far fewer allocations, though there is much work that needs to be done within
UnionInPlace to further improve things.
unrelatedly, I added a TODO to change the long-query-time option to move it out
of cluster. It should probably be happening at the API level so that different
handlers can reuse it, but if we're going to do that we'll want to make sure
that any potentially time intensive operations are pulled into api from
handler (e.g. protobuf decoding)
This commit adds support for advertise address by using a new config
option `advertise`, or by defaulting its value to that
specified in `bind`.
Also adds support for listening on 0.0.0.0 by trying to determine
the preferred outbound IP to use for the advertise address.
There was a situation where availableShards on a new
node were not in sync with the cluster, so queries
following a resize were incorrect.
- Start a one-node cluster.
- Write data to shards 0 and 1
- Start a second node.
In the case where the hash algo was moving shard 0 to
node1, then node1 only knew about shard 0, so queries
to node1 would be incomplete.
This PR modifies the ResizeInstruction message to replace
`Schema` with `NodeStatus` (which contains both `Schema` and
`AvailableShards`). So now when a resize instruction is received,
the receiving node is able to sync its schema and availableShards.
I'd like to add stat tracking to Roaring, which means it
has to be able to import the stats package, which means
stats has to be a package rather than part of the pilosa
package. If stats stops being in pilosa, it still needs
a way to import logger, so logger also has to leave the
pilosa package. Then everything using them needs to import
them and use package selectors on their names.
This doesn't actually add the stats support to roaring,
it just makes it so there's a way to import the stats
code from something in the roaring package.
