close files after using them if global max is passed.
I originally implemented this without the global count—just always closing files
when done with them, and reopening for new writes. This was crazy slow for that
one test that uses mustSetBits in a big loop. I modified the test to use
importRoaring and everything worked better (though much more slowly).
After adding the global counter, I ran the tests with that one test using
mustSetBits again, and the performance was similar to master. After completing
this PR, I ran the tests with the max limit set to 5—they still passed but were
much slower.
get the count of the existing fragment and compare it to the incoming bits to
decide which should be unioned into the other. This should generally result in
far fewer allocations, though there is much work that needs to be done within
UnionInPlace to further improve things.
unrelatedly, I added a TODO to change the long-query-time option to move it out
of cluster. It should probably be happening at the API level so that different
handlers can reuse it, but if we're going to do that we'll want to make sure
that any potentially time intensive operations are pulled into api from
handler (e.g. protobuf decoding)
This commit adds support for advertise address by using a new config
option `advertise`, or by defaulting its value to that
specified in `bind`.
Also adds support for listening on 0.0.0.0 by trying to determine
the preferred outbound IP to use for the advertise address.
There was a situation where availableShards on a new
node were not in sync with the cluster, so queries
following a resize were incorrect.
- Start a one-node cluster.
- Write data to shards 0 and 1
- Start a second node.
In the case where the hash algo was moving shard 0 to
node1, then node1 only knew about shard 0, so queries
to node1 would be incomplete.
This PR modifies the ResizeInstruction message to replace
`Schema` with `NodeStatus` (which contains both `Schema` and
`AvailableShards`). So now when a resize instruction is received,
the receiving node is able to sync its schema and availableShards.
I'd like to add stat tracking to Roaring, which means it
has to be able to import the stats package, which means
stats has to be a package rather than part of the pilosa
package. If stats stops being in pilosa, it still needs
a way to import logger, so logger also has to leave the
pilosa package. Then everything using them needs to import
them and use package selectors on their names.
This doesn't actually add the stats support to roaring,
it just makes it so there's a way to import the stats
code from something in the roaring package.
