We want to be able to control whether or not we use roaring to
serialize Rows, which means serializers have to be able to be
distinct.
We also make corresponding changes to http/handler.go to have
it use the exported serializers directly rather than the API's
serializer (which is always the base protobuf serializer
right now, and if it weren't, that would be bad because we
were assuming it was).
When we're accepting protobuf from a pilosa server, flag that
we'll accept roaring bitmaps as opposed to the naive column
representation.
I think this will improve the transaction response messages Kuba
mentioned where it was an empty transaction instead of a nil or not
there... if not it should make it easier to do that anyhow.
- Remove YAML magic
- Remove a lot of duplication
- Update linter
- Use parameterized jobs and matrix build
- Update Docker Hub CD to produce versioned and "latest" images
- Add custom shard width test to workflow
This commit adds `TranslationSources` to the cluster
`ResizeInstruction`. These are the sources of translation
partitions which the receiving node needs in order to support
partition distribution in the new, resized cluster.
This also fixes a bug where index options were not being
encode in the proto Index object. That meant that the schema
transferred via protobuf was not correct. The reason why
things normally worked is because index creation typically
happens on the CreateIndex message, which does include the
options.
TODO:
- [ ] implement the TranslateStore interface for `InMemTranslateStore`
and `mock.TranslateStore`
- [ ] surely need some more tests around the `ReadFrom` and `WriteTo`
This PR adds support for anti-entropy syncing for integer
and decimal fields. It differs from the logic for other
field types in that it does not rely on a consensus to determine
what the value should be; instead, it considers the correct
values to be those of the primary replica. From there, data
is pushed to all non-primary replicas.
This allows a BSI field to have an option indicating
that it is a foreign key to another index. If the foreign
index has column keys, then this field handles string values
by using the foreign index's translate store.
This PR adds a field name (string) to the return types
which represent the values from a specific field. For example,
a TopN query on field `x` would be `TopN(x)` and have results
like:
```
[]Pair{
{ID: 14, Count: 10},
{ID: 3, Count: 8},
{ID: 7, Count: 3},
}
```
In order to know what field this result type refers to, we wrap
`[]Pair` in a new struct called `PairsField` which contains an
addition `Field` string where `x` is stored.
This is useful for informing the gRPC server how to construct
more appropriate headers for the result stream (in this case,
the column headers can now be "x" and "count").
Similar logic was applied to `RowIdentifiers` and `Pair` as well.
had to workaround some cruft in the parser that was trying to only
support a BETWEEN query as LTE, LTE. Now we have operations for all
combinations of LT and LTE.
unrelated - changed the port a test was binding to as it conflicted
with a port I was using locally.
This commit adds a Decimal field type which is implemented mostly with
the Int field. It adds an optional "Scale" value to the Int field
which means that the values stored in that field are actually meant to
be divided by 10^Scale before being interpreted.
In order to make use of this functionality, we extend the importValue
request to allow a slice of floats rather than just int64. If the
slice of floats is present, each float in the slice is multiplied by
10^Scale and converted to an int64 before being imported. If a slice
of int64 is imported to a Decimal field, it is treated normally, and
scale is ignored. This allows the conversion to be handled at the
client side if desired.
Currently there are Field level methods for querying Float values out
of a decimal field, but no support in PQL or the executor for getting
float values. Going to wait until I can use the generic result type
before doing that, so for now, any values queried will be the scaled
integer values.
needed to add client support for importing float values, and did this
by adding a more general and simplified client method for value
imports.
rewrote api.ImportValue to use the new method which should be more
performant and efficient.
allow floats to be "pilosa import"ed into decimal fields
What if you could ?profile=true on a query and get some
numbers back? That'd be really cool.
We already have tracing/spans, but right now, those only generate
any data if you have something set up for them to trace to. Add a
fancy wrapper that lets us generate our own tracing data, and dump
it into the request response, if ?profile=true.
We track wall-clock execution time, plus possible arbitrary K/V
pairs. Memory stats are not included, because obtaining them is
surprisingly expensive.
This change should have been adding the "GetClientCertificate"
function in server/tlsconfig.go. This is in addition to the
GetCertificate func which is only used by servers. It ended up being
much more involved for a few reasons:
1. We had no way of passing a configured HTTP client into the
translate store stuff.
2. Our cluster tests assumed http, not HTTPS, and didn't have any way
to pass the necessary configuration in.
3. I encountered what turned out to be an unrelated bug in
cmd/server_test.go which is why I moved "close(m.Started)" in
server/server.go. Basically, I was running something on port 10111
which caused the test to fail (because it was trying to bind to that),
but the failure was not immediately caught during server startup
because the m.Started channel got closed which allowed the test code
to fall through to where it called m.Close() which then got a nil
pointer exception because m.Handler had never been set up.
4. Our test code was assuming that it could create clients that
ignored the config, which meant they didn't do TLS. I added an
InternalClient() method to pilosa.Server to expose the configured
client.
