If an index is provided to a bare distinct which happens
to be the index handling the query, then the query needs
to behave as if no index argument was provided.
For example:
When querying against index `i`,
```
Distinct(index="i", field="ints")`
```
should behave exactly like
```
Distinct(field="ints")
```
Problem: A top-level bare "Distinct" call returns results only
for shards on the current node.
Analysis: We don't actually want to limit Distinct calls to "available"
shards at all. We just want to run them on everything. But we already
did that in generating the precomputed results; all we need to do is,
if we get a non-shard-specific request for precomputed values, just
return all the values.
It's pretty hard to create logic for this using our fancy mapReduce,
but also we could just... not do that.
This commit introduces a new type: pql.Decimal
We use that instead of float64 in order to ensure
that the string representation is consistent.
One unfortunate discovery during implementation is
that the RowAttrs and ColAttrs support floats, and
the PEG file was treating them as such. So I had
to split the PEG definitions into float-specific
items and decimal-specific items.
this avoids compounding floating point errors while summing up the
numbers, and means less logic needs to change. Should probably convert
min and max to use this approach as well, though they don't suffer
from the compounding error issue, it is simpler.
this involved adding an optional float value to the ValCount struct
which complicated result types, necessitated grpc changes, and needed
quite a few tests at different layers.
For Fields with ForeignIndex (which have keys), the API was missing
the logic to do that translation against the translateStore of
the foreign index. This commit adds that logic, as well as some
missing translateStore-related logic in the gRPC code.
In the case where a field with a foreign index opens before the
foreign index has opened (and is available as a reference in the
holder), push the field into a queue to have its foreign index
applied once all indexes have opened.
In the `Inspect` function in `server/grpc.go`, getting
the value of an `int` field with a foreign index to
an index with `Keys()`, we need to return the string
key value instead of the BSI int value for the field.
This commit also changes the method `Field.keys()` to be
exported as `Field.Keys()` so that it's accessible in
the server package.
This commit changes the order of FieldOption application so that
it's always set before field.Open() is called.
This was required because field.Open() now uses some of the values
from FieldOptions to determine if/when to use a particular
translateStore. For example, when FieldOptions.ForeignIndex is set,
the translateStore from the foreign index is retrieved during
field.Open().
This allows a BSI field to have an option indicating
that it is a foreign key to another index. If the foreign
index has column keys, then this field handles string values
by using the foreign index's translate store.
After a few experiments with pkg/plugin, I'm ready to concede that the
people warning me it was unsuitable for production use were in fact
correct.
In the brave new world, the "ext" package is moved to its own module
outside pilosa. This means that importing it doesn't imply any need to
version-check against pilosa; we can just use versioned copies of the
ext package, which can be public because it doesn't contain anything
we need to care about keeping proprietary.
Then we can, conditional on build tags, import modules from a
neighboring repo which contains the actual implementations, and if
they're imported, their init functions register them.
This PR is meant to get all columns from an index
based on the TrackExistence row.
`All()` is a PQL function that can be used as a typical
row object. Optional arguments are `limit` and `offset`.
This PR adds a field name (string) to the return types
which represent the values from a specific field. For example,
a TopN query on field `x` would be `TopN(x)` and have results
like:
```
[]Pair{
{ID: 14, Count: 10},
{ID: 3, Count: 8},
{ID: 7, Count: 3},
}
```
In order to know what field this result type refers to, we wrap
`[]Pair` in a new struct called `PairsField` which contains an
addition `Field` string where `x` is stored.
This is useful for informing the gRPC server how to construct
more appropriate headers for the result stream (in this case,
the column headers can now be "x" and "count").
Similar logic was applied to `RowIdentifiers` and `Pair` as well.
This PR adds support for a `having` argument in a `GroupBy` query.
Usage looks like this:
```
GroupBy(Rows(a), having=Condition(count > 10))
GroupBy(Rows(a), aggregate=Sum(field=b), having=Condition(sum > 100))
```
This change allows one to query Pilosa fields and indexes directly
with integer row and column ids even when key translation is
enabled. This was previously disallowed during query
translation... I'm not sure why, but it can be quite useful for
debugging and testing to be able to use IDs directly. I have a test in
go-pilosa which uses this functionality.
I also simplified a bunch of the test code which was of the form:
```
else {
if blah {
}
}
```
to be:
```
else if blah {
```
which I think is pretty harmless.
I also changed a snapshot log line that has been bugging me to be
Debug level so that it isn't generating lots of useless logs for long
running Pilosa instances.
It turns out that we need to recompute the set of shards whenever
a query is cross-index. Otherwise we get partial results in unexpected
ways sometimes.
