Commit graph

256 commits

Author SHA1 Message Date
Travis
361e51cb41 Add All() support to PQL, including limit and offset
This PR is meant to get all columns from an index
based on the TrackExistence row.

`All()` is a PQL function that can be used as a typical
row object. Optional arguments are `limit` and `offset`.
2019-12-18 18:00:15 -06:00
Travis
5cb37834a0 Wrap return types: RowIdentifiers, Pair, and []Pair
This PR adds a field name (string) to the return types
which represent the values from a specific field. For example,
a TopN query on field `x` would be `TopN(x)` and have results
like:
```
[]Pair{
  {ID: 14, Count: 10},
  {ID: 3, Count: 8},
  {ID: 7, Count: 3},
}
```
In order to know what field this result type refers to, we wrap
`[]Pair` in a new struct called `PairsField` which contains an
addition `Field` string where `x` is stored.

This is useful for informing the gRPC server how to construct
more appropriate headers for the result stream (in this case,
the column headers can now be "x" and "count").

Similar logic was applied to `RowIdentifiers` and `Pair` as well.
2019-12-13 15:36:51 -06:00
Todd Gruben
4523a4d693
removed uneeded test run 2019-12-13 15:10:28 -06:00
Todd Gruben
16171b3e65
return total match counts for either min or max 2019-12-13 15:10:28 -06:00
Travis
24d02c1920 Add "having" support to GroupBy() queries
This PR adds support for a `having` argument in a `GroupBy` query.
Usage looks like this:
```
GroupBy(Rows(a), having=Condition(count > 10))
GroupBy(Rows(a), aggregate=Sum(field=b), having=Condition(sum > 100))
```
2019-11-28 18:40:57 -06:00
Ben Johnson
ea9914dba0
Add optional GroupBy() 'aggregate' field.
This commit adds an `aggregate` field that allows a `Sum()` call
to be executed for every returned group.
2019-11-27 11:00:13 -06:00
Matt Jaffee
81a4d32cdd
allow IDs to be passed even when keys enabled
This change allows one to query Pilosa fields and indexes directly
with integer row and column ids even when key translation is
enabled. This was previously disallowed during query
translation... I'm not sure why, but it can be quite useful for
debugging and testing to be able to use IDs directly. I have a test in
go-pilosa which uses this functionality.

I also simplified a bunch of the test code which was of the form:
```
else {
    if blah {
    }
}
```

to be:

```
else if blah {
```

which I think is pretty harmless.

I also changed a snapshot log line that has been bugging me to be
Debug level so that it isn't generating lots of useless logs for long
running Pilosa instances.
2019-11-27 08:41:40 -06:00
Travis
625125bac6 add column keys support to IncludeColumn 2019-11-13 11:34:21 -06:00
Travis
ed37ef5dcf Add an IncludesColumn() function to PQL
Usage:
`IncludesColumn(Intersect(Row(a=1), Row(b=2)), column=10)`

The above query will return a `bool` indicating whether the
intersection of rows a-1 and b-2 contains column 10. Because
a single column is specified, this executes on a single shard
(shard=0 in this example).
2019-11-11 08:17:28 -06:00
Matt Jaffee
2651bfbd88
test and fix authentication of client certs btwn cluster nodes
This change should have been adding the "GetClientCertificate"
function in server/tlsconfig.go. This is in addition to the
GetCertificate func which is only used by servers. It ended up being
much more involved for a few reasons:

1. We had no way of passing a configured HTTP client into the
translate store stuff.

2. Our cluster tests assumed http, not HTTPS, and didn't have any way
to pass the necessary configuration in.

3. I encountered what turned out to be an unrelated bug in
cmd/server_test.go which is why I moved "close(m.Started)" in
server/server.go. Basically, I was running something on port 10111
which caused the test to fail (because it was trying to bind to that),
but the failure was not immediately caught during server startup
because the m.Started channel got closed which allowed the test code
to fall through to where it called m.Close() which then got a nil
pointer exception because m.Handler had never been set up.

4. Our test code was assuming that it could create clients that
ignored the config, which meant they didn't do TLS. I added an
InternalClient() method to pilosa.Server to expose the configured
client.
2019-10-18 15:17:35 -05:00
Ben Johnson
e844e1ad75
Translation store refactor 2019-10-09 08:59:41 -06:00
Ben Johnson
c7c9c1e1d7
v2.0.0
Co-authored-by: Cody Soyland <codysoyland@gmail.com>
2019-10-08 14:56:17 -06:00
Matt Jaffee
5b655418b1
add another single day query test 2019-07-25 08:12:19 -05:00
Matt Jaffee
2c311e685e
add bounds check when getting time rows 2019-07-25 08:12:19 -05:00
Yuce Tekol
8be7bd6956
add tests for MinRow and MaxRow 2019-06-03 13:56:50 +03:00
Ben Johnson
dd4227f5e3
Improve TopN() errors
This commit improves field not found, integer field, and cache errors
for the `TopN()` command.
2019-05-25 15:16:45 -06:00
Ben Johnson
40803372dd
Add min/max constraints; fix tests 2019-05-19 16:05:22 -06:00
Ben Johnson
d4de122549
Add min/max constraints 2019-05-17 15:52:17 -06:00
Ben Johnson
7ed9fba335
Unbounded BSI w/ sign magnitude
This commit implements BSI with variable bit depth using a
sign magnitudeto indicate whether a value is positive or negative.
This also rearranges the existence bit to be the first bit instead
of the last bit.
2019-05-17 15:52:17 -06:00
Matt Jaffee
e185a01e67
add translation for groupby filter arg, improve test 2019-05-10 14:15:06 -05:00
Matt Jaffee
de61d04172
failing test for group by with filter using string keys
also, apparently our API code was assuming that imports with keys
always had timestamps which seemed wrong, so I fixed that.
2019-05-10 13:55:21 -05:00
Matt Jaffee
9e6662fb00
send POSTed schema to all nodes in cluster
also fix a *bunch* of tests that weren't closing the clusters they
created. Cleaned up one test to use t.Run instead of just checking
everything in a loop
2019-04-29 19:31:23 -05:00
Cody Soyland
7ede65bf80
Merge branch 'master' into shardwidth22 2019-04-11 10:10:47 -05:00
Matt Jaffee
55d9d49f2f
add executor test for deprecated range query style 2019-04-05 07:52:25 -05:00
Matt Jaffee
836b467d3d
add support to modify shard width at build time
use "make <x> SHARD_WIDTH=nn"

fix tests to run and pass at different shardwidths

add shardwidth22 test to circle ci
2019-04-04 13:46:26 -05:00
Travis Turner
87b3438cb1
add logic to restrict time range to available views 2019-02-01 15:30:10 -06:00
Travis Turner
a242b8cbaf
add from/to range arguments to Rows() 2019-01-30 16:27:07 -06:00
Travis Turner
30711664e8
ensure ClearRow() arguments get translated 2019-01-29 12:36:54 -06:00
Travis Turner
687b67dc54
prevent omitting zero ids on columnattrs 2019-01-28 13:16:43 -06:00
Travis Turner
8fe966e8a0
Modifying some of the logic around Shift()
add some comments to the shift() logic
improve test coverage
fix full bitmap overflow
add support to specify shift-by amount
2019-01-25 12:54:44 -06:00
Travis Turner
2467d88ddc
Merge branch 'master' into shift-op 2019-01-24 13:56:28 -06:00
Matt Jaffee
b7973b75f1
get rid of init 2019-01-21 16:23:28 -06:00
Matt Jaffee
04c7ab5034
add nolint for init 2019-01-21 16:23:27 -06:00
Matt Jaffee
fe7b926773
make sure more tests and benchmarks can have their temp dir set by flag
This is to allow the directory to be set to where a particular disk is mounted
during benchmarking.
2019-01-21 16:23:27 -06:00
Yuce Tekol
773d661b68
GroupBy legacy Rows 2019-01-21 18:02:51 +03:00
Yuce Tekol
ca7dc073ab
Merge branch 'master' into 1805-rows-call-format 2019-01-18 17:48:58 +03:00
Yuce Tekol
c30f9bc192
Rows accepts a fields param for backward compat. 2019-01-18 17:41:54 +03:00
Matthew Jaffee
6f21eb32de
Apply suggestions from code review
Co-Authored-By: travisturner <travis@pilosa.com>
2019-01-17 14:47:35 -06:00
Yuce Tekol
dc735c17b8
Fixes #1805. Fixes Store call error messages 2019-01-16 20:52:33 +03:00
tgruben
e5df52836e
Merge branch 'master' into shift-op 2019-01-15 08:57:28 -06:00
Travis Turner
44f53a5f1d
raise an error on Rows() query against a time field with noStandardView:true 2019-01-14 16:14:45 -06:00
Yuce Tekol
76de81dacf
updated row range test 2019-01-14 23:46:34 +03:00
Yuce Tekol
d75e9eb772
updated row range test 2019-01-14 23:45:58 +03:00
Yuce Tekol
600b39e4e5
updates row range test with a timestamp > the default end timestamp 2019-01-14 23:40:31 +03:00
Yuce Tekol
9f6d489be8
fixes #1823. Updates tests and docs for row range 2019-01-14 14:39:08 +03:00
Travis Turner
f7e3296f62
fixes a bug on upper end of bsi range queries 2019-01-11 17:05:29 -06:00
Ben Johnson
9d1e5ca8ce
Merge Range() into Row() call.
This commit refactors the `Range()` call and merges its functionality
into the `Row()` call.
2019-01-09 15:10:49 -07:00
Yuce Tekol
65c30283d7
adds tests for GroupBy with keys; removes unused Bit message from proto 2019-01-03 15:20:02 +03:00
Matt Jaffee
ed637e6921
add test for group by with invalid filter 2018-12-21 17:24:44 -06:00
Matt Jaffee
81d08be044
fix PQL, Rows and Group By problems
make sure that args which are Uints are positive and return an error if not.

improve group by error messages if field for Rows query is invalid
2018-12-21 14:16:55 -06:00