Merge pull request #1804 from benbjohnson/deprecate-range

Merge Range() into Row() call.
This commit is contained in:
Travis Turner 2019-01-10 09:40:13 -06:00 committed by GitHub
commit 4e7a3feee2
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
13 changed files with 1841 additions and 1586 deletions

View file

@ -223,7 +223,7 @@ func TestAPI_ImportValue(t *testing.T) {
t.Fatal(err)
}
pql := fmt.Sprintf("Range(%s>0)", field)
pql := fmt.Sprintf("Row(%s>0)", field)
// Query node0.
if res, err := m0.API.Query(ctx, &pilosa.QueryRequest{Index: index, Query: pql}); err != nil {

View file

@ -323,7 +323,7 @@ We currently track the following events
- **Xor:** Count of Xor queries.
- **Not:** Count of Not queries.
- **Count:** Count of Count queries.
- **Range:** Count of Range queries.
- **Range:** Count of ranged Row queries.
- **Snapshot:** Event count when the snapshot process is triggered.
- **BlockRepair:** Count of data blocks that were out of sync and repaired.
- **GarbageCollection:** Event count when garbage collection occurs.

View file

@ -64,7 +64,7 @@ Simple queries:
Relational | Pilosa
-----------------------------------------------|------------------------------------
`select ID from People where Name = 'Bob'` | `Row(Name="Bob")`
`select ID from People where Age > 30` | `Range(Age > 30)`
`select ID from People where Age > 30` | `Row(Age > 30)`
`select ID from People where Member = true` | `Row(Member=0)`
Note that `Row(Member=0)` selects all entities with a bit set in row 0 of the Member field. We could just as well use row 1 to store this, in which case we would use `Row(Member=1)`, which looks a bit more intuitive. In the relational model, joins are often necessary. Because Pilosa supports extremely high cardinality in both rows and columns, many types of joins are accomplished with basic Pilosa queries across multiple fields. For example, this SQL join:
@ -100,7 +100,7 @@ The LRU cache maintains the most recently accessed Rows.
### Time Quantum
Setting a time quantum on a field creates extra views which allow Range queries down to the time interval specified. For example, if the time quantum is set to `YMD`, Range queries down to the granularity of a day are supported.
Setting a time quantum on a field creates extra views which allow ranged Row queries down to the time interval specified. For example, if the time quantum is set to `YMD`, ranged Row queries down to the granularity of a day are supported.
### Attribute
@ -145,7 +145,7 @@ curl localhost:10101/index/repository/field/quantity \
##### BSI Range-Encoding
Bit-Sliced Indexing (BSI) is the storage method Pilosa uses to represent multi-bit integers in a bitmap index. Integers are stored as n-bit, range-encoded bit-sliced indexes of base-2, along with an additional row indicating "not null". This means that a 16-bit integer will require 17 rows: one for each 0-bit of the 16 bit-slice components (the 1-bit does not need to be stored because with range-encoding the highest bit position is always 1) and one for the non-null row. Pilosa can evaluate `Range`, `Min`, `Max`, and `Sum` queries on these BSI integers. The result of a `Sum` query includes a count, which can be used to compute an average with no other overhead.
Bit-Sliced Indexing (BSI) is the storage method Pilosa uses to represent multi-bit integers in a bitmap index. Integers are stored as n-bit, range-encoded bit-sliced indexes of base-2, along with an additional row indicating "not null". This means that a 16-bit integer will require 17 rows: one for each 0-bit of the 16 bit-slice components (the 1-bit does not need to be stored because with range-encoding the highest bit position is always 1) and one for the non-null row. Pilosa can evaluate `Row`, `Min`, `Max`, and `Sum` queries on these BSI integers. The result of a `Sum` query includes a count, which can be used to compute an average with no other overhead.
Internally Pilosa stores each BSI `field` as a `view`. The rows of the `view` contain the base-2 representations of the integer values. Pilosa manages the base-2 offset and translation that efficiently packs the integer value within the minimum set of rows.

View file

@ -46,16 +46,16 @@ nav = []
<strong id="protobuf">[Protobuf](https://developers.google.com/protocol-buffers/):</strong> Protocol Buffers is a binary serialization format which Pilosa uses for internal messages, and can be used by clients as an alternative to JSON.
<strong id="range">[Range](../query-language/#range-queries):</strong>: A [PQL](#pql) query that returns bits based on comparison to timestamps, set according to the [time quantum](#time-quantum).
<strong id="range-bsi">[Range (BSI)](../query-language/#range-bsi):</strong>: A [PQL](#pql) query that returns bits based on comparison to integers stored in [BSI](#bsi) [fields](#field).
<strong id="replica">[Replica](../configuration/#cluster-replicas):</strong> A copy of a [fragment](#fragment) on a different [node](#node) than the original. The `cluster.replicas` configuration parameter determines how many replicas of a fragment exist in the cluster. This includes the original, so a value of 1 means no extra copies are made.
<strong id="roaring-bitmap">[Roaring Bitmap](http://roaringbitmap.org):</strong> the compressed bitmap format which Pilosa uses to [implement bitmaps](../architecture/#roaring-bitmap-storage-format), for both storage and logical query operations.
<strong id="row">[Row](../data-model/#row):</strong> Rows are the fundamental vertical data axis within Pilosa. They are namespaced to each [field](#field) within an [index](#index). Represented as a [Bitmap](#bitmap).
<strong id="range">[Row (Ranged)](../query-language/#range-queries):</strong>: A [PQL](#pql) query that returns bits based on comparison to timestamps, set according to the [time quantum](#time-quantum).
<strong id="range-bsi">[Row (BSI)](../query-language/#range-bsi):</strong>: A [PQL](#pql) query that returns bits based on comparison to integers stored in [BSI](#bsi) [fields](#field).
<strong id="slice">[Slice](../data-model/#shard):</strong> Prior to Pilosa 1.0, shards were known as slices.
<strong id="shard">[Shard](../data-model/#shard):</strong> [Columns](#column) are [sharded](https://en.wikipedia.org/wiki/Shard_(database_architecture)) on a preset [width](#shardwidth). Shards are operated on in parallel and are evenly distributed across the cluster via a [consistent hash](#jump-consistent-hash).
@ -64,7 +64,7 @@ nav = []
<strong id="sum">[Sum](../query-language/#sum):</strong> A [PQL](#pql) query that returns the sum of integers stored in an [integer](#bsi) [field](#field).
<strong id="time-quantum">[Time quantum](../data-model/#time-quantum):</strong> Defines the granularity to be used for [Range](#range) queries on time [fields](#field).
<strong id="time-quantum">[Time quantum](../data-model/#time-quantum):</strong> Defines the granularity to be used for [ranged Row](#range) queries on time [fields](#field).
<strong id="toml">[TOML](https://github.com/toml-lang/toml):</strong> the language used for Pilosa's [configuration file](../configuration/).

View file

@ -50,7 +50,7 @@ curl localhost:10101/index/repository/query \
* `ATTR_NAME` Must be a valid identifier `[A-Za-z][A-Za-z0-9._-]*`
* `ATTR_VALUE` Can be a string, float, integer, or bool.
* `CALL` Any query
* `ROW_CALL` Any query which returns a row, such as `Row`, `Union`, `Difference`, `Xor`, `Intersect`, `Range`, `Not`
* `ROW_CALL` Any query which returns a row, such as `Row`, `Union`, `Difference`, `Xor`, `Intersect`, `Not`
* `[]ATTR_VALUE` Denotes an array of `ATTR_VALUE`s. (e.g. `["a", "b", "c"]`)
### Write Operations
@ -335,6 +335,89 @@ Row(stargazer=1)
* attrs are the attributes for user 1
* columns are the repositories which user 1 has starred.
#### Row (Range)
**Spec:**
```
Row(<FIELD>=<ROW>, <TIMESTAMP>, <TIMESTAMP>)
```
**Description:**
Similar to `Row`, but only returns bits which were set with timestamps
between the given `start` (first) and `end` (second) timestamps.
**Result Type:** object with attrs and bits
**Examples:**
Query all columns with a bit set in row 1 of a field (repositories that a user has starred), within a date range:
```request
Row(stargazer=1, 2010-01-01T00:00, 2017-03-02T03:00)
```
```response
{{"attrs":{},"columns":[10]}
```
This example assumes timestamps have been set on some bits.
* columns are repositories which were starred by user 1 in the time range 2010-01-01 to 2017-03-02.
#### Row (BSI)
**Spec:**
```
Row([<COMPARISON_VALUE> <COMPARISON_OPERATOR>] <FIELD> <COMPARISON_OPERATOR> <COMPARISON_VALUE>)
```
**Description:**
The `Row` query is overloaded to work on `integer` values as well as `timestamp` values.
Returns bits that are true for the comparison operator.
**Result Type:** object with attrs and columns
**Examples:**
In our source data, commitactivity was counted over the last year.
The following greater-than `Row` query returns all columns with a field value greater than 100 (repositories having more than 100 commits):
```request
Row(commitactivity > 100)
```
```response
{{"attrs":{},"columns":[10]}
```
* columns are repositories which had at least 100 commits in the last year.
BSI range queries support the following operators:
Operator | Name | Value
----------|-------------------------------|--------------------
`>` | greater-than, GT | integer
`<` | less-than, LT | integer
`<=` | less-than-or-equal-to, LTE | integer
`>=` | greater-than-or-equal-to, GTE | integer
`==` | equal-to, EQ | integer
`!=` | not-equal-to, NEQ | integer or `null`
`<`, and `<=` can be chained together to represent a bounded interval. For example:
```request
Row(50 < commitactivity < 150)
```
```response
{{"attrs":{},"columns":[10]}
```
As of Pilosa 1.0, the "between" syntax `Row(frame=stats, commitactivity >< [50, 150])` is no longer supported.
#### Union
**Spec:**
@ -584,87 +667,6 @@ TopN(stargazer, n=2, attrName=active, attrValues=[true])
* Results are the top two users (rows) which have the "active" attribute set to "true", sorted by the number of bits set (repositories that they've starred).
#### Range Queries
**Spec:**
```
Range(<FIELD>=<ROW>, <TIMESTAMP>, <TIMESTAMP>)
```
**Description:**
Similar to `Row`, but only returns bits which were set with timestamps
between the given `start` (first) and `end` (second) timestamps.
**Result Type:** object with attrs and bits
**Examples:**
Query all columns with a bit set in row 1 of a field (repositories that a user has starred), within a date range:
```request
Range(stargazer=1, 2010-01-01T00:00, 2017-03-02T03:00)
```
```response
{{"attrs":{},"columns":[10]}
```
This example assumes timestamps have been set on some bits.
* columns are repositories which were starred by user 1 in the time range 2010-01-01 to 2017-03-02.
#### Range (BSI)
**Spec:**
```
Range([<COMPARISON_VALUE> <COMPARISON_OPERATOR>] <FIELD> <COMPARISON_OPERATOR> <COMPARISON_VALUE>)
```
**Description:**
The `Range` query is overloaded to work on `integer` values as well as `timestamp` values.
Returns bits that are true for the comparison operator.
**Result Type:** object with attrs and columns
**Examples:**
In our source data, commitactivity was counted over the last year.
The following greater-than `Range` query returns all columns with a field value greater than 100 (repositories having more than 100 commits):
```request
Range(commitactivity > 100)
```
```response
{{"attrs":{},"columns":[10]}
```
* columns are repositories which had at least 100 commits in the last year.
BSI range queries support the following operators:
Operator | Name | Value
----------|-------------------------------|--------------------
`>` | greater-than, GT | integer
`<` | less-than, LT | integer
`<=` | less-than-or-equal-to, LTE | integer
`>=` | greater-than-or-equal-to, GTE | integer
`==` | equal-to, EQ | integer
`!=` | not-equal-to, NEQ | integer or `null`
`<`, and `<=` can be chained together to represent a bounded interval. For example:
```request
Range(50 < commitactivity < 150)
```
```response
{{"attrs":{},"columns":[10]}
```
As of Pilosa 1.0, the "between" syntax `Range(frame=stats, commitactivity >< [50, 150])` is no longer supported.
#### Min

View file

@ -444,7 +444,7 @@ Refer to the [Docker documentation](https://docs.docker.com) to see your options
#### Introduction
Pilosa can store integer values associated to the columns in an index, and those values are used to support `Range`, `Min`, `Max`, and `Sum` queries. In this tutorial we will show how to set up integer fields, populate those fields with data, and query the fields. The example index we're going to create will represent fictional patients at a medical facility and various bits of information about those patients.
Pilosa can store integer values associated to the columns in an index, and those values are used to support `Row`, `Min`, `Max`, and `Sum` queries. In this tutorial we will show how to set up integer fields, populate those fields with data, and query the fields. The example index we're going to create will represent fictional patients at a medical facility and various bits of information about those patients.
First, create an index called `patients`:
``` request
@ -534,17 +534,17 @@ pilosa import -i patients --field age ages.csv
Now that we have some data in our index, let's run a few queries to demonstrate how to use that data.
In order to find all patients over the age of 40, then simply run a `Range` query against the `age` field.
In order to find all patients over the age of 40, then simply run a `Row` query against the `age` field.
``` request
curl localhost:10101/index/patients/query \
-X POST \
-d 'Range(age > 40)'
-d 'Row(age > 40)'
```
``` response
{"results":[{"attrs":{},"columns":[2,6,9]}]}
```
You can find a list of supported range operators in the [Range Query](../query-language/#range-bsi) documentation.
You can find a list of supported range operators in the [Row (BSI) Query](../query-language/#range-bsi) documentation.
To find the average age of all patients, run a `Sum` query:
``` request
@ -561,7 +561,7 @@ You can also provide a filter to the `Sum()` function to find the average age of
``` request
curl localhost:10101/index/patients/query \
-X POST \
-d 'Sum(Range(age > 40), field="age")'
-d 'Sum(Row(age > 40), field="age")'
```
``` response
{"results":[{"value":191,"count":3}]}
@ -583,7 +583,7 @@ You can also provide a filter to the `Min()` function to find the minimum age of
``` request
curl localhost:10101/index/patients/query \
-X POST \
-d 'Min(Range(age > 40), field="age")'
-d 'Min(Row(age > 40), field="age")'
```
``` response
{"results":[{"value":57,"count":1}]}
@ -604,7 +604,7 @@ You can also provide a filter to the `Max()` function to find the maximum age of
``` request
curl localhost:10101/index/patients/query \
-X POST \
-d 'Max(Range(age < 40), field="age")'
-d 'Max(Row(age < 40), field="age")'
```
``` response
{"results":[{"value":34,"count":1}]}

View file

@ -18,6 +18,7 @@ import (
"context"
"encoding/json"
"fmt"
"log"
"sort"
"time"
@ -492,11 +493,11 @@ func (e *executor) executeBitmapCall(ctx context.Context, index string, c *pql.C
return nil, errors.Wrap(err, "map reduce")
}
// Attach attributes for Row() calls.
// Attach attributes for non-BSI Row() calls.
// If the column label is used then return column attributes.
// If the row label is used then return bitmap attributes.
row, _ := other.(*Row)
if c.Name == "Row" {
if c.Name == "Row" && !c.HasConditionArg() {
if opt.ExcludeRowAttrs {
row.Attrs = map[string]interface{}{}
} else {
@ -546,14 +547,12 @@ func (e *executor) executeBitmapCallShard(ctx context.Context, index string, c *
defer span.Finish()
switch c.Name {
case "Row":
return e.executeBitmapShard(ctx, index, c, shard)
case "Row", "Range":
return e.executeRowShard(ctx, index, c, shard)
case "Difference":
return e.executeDifferenceShard(ctx, index, c, shard)
case "Intersect":
return e.executeIntersectShard(ctx, index, c, shard)
case "Range":
return e.executeRangeShard(ctx, index, c, shard)
case "Union":
return e.executeUnionShard(ctx, index, c, shard)
case "Xor":
@ -1170,10 +1169,19 @@ func (e *executor) executeRowsShard(_ context.Context, index string, c *pql.Call
return frag.rows(start, filters...), nil
}
func (e *executor) executeBitmapShard(ctx context.Context, index string, c *pql.Call, shard uint64) (*Row, error) {
span, _ := tracing.StartSpanFromContext(ctx, "Executor.executeBitmapShard")
func (e *executor) executeRowShard(ctx context.Context, index string, c *pql.Call, shard uint64) (*Row, error) {
span, _ := tracing.StartSpanFromContext(ctx, "Executor.executeRowShard")
defer span.Finish()
if c.Name == "Range" {
log.Print("DEPRECATED: Range() is deprecated, please use Row() instead.")
}
// Handle bsiGroup ranges differently.
if c.HasConditionArg() {
return e.executeRowBSIGroupShard(ctx, index, c, shard)
}
// Fetch column label from index.
idx := e.Holder.Index(index)
if idx == nil {
@ -1197,93 +1205,43 @@ func (e *executor) executeBitmapShard(ctx context.Context, index string, c *pql.
return nil, fmt.Errorf("Row() must specify %v", rowLabel)
}
frag := e.Holder.fragment(index, fieldName, viewStandard, shard)
if frag == nil {
return NewRow(), nil
}
return frag.row(rowID), nil
}
// executeIntersectShard executes a intersect() call for a local shard.
func (e *executor) executeIntersectShard(ctx context.Context, index string, c *pql.Call, shard uint64) (*Row, error) {
span, ctx := tracing.StartSpanFromContext(ctx, "Executor.executeIntersectShard")
defer span.Finish()
var other *Row
if len(c.Children) == 0 {
return nil, fmt.Errorf("empty Intersect query is currently not supported")
}
for i, input := range c.Children {
row, err := e.executeBitmapCallShard(ctx, index, input, shard)
if err != nil {
return nil, err
}
if i == 0 {
other = row
} else {
other = other.Intersect(row)
// Parse "from" time, if set.
var fromTime time.Time
if _, ok := c.Args["from"]; ok {
switch v := c.Args["from"].(type) {
case string:
if fromTime, err = time.Parse(TimeFormat, v); err != nil {
return nil, errors.New("cannot parse Row() 'from' time")
}
case int64:
fromTime = time.Unix(v, 0).UTC()
default:
return nil, errors.New("Row() 'from' arg must be a timestamp")
}
}
other.invalidateCount()
return other, nil
}
// executeRangeShard executes a range() call for a local shard.
func (e *executor) executeRangeShard(ctx context.Context, index string, c *pql.Call, shard uint64) (*Row, error) {
span, ctx := tracing.StartSpanFromContext(ctx, "Executor.executeRangeShard")
defer span.Finish()
// Handle bsiGroup ranges differently.
if c.HasConditionArg() {
return e.executeBSIGroupRangeShard(ctx, index, c, shard)
// Parse "to" time, if set.
var toTime time.Time
if _, ok := c.Args["to"]; ok {
switch v := c.Args["to"].(type) {
case string:
if toTime, err = time.Parse(TimeFormat, v); err != nil {
return nil, errors.New("cannot parse Row() 'to' time")
}
case int64:
toTime = time.Unix(v, 0).UTC()
default:
return nil, errors.New("Row() 'to' arg must be a timestamp")
}
}
// Parse field.
fieldName, err := c.FieldArg()
if err != nil {
return nil, errors.New("Range() argument required: field")
}
// Retrieve column label.
idx := e.Holder.Index(index)
if idx == nil {
return nil, ErrIndexNotFound
}
// Retrieve base field.
f := idx.Field(fieldName)
if f == nil {
return nil, ErrFieldNotFound
}
// Read row & column id.
rowID, rowOK, err := c.UintArg(fieldName)
if err != nil {
return nil, fmt.Errorf("executeRangeShard - reading row: %v", err)
}
if !rowOK {
return nil, fmt.Errorf("Range() must specify %q", rowLabel)
}
// Parse start time.
startTimeStr, ok := c.Args["_start"].(string)
if !ok {
return nil, errors.New("Range() start time required")
}
startTime, err := time.Parse(TimeFormat, startTimeStr)
if err != nil {
return nil, errors.New("cannot parse Range() start time")
}
// Parse end time.
endTimeStr, ok := c.Args["_end"].(string)
if !ok {
return nil, errors.New("Range() end time required")
}
endTime, err := time.Parse(TimeFormat, endTimeStr)
if err != nil {
return nil, errors.New("cannot parse Range() end time")
// Simply return row if times are not set.
if c.Name == "Row" && fromTime.IsZero() && toTime.IsZero() {
frag := e.Holder.fragment(index, fieldName, viewStandard, shard)
if frag == nil {
return NewRow(), nil
}
return frag.row(rowID), nil
}
// If no quantum exists then return an empty bitmap.
@ -1292,9 +1250,17 @@ func (e *executor) executeRangeShard(ctx context.Context, index string, c *pql.C
return &Row{}, nil
}
// Set maximum "to" value if only "from" is set. We don't need to worry
// about setting the minimum "from" since it is the zero value if omitted.
if toTime.IsZero() {
// This is the maximum comparable time.Time value.
// https://stackoverflow.com/a/32620397
toTime = time.Unix(1<<63-62135596801, 999999999)
}
// Union bitmaps across all time-based views.
row := &Row{}
for _, view := range viewsByTimeRange(viewStandard, startTime, endTime, q) {
for _, view := range viewsByTimeRange(viewStandard, fromTime, toTime, q) {
f := e.Holder.fragment(index, fieldName, view, shard)
if f == nil {
continue
@ -1303,18 +1269,19 @@ func (e *executor) executeRangeShard(ctx context.Context, index string, c *pql.C
}
f.Stats.Count("range", 1, 1.0)
return row, nil
}
// executeBSIGroupRangeShard executes a range(bsiGroup) call for a local shard.
func (e *executor) executeBSIGroupRangeShard(ctx context.Context, index string, c *pql.Call, shard uint64) (*Row, error) {
span, _ := tracing.StartSpanFromContext(ctx, "Executor.executeBSIGroupRangeShard")
// executeRowBSIGroupShard executes a range(bsiGroup) call for a local shard.
func (e *executor) executeRowBSIGroupShard(ctx context.Context, index string, c *pql.Call, shard uint64) (*Row, error) {
span, _ := tracing.StartSpanFromContext(ctx, "Executor.executeRowBSIGroupShard")
defer span.Finish()
// Only one conditional should be present.
if len(c.Args) == 0 {
return nil, errors.New("Range(): condition required")
return nil, errors.New("Row(): condition required")
} else if len(c.Args) > 1 {
return nil, errors.New("Range(): too many arguments")
return nil, errors.New("Row(): too many arguments")
}
// Extract conditional.
@ -1323,7 +1290,7 @@ func (e *executor) executeBSIGroupRangeShard(ctx context.Context, index string,
for k, v := range c.Args {
vv, ok := v.(*pql.Condition)
if !ok {
return nil, fmt.Errorf("Range(): %q: expected condition argument, got %v", k, v)
return nil, fmt.Errorf("Row(): %q: expected condition argument, got %v", k, v)
}
fieldName, cond = k, vv
}
@ -1335,7 +1302,7 @@ func (e *executor) executeBSIGroupRangeShard(ctx context.Context, index string,
// EQ null (not implemented: flip frag.NotNull with max ColumnID)
// NEQ null frag.NotNull()
// BETWEEN a,b(in) BETWEEN/frag.RangeBetween()
// BETWEEN a,b(in) BETWEEN/frag.RowBetween()
// BETWEEN a,b(out) BETWEEN/frag.NotNull()
// EQ <int> frag.RangeOp
// NEQ <int> frag.RangeOp
@ -1365,11 +1332,11 @@ func (e *executor) executeBSIGroupRangeShard(ctx context.Context, index string,
// Only support two integers for the between operation.
if len(predicates) != 2 {
return nil, errors.New("Range(): BETWEEN condition requires exactly two integer values")
return nil, errors.New("Row(): BETWEEN condition requires exactly two integer values")
}
// The reason we don't just call:
// return f.RangeBetween(fieldName, predicates[0], predicates[1])
// return f.RowBetween(fieldName, predicates[0], predicates[1])
// here is because we need the call to be shard-specific.
// Find bsiGroup.
@ -1402,7 +1369,7 @@ func (e *executor) executeBSIGroupRangeShard(ctx context.Context, index string,
// Only support integers for now.
value, ok := cond.Value.(int64)
if !ok {
return nil, errors.New("Range(): conditions only support integer values")
return nil, errors.New("Row(): conditions only support integer values")
}
// Find bsiGroup.
@ -1438,6 +1405,31 @@ func (e *executor) executeBSIGroupRangeShard(ctx context.Context, index string,
}
}
// executeIntersectShard executes a intersect() call for a local shard.
func (e *executor) executeIntersectShard(ctx context.Context, index string, c *pql.Call, shard uint64) (*Row, error) {
span, ctx := tracing.StartSpanFromContext(ctx, "Executor.executeIntersectShard")
defer span.Finish()
var other *Row
if len(c.Children) == 0 {
return nil, fmt.Errorf("empty Intersect query is currently not supported")
}
for i, input := range c.Children {
row, err := e.executeBitmapCallShard(ctx, index, input, shard)
if err != nil {
return nil, err
}
if i == 0 {
other = row
} else {
other = other.Intersect(row)
}
}
other.invalidateCount()
return other, nil
}
// executeUnionShard executes a union() call for a local shard.
func (e *executor) executeUnionShard(ctx context.Context, index string, c *pql.Call, shard uint64) (*Row, error) {
span, ctx := tracing.StartSpanFromContext(ctx, "Executor.executeUnionShard")

View file

@ -1478,7 +1478,7 @@ func TestExecutor_Execute_Sum(t *testing.T) {
}
// Ensure a range query can be executed.
func TestExecutor_Execute_Range(t *testing.T) {
func TestExecutor_Execute_Row_Range(t *testing.T) {
t.Run("RowIDColumnID", func(t *testing.T) {
writeQuery := `
Set(2, f=1, 1999-12-31T00:00)
@ -1492,9 +1492,9 @@ func TestExecutor_Execute_Range(t *testing.T) {
Set(2, f=1, 2002-02-01T00:00)
Set(2, f=10, 2001-01-01T00:00)`
readQueries := []string{
`Range(f=1, 1999-12-31T00:00, 2002-01-01T03:00)`,
`Row(f=1, from=1999-12-31T00:00, to=2002-01-01T03:00)`,
`Clear( 2, f=1)`,
`Range(f=1, 1999-12-31T00:00, 2002-01-01T03:00)`,
`Row(f=1, from=1999-12-31T00:00, to=2002-01-01T03:00)`,
}
responses := runCallTest(t, writeQuery, readQueries,
nil, pilosa.OptFieldTypeTime(pilosa.TimeQuantum("YMDH")))
@ -1525,9 +1525,9 @@ func TestExecutor_Execute_Range(t *testing.T) {
Set("two", f=1, 2002-02-01T00:00)
Set("two", f=10, 2001-01-01T00:00)`
readQueries := []string{
`Range(f=1, 1999-12-31T00:00, 2002-01-01T03:00)`,
`Row(f=1, from=1999-12-31T00:00, to=2002-01-01T03:00)`,
`Clear("two", f=1)`,
`Range(f=1, 1999-12-31T00:00, 2002-01-01T03:00)`,
`Row(f=1, from=1999-12-31T00:00, to=2002-01-01T03:00)`,
}
responses := runCallTest(t, writeQuery, readQueries,
&pilosa.IndexOptions{Keys: true},
@ -1559,9 +1559,9 @@ func TestExecutor_Execute_Range(t *testing.T) {
Set(2, f="foo", 2002-02-01T00:00)
Set(2, f="bar", 2001-01-01T00:00)`
readQueries := []string{
`Range(f="foo", 1999-12-31T00:00, 2002-01-01T03:00)`,
`Row(f="foo", from=1999-12-31T00:00, to=2002-01-01T03:00)`,
`Clear( 2, f="foo")`,
`Range(f="foo", 1999-12-31T00:00, 2002-01-01T03:00)`,
`Row(f="foo", from=1999-12-31T00:00, to=2002-01-01T03:00)`,
}
responses := runCallTest(t, writeQuery, readQueries,
nil,
@ -1594,9 +1594,182 @@ func TestExecutor_Execute_Range(t *testing.T) {
Set("two", f="foo", 2002-02-01T00:00)
Set("two", f="bar", 2001-01-01T00:00)`
readQueries := []string{
`Range(f="foo", 1999-12-31T00:00, 2002-01-01T03:00)`,
`Row(f="foo", from=1999-12-31T00:00, to=2002-01-01T03:00)`,
`Clear("two", f="foo")`,
`Range(f="foo", 1999-12-31T00:00, 2002-01-01T03:00)`,
`Row(f="foo", from=1999-12-31T00:00, to=2002-01-01T03:00)`,
}
responses := runCallTest(t, writeQuery, readQueries,
&pilosa.IndexOptions{Keys: true},
pilosa.OptFieldTypeTime(pilosa.TimeQuantum("YMDH")),
pilosa.OptFieldKeys())
t.Run("Standard", func(t *testing.T) {
if keys := responses[0].Results[0].(*pilosa.Row).Keys; !reflect.DeepEqual(keys, []string{"two", "three", "four", "five", "six", "seven"}) {
t.Fatalf("unexpected keys: %+v", keys)
}
})
t.Run("Clear", func(t *testing.T) {
if keys := responses[2].Results[0].(*pilosa.Row).Keys; !reflect.DeepEqual(keys, []string{"three", "four", "five", "six", "seven"}) {
t.Fatalf("unexpected keys: %+v", keys)
}
})
})
t.Run("UnixTimestamp", func(t *testing.T) {
writeQuery := `
Set(2, f=1, 1999-12-31T00:00)
Set(3, f=1, 2000-01-01T00:00)
Set(4, f=1, 2000-01-02T00:00)
Set(5, f=1, 2000-02-01T00:00)
Set(6, f=1, 2001-01-01T00:00)
Set(7, f=1, 2002-01-01T02:00)
Set(2, f=1, 1999-12-30T00:00)
Set(2, f=1, 2002-02-01T00:00)
Set(2, f=10, 2001-01-01T00:00)`
readQueries := []string{
`Row(f=1, from=946598400, to=1009854000)`,
`Clear( 2, f=1)`,
`Row(f=1, from=946598400, to=1009854000)`,
}
responses := runCallTest(t, writeQuery, readQueries,
nil, pilosa.OptFieldTypeTime(pilosa.TimeQuantum("YMDH")))
t.Run("Standard", func(t *testing.T) {
if columns := responses[0].Results[0].(*pilosa.Row).Columns(); !reflect.DeepEqual(columns, []uint64{2, 3, 4, 5, 6, 7}) {
t.Fatalf("unexpected columns: %+v", columns)
}
})
t.Run("Clear", func(t *testing.T) {
if columns := responses[2].Results[0].(*pilosa.Row).Columns(); !reflect.DeepEqual(columns, []uint64{3, 4, 5, 6, 7}) {
t.Fatalf("unexpected columns: %+v", columns)
}
})
})
}
// Ensure a range query can be executed.
func TestExecutor_Execute_Range_Deprecated(t *testing.T) {
t.Run("RowIDColumnID", func(t *testing.T) {
writeQuery := `
Set(2, f=1, 1999-12-31T00:00)
Set(3, f=1, 2000-01-01T00:00)
Set(4, f=1, 2000-01-02T00:00)
Set(5, f=1, 2000-02-01T00:00)
Set(6, f=1, 2001-01-01T00:00)
Set(7, f=1, 2002-01-01T02:00)
Set(2, f=1, 1999-12-30T00:00)
Set(2, f=1, 2002-02-01T00:00)
Set(2, f=10, 2001-01-01T00:00)`
readQueries := []string{
`Range(f=1, from=1999-12-31T00:00, to=2002-01-01T03:00)`,
`Clear( 2, f=1)`,
`Range(f=1, from=1999-12-31T00:00, to=2002-01-01T03:00)`,
}
responses := runCallTest(t, writeQuery, readQueries,
nil, pilosa.OptFieldTypeTime(pilosa.TimeQuantum("YMDH")))
t.Run("Standard", func(t *testing.T) {
if columns := responses[0].Results[0].(*pilosa.Row).Columns(); !reflect.DeepEqual(columns, []uint64{2, 3, 4, 5, 6, 7}) {
t.Fatalf("unexpected columns: %+v", columns)
}
})
t.Run("Clear", func(t *testing.T) {
if columns := responses[2].Results[0].(*pilosa.Row).Columns(); !reflect.DeepEqual(columns, []uint64{3, 4, 5, 6, 7}) {
t.Fatalf("unexpected columns: %+v", columns)
}
})
})
t.Run("RowIDColumnKey", func(t *testing.T) {
writeQuery := `
Set("two", f=1, 1999-12-31T00:00)
Set("three", f=1, 2000-01-01T00:00)
Set("four", f=1, 2000-01-02T00:00)
Set("five", f=1, 2000-02-01T00:00)
Set("six", f=1, 2001-01-01T00:00)
Set("seven", f=1, 2002-01-01T02:00)
Set("two", f=1, 1999-12-30T00:00)
Set("two", f=1, 2002-02-01T00:00)
Set("two", f=10, 2001-01-01T00:00)`
readQueries := []string{
`Range(f=1, from=1999-12-31T00:00, to=2002-01-01T03:00)`,
`Clear("two", f=1)`,
`Range(f=1, from=1999-12-31T00:00, to=2002-01-01T03:00)`,
}
responses := runCallTest(t, writeQuery, readQueries,
&pilosa.IndexOptions{Keys: true},
pilosa.OptFieldTypeTime(pilosa.TimeQuantum("YMDH")))
t.Run("Standard", func(t *testing.T) {
if keys := responses[0].Results[0].(*pilosa.Row).Keys; !reflect.DeepEqual(keys, []string{"two", "three", "four", "five", "six", "seven"}) {
t.Fatalf("unexpected keys: %+v", keys)
}
})
t.Run("Clear", func(t *testing.T) {
if keys := responses[2].Results[0].(*pilosa.Row).Keys; !reflect.DeepEqual(keys, []string{"three", "four", "five", "six", "seven"}) {
t.Fatalf("unexpected keys: %+v", keys)
}
})
})
t.Run("RowKeyColumnID", func(t *testing.T) {
writeQuery := `
Set(2, f="foo", 1999-12-31T00:00)
Set(3, f="foo", 2000-01-01T00:00)
Set(4, f="foo", 2000-01-02T00:00)
Set(5, f="foo", 2000-02-01T00:00)
Set(6, f="foo", 2001-01-01T00:00)
Set(7, f="foo", 2002-01-01T02:00)
Set(2, f="foo", 1999-12-30T00:00)
Set(2, f="foo", 2002-02-01T00:00)
Set(2, f="bar", 2001-01-01T00:00)`
readQueries := []string{
`Range(f="foo", from=1999-12-31T00:00, to=2002-01-01T03:00)`,
`Clear( 2, f="foo")`,
`Range(f="foo", from=1999-12-31T00:00, to=2002-01-01T03:00)`,
}
responses := runCallTest(t, writeQuery, readQueries,
nil,
pilosa.OptFieldTypeTime(pilosa.TimeQuantum("YMDH")),
pilosa.OptFieldKeys())
t.Run("Standard", func(t *testing.T) {
if columns := responses[0].Results[0].(*pilosa.Row).Columns(); !reflect.DeepEqual(columns, []uint64{2, 3, 4, 5, 6, 7}) {
t.Fatalf("unexpected columns: %+v", columns)
}
})
t.Run("Clear", func(t *testing.T) {
if columns := responses[2].Results[0].(*pilosa.Row).Columns(); !reflect.DeepEqual(columns, []uint64{3, 4, 5, 6, 7}) {
t.Fatalf("unexpected columns: %+v", columns)
}
})
})
t.Run("RowKeyColumnKey", func(t *testing.T) {
writeQuery := `
Set("two", f="foo", 1999-12-31T00:00)
Set("three", f="foo", 2000-01-01T00:00)
Set("four", f="foo", 2000-01-02T00:00)
Set("five", f="foo", 2000-02-01T00:00)
Set("six", f="foo", 2001-01-01T00:00)
Set("seven", f="foo", 2002-01-01T02:00)
Set("two", f="foo", 1999-12-30T00:00)
Set("two", f="foo", 2002-02-01T00:00)
Set("two", f="bar", 2001-01-01T00:00)`
readQueries := []string{
`Range(f="foo", from=1999-12-31T00:00, to=2002-01-01T03:00)`,
`Clear("two", f="foo")`,
`Range(f="foo", from=1999-12-31T00:00, to=2002-01-01T03:00)`,
}
responses := runCallTest(t, writeQuery, readQueries,
&pilosa.IndexOptions{Keys: true},
@ -1617,8 +1790,174 @@ func TestExecutor_Execute_Range(t *testing.T) {
})
}
// Ensure a Range(bsiGroup) query can be executed.
func TestExecutor_Execute_BSIGroupRange(t *testing.T) {
// Ensure a Row(bsiGroup) query can be executed.
func TestExecutor_Execute_Row_BSIGroup(t *testing.T) {
c := test.MustRunCluster(t, 1)
defer c.Close()
hldr := test.Holder{Holder: c[0].Server.Holder()}
idx, err := hldr.CreateIndex("i", pilosa.IndexOptions{})
if err != nil {
t.Fatal(err)
}
if _, err := idx.CreateField("f", pilosa.OptFieldTypeDefault()); err != nil {
t.Fatal(err)
}
if _, err := idx.CreateField("foo", pilosa.OptFieldTypeInt(10, 100)); err != nil {
t.Fatal(err)
}
if _, err := idx.CreateField("bar", pilosa.OptFieldTypeInt(0, 100000)); err != nil {
t.Fatal(err)
}
if _, err := idx.CreateField("other", pilosa.OptFieldTypeInt(0, 1000)); err != nil {
t.Fatal(err)
}
if _, err := idx.CreateField("edge", pilosa.OptFieldTypeInt(-100, 100)); err != nil {
t.Fatal(err)
}
if _, err := c[0].API.Query(context.Background(), &pilosa.QueryRequest{Index: "i", Query: `
Set(0, f=0)
Set(` + strconv.Itoa(ShardWidth+1) + `, f=0)
Set(50, foo=20)
Set(50, bar=2000)
Set(` + strconv.Itoa(ShardWidth) + `, foo=30)
Set(` + strconv.Itoa(ShardWidth+2) + `, foo=10)
Set(` + strconv.Itoa((5*ShardWidth)+100) + `, foo=20)
Set(` + strconv.Itoa(ShardWidth+1) + `, foo=60)
Set(0, other=1000)
Set(0, edge=100)
Set(1, edge=-100)
`}); err != nil {
t.Fatal(err)
}
t.Run("EQ", func(t *testing.T) {
if result, err := c[0].API.Query(context.Background(), &pilosa.QueryRequest{Index: "i", Query: `Row(foo == 20)`}); err != nil {
t.Fatal(err)
} else if !reflect.DeepEqual([]uint64{50, (5 * ShardWidth) + 100}, result.Results[0].(*pilosa.Row).Columns()) {
t.Fatalf("unexpected result: %s", spew.Sdump(result))
}
})
t.Run("NEQ", func(t *testing.T) {
// NEQ null
if result, err := c[0].API.Query(context.Background(), &pilosa.QueryRequest{Index: "i", Query: `Row(other != null)`}); err != nil {
t.Fatal(err)
} else if !reflect.DeepEqual([]uint64{0}, result.Results[0].(*pilosa.Row).Columns()) {
t.Fatalf("unexpected result: %s", spew.Sdump(result))
}
// NEQ <int>
if result, err := c[0].API.Query(context.Background(), &pilosa.QueryRequest{Index: "i", Query: `Row(foo != 20)`}); err != nil {
t.Fatal(err)
} else if !reflect.DeepEqual([]uint64{ShardWidth, ShardWidth + 1, ShardWidth + 2}, result.Results[0].(*pilosa.Row).Columns()) {
t.Fatalf("unexpected result: %s", spew.Sdump(result))
}
// NEQ -<int>
if result, err := c[0].API.Query(context.Background(), &pilosa.QueryRequest{Index: "i", Query: `Row(other != -20)`}); err != nil {
t.Fatal(err)
} else if !reflect.DeepEqual([]uint64{0}, result.Results[0].(*pilosa.Row).Columns()) {
//t.Fatalf("unexpected result: %s", spew.Sdump(result))
t.Fatalf("unexpected result: %v", result.Results[0].(*pilosa.Row).Columns())
}
})
t.Run("LT", func(t *testing.T) {
if result, err := c[0].API.Query(context.Background(), &pilosa.QueryRequest{Index: "i", Query: `Row(foo < 20)`}); err != nil {
t.Fatal(err)
} else if !reflect.DeepEqual([]uint64{ShardWidth + 2}, result.Results[0].(*pilosa.Row).Columns()) {
t.Fatalf("unexpected result: %s", spew.Sdump(result))
}
})
t.Run("LTE", func(t *testing.T) {
if result, err := c[0].API.Query(context.Background(), &pilosa.QueryRequest{Index: "i", Query: `Row(foo <= 20)`}); err != nil {
t.Fatal(err)
} else if !reflect.DeepEqual([]uint64{50, ShardWidth + 2, (5 * ShardWidth) + 100}, result.Results[0].(*pilosa.Row).Columns()) {
t.Fatalf("unexpected result: %s", spew.Sdump(result))
}
})
t.Run("GT", func(t *testing.T) {
if result, err := c[0].API.Query(context.Background(), &pilosa.QueryRequest{Index: "i", Query: `Row(foo > 20)`}); err != nil {
t.Fatal(err)
} else if !reflect.DeepEqual([]uint64{ShardWidth, ShardWidth + 1}, result.Results[0].(*pilosa.Row).Columns()) {
t.Fatalf("unexpected result: %s", spew.Sdump(result))
}
})
t.Run("GTE", func(t *testing.T) {
if result, err := c[0].API.Query(context.Background(), &pilosa.QueryRequest{Index: "i", Query: `Row(foo >= 20)`}); err != nil {
t.Fatal(err)
} else if !reflect.DeepEqual([]uint64{50, ShardWidth, ShardWidth + 1, (5 * ShardWidth) + 100}, result.Results[0].(*pilosa.Row).Columns()) {
t.Fatalf("unexpected result: %s", spew.Sdump(result))
}
})
t.Run("BETWEEN", func(t *testing.T) {
if result, err := c[0].API.Query(context.Background(), &pilosa.QueryRequest{Index: "i", Query: `Row(0 < other < 1000)`}); err != nil {
t.Fatal(err)
} else if !reflect.DeepEqual([]uint64{0}, result.Results[0].(*pilosa.Row).Columns()) {
t.Fatalf("unexpected result: %s", spew.Sdump(result))
}
})
// Ensure that the NotNull code path gets run.
t.Run("NotNull", func(t *testing.T) {
if result, err := c[0].API.Query(context.Background(), &pilosa.QueryRequest{Index: "i", Query: `Row(-1 < other < 1000)`}); err != nil {
t.Fatal(err)
} else if !reflect.DeepEqual([]uint64{0}, result.Results[0].(*pilosa.Row).Columns()) {
t.Fatalf("unexpected result: %s", spew.Sdump(result))
}
})
t.Run("BelowMin", func(t *testing.T) {
if result, err := c[0].API.Query(context.Background(), &pilosa.QueryRequest{Index: "i", Query: `Row(foo == 0)`}); err != nil {
t.Fatal(err)
} else if !reflect.DeepEqual([]uint64{}, result.Results[0].(*pilosa.Row).Columns()) {
t.Fatalf("unexpected result: %s", spew.Sdump(result))
}
})
t.Run("AboveMax", func(t *testing.T) {
if result, err := c[0].API.Query(context.Background(), &pilosa.QueryRequest{Index: "i", Query: `Row(foo == 200)`}); err != nil {
t.Fatal(err)
} else if !reflect.DeepEqual([]uint64{}, result.Results[0].(*pilosa.Row).Columns()) {
t.Fatalf("unexpected result: %s", spew.Sdump(result))
}
})
t.Run("LTAboveMax", func(t *testing.T) {
if result, err := c[0].API.Query(context.Background(), &pilosa.QueryRequest{Index: "i", Query: `Row(edge < 200)`}); err != nil {
t.Fatal(err)
} else if !reflect.DeepEqual([]uint64{0, 1}, result.Results[0].(*pilosa.Row).Columns()) {
t.Fatalf("unexpected result: %s", spew.Sdump(result.Results[0].(*pilosa.Row).Columns()))
}
})
t.Run("GTBelowMin", func(t *testing.T) {
if result, err := c[0].API.Query(context.Background(), &pilosa.QueryRequest{Index: "i", Query: `Row(edge > -200)`}); err != nil {
t.Fatal(err)
} else if !reflect.DeepEqual([]uint64{0, 1}, result.Results[0].(*pilosa.Row).Columns()) {
t.Fatalf("unexpected result: %s", spew.Sdump(result.Results[0].(*pilosa.Row).Columns()))
}
})
t.Run("ErrFieldNotFound", func(t *testing.T) {
if _, err := c[0].API.Query(context.Background(), &pilosa.QueryRequest{Index: "i", Query: `Row(bad_field >= 20)`}); errors.Cause(err) != pilosa.ErrFieldNotFound {
t.Fatal(err)
}
})
}
// Ensure a Range(bsiGroup) query can be executed. (Deprecated)
func TestExecutor_Execute_Range_BSIGroup_Deprecated(t *testing.T) {
c := test.MustRunCluster(t, 1)
defer c.Close()
hldr := test.Holder{Holder: c[0].Server.Holder()}
@ -2010,7 +2349,7 @@ func TestExecutor_Time_Clear_Quantums(t *testing.T) {
Set(2, f=10, 2001-01-01T00:00)
`
clearColumn := `Clear( 2, f=1)`
rangeCheckQuery := `Range(f=1, 1999-12-31T00:00, 2002-01-01T03:00)`
rangeCheckQuery := `Row(f=1, from=1999-12-31T00:00, to=2002-01-01T03:00)`
for i, tt := range rangeTests {
t.Run(fmt.Sprintf("#%d Quantum %s", i+1, tt.quantum), func(t *testing.T) {
@ -2346,10 +2685,10 @@ func TestExecutor_Execute_ClearRow(t *testing.T) {
Set(2, f=1, 2002-02-01T00:00)
Set(2, f=10, 2001-01-01T00:00)`
readQueries := []string{
`Range(f=1, 1999-12-31T00:00, 2003-01-01T03:00)`,
`Row(f=1, from=1999-12-31T00:00, to=2003-01-01T03:00)`,
`ClearRow(f=1)`,
`Range(f=1, 1999-12-31T00:00, 2003-01-01T03:00)`,
`Range(f=10, 1999-12-31T00:00, 2003-01-01T03:00)`,
`Row(f=1, from=1999-12-31T00:00, to=2003-01-01T03:00)`,
`Row(f=10, from=1999-12-31T00:00, to=2003-01-01T03:00)`,
}
responses := runCallTest(t, writeQuery, readQueries,
&pilosa.IndexOptions{TrackExistence: true},
@ -3220,6 +3559,8 @@ func BenchmarkGroupBy(b *testing.B) {
}
func runCallTest(t *testing.T, writeQuery string, readQueries []string, indexOptions *pilosa.IndexOptions, fieldOption ...pilosa.FieldOption) []pilosa.QueryResponse {
t.Helper()
if indexOptions == nil {
indexOptions = &pilosa.IndexOptions{}
}

View file

@ -710,9 +710,9 @@ func TestClient_ImportKeys(t *testing.T) {
t.Fatalf("unexpected values: got sum=%v, count=%v; expected sum=50, cnt=3", sum, cnt)
}
// Verify Range.
// Verify range.
queryRequest := &pilosa.QueryRequest{
Query: fmt.Sprintf(`Range(%s>10)`, fldName),
Query: fmt.Sprintf(`Row(%s>10)`, fldName),
Remote: false,
}
@ -743,7 +743,7 @@ func TestClient_ImportKeys(t *testing.T) {
// Verify Range.
queryRequest = &pilosa.QueryRequest{
Query: fmt.Sprintf(`Range(%s>10)`, fldName),
Query: fmt.Sprintf(`Row(%s>10)`, fldName),
Remote: false,
}

View file

@ -255,13 +255,25 @@ type Call struct {
// Returns the field as a string if present, or an error if not.
func (c *Call) FieldArg() (string, error) {
for arg := range c.Args {
if !strings.HasPrefix(arg, "_") {
if !IsReservedArg(arg) {
return arg, nil
}
}
return "", fmt.Errorf("No field argument specified")
}
func IsReservedArg(name string) bool {
if strings.HasPrefix(name, "_") {
return true
}
switch name {
case "from", "to":
return true
default:
return false
}
}
// BoolArg is for reading the value at key from call.Args as a bool. If the
// key is not in Call.Args, the value of the returned bool will be false, and
// the error will be nil. The value is assumed to be a bool. An error is

View file

@ -13,12 +13,12 @@ Call <- 'Set' {p.startCall("Set")} open col comma args (comma timestamp)? close
/ 'ClearRow' {p.startCall("ClearRow")} open arg close {p.endCall()}
/ 'Store' {p.startCall("Store")} open Call comma arg close {p.endCall()}
/ 'TopN' {p.startCall("TopN")} open posfield (comma allargs)? close {p.endCall()}
/ 'Range' {p.startCall("Range")} open (timerange / conditional / arg) close {p.endCall()}
/ < IDENT > { p.startCall(buffer[begin:end] ) } open allargs comma? close { p.endCall() }
allargs <- Call (comma Call)* (comma args)? / args / sp
args <- arg (comma args)? sp
arg <- ( field sp '=' sp value
/ field sp COND sp value
/ conditional
)
COND <- ( '><' { p.addBTWN() }
/ '<=' { p.addLTE() }
@ -28,13 +28,12 @@ COND <- ( '><' { p.addBTWN() }
/ '<' { p.addLT() }
/ '>' { p.addGT() }
)
conditional <- {p.startConditional()} condint condLT condfield condLT condint {p.endConditional()}
condint <- <'-'? [1-9] [0-9]* / '0'> sp {p.condAdd(buffer[begin:end])}
condLT <- <('<=' / '<')> sp {p.condAdd(buffer[begin:end])}
condfield <- <fieldExpr> sp {p.condAdd(buffer[begin:end])}
timerange <- field sp '=' sp value comma <timestampfmt> {p.addPosStr("_start", buffer[begin:end])} comma <timestampfmt> {p.addPosStr("_end", buffer[begin:end])}
value <- ( item
/ lbrack { p.startList() } list rbrack { p.endList() }
)
@ -42,6 +41,7 @@ list <- item (comma list)?
item <- ( 'null' &(comma / sp close) { p.addVal(nil) }
/ 'true' &(comma / sp close) { p.addVal(true) }
/ 'false' &(comma / sp close) { p.addVal(false) }
/ timestampfmt { p.addVal(buffer[begin:end]) }
/ < '-'? [0-9]+ ('.'[0-9]*)? > { p.addNumVal(buffer[begin:end]) }
/ < '-'? '.'[0-9]+ > { p.addNumVal(buffer[begin:end]) }
/ < IDENT > { p.startCall(buffer[begin:end]) } open allargs comma? close { p.addVal(p.endCall()) }
@ -77,5 +77,5 @@ IDENT <- [[A-Z]] ([[A-Z]] / [0-9])*
timestampbasicfmt <- [0-9][0-9][0-9][0-9]'-'[01][0-9]'-'[0-3][0-9]'T'[0-9][0-9]':'[0-9][0-9]
timestampfmt <- '"' timestampbasicfmt '"' / '\'' timestampbasicfmt '\'' / timestampbasicfmt
timestampfmt <- '"' <timestampbasicfmt> '"' / '\'' <timestampbasicfmt> '\'' / <timestampbasicfmt>
timestamp <- <timestampfmt> {p.addPosStr("_timestamp", buffer[begin:end])}

File diff suppressed because it is too large Load diff

View file

@ -8,7 +8,7 @@ import (
func TestPEG(t *testing.T) {
p := PQL{Buffer: `
SetBit(Union(Zitmap(row==4), Intersect(Qitmap(blah>4), Ritmap(field="http://zoo9.com=\\'hello' and \"hello\"")), Hitmap(row=ag-bee)), a="4z", b=5) Count(Union(Witmap(row=5.73, frame=.10), Range(zztop><[2, 9]))) TopN(blah, fields=["hello", "goodbye", "zero"])`[1:]}
SetBit(Union(Zitmap(row==4), Intersect(Qitmap(blah>4), Ritmap(field="http://zoo9.com=\\'hello' and \"hello\"")), Hitmap(row=ag-bee)), a="4z", b=5) Count(Union(Witmap(row=5.73, frame=.10), Row(zztop><[2, 9]))) TopN(blah, fields=["hello", "goodbye", "zero"])`[1:]}
p.Init()
err := p.Parse()
if err != nil {
@ -202,51 +202,51 @@ func TestPEGWorking(t *testing.T) {
ncalls: 1},
{
name: "RangeLT",
input: "Range(a < 4)",
input: "Row(a < 4)",
ncalls: 1},
{
name: "RangeGT",
input: "Range(a > 4)",
input: "Row(a > 4)",
ncalls: 1},
{
name: "RangeLTE",
input: "Range(a <= 4)",
input: "Row(a <= 4)",
ncalls: 1},
{
name: "RangeGTE",
input: "Range(a >= 4)",
input: "Row(a >= 4)",
ncalls: 1},
{
name: "RangeEQ",
input: "Range(a == 4)",
input: "Row(a == 4)",
ncalls: 1},
{
name: "RangeNEQ",
input: "Range(a != null)",
input: "Row(a != null)",
ncalls: 1},
{
name: "RangeLTLT",
input: "Range(4 < a < 9)",
input: "Row(4 < a < 9)",
ncalls: 1},
{
name: "RangeLTLTE",
input: "Range(4 < a <= 9)",
input: "Row(4 < a <= 9)",
ncalls: 1},
{
name: "RangeLTELT",
input: "Range(4 <= a < 9)",
input: "Row(4 <= a < 9)",
ncalls: 1},
{
name: "RangeLTELTE",
input: "Range(4 <= a <= 9)",
input: "Row(4 <= a <= 9)",
ncalls: 1},
{
name: "RangeTime",
input: "Range(a=4, 2010-07-04T00:00, 2010-08-04T00:00)",
input: "Row(a=4, from=2010-07-04T00:00, to=2010-08-04T00:00)",
ncalls: 1},
{
name: "RangeTimeQuotes",
input: `Range(a=4, '2010-07-04T00:00', "2010-08-04T00:00")`,
input: `Row(a=4, from='2010-07-04T00:00', to="2010-08-04T00:00")`,
ncalls: 1},
{
name: "Dashed Frame",
@ -302,10 +302,10 @@ func TestPEGErrors(t *testing.T) {
input: "Clear(9)"},
{
name: "RangeTimeGT",
input: "Range(a>4, 2010-07-04T00:00, 2010-08-04T00:00)"},
input: "Row(a>4, 2010-07-04T00:00, 2010-08-04T00:00)"},
{
name: "RangeTimeOneStamp",
input: "Range(a=4, 2010-07-04T00:00)"},
input: "Row(a=4, 2010-07-04T00:00)"},
}
for i, test := range tests {
@ -423,9 +423,9 @@ func TestPQLDeepEquality(t *testing.T) {
}},
{
name: "RangeEQ",
call: "Range(a==7)",
call: "Row(a==7)",
exp: &Call{
Name: "Range",
Name: "Row",
Args: map[string]interface{}{
"a": &Condition{
Op: EQ,
@ -435,9 +435,9 @@ func TestPQLDeepEquality(t *testing.T) {
}},
{
name: "RangeLT",
call: "Range(a<7)",
call: "Row(a<7)",
exp: &Call{
Name: "Range",
Name: "Row",
Args: map[string]interface{}{
"a": &Condition{
Op: LT,
@ -447,9 +447,9 @@ func TestPQLDeepEquality(t *testing.T) {
}},
{
name: "RangeLTE",
call: "Range(a<=7)",
call: "Row(a<=7)",
exp: &Call{
Name: "Range",
Name: "Row",
Args: map[string]interface{}{
"a": &Condition{
Op: LTE,
@ -459,9 +459,9 @@ func TestPQLDeepEquality(t *testing.T) {
}},
{
name: "RangeGTE",
call: "Range(a>=7)",
call: "Row(a>=7)",
exp: &Call{
Name: "Range",
Name: "Row",
Args: map[string]interface{}{
"a": &Condition{
Op: GTE,
@ -471,9 +471,9 @@ func TestPQLDeepEquality(t *testing.T) {
}},
{
name: "RangeGT",
call: "Range(a>7)",
call: "Row(a>7)",
exp: &Call{
Name: "Range",
Name: "Row",
Args: map[string]interface{}{
"a": &Condition{
Op: GT,
@ -483,9 +483,9 @@ func TestPQLDeepEquality(t *testing.T) {
}},
{
name: "RangeNEQ",
call: "Range(a!=null)",
call: "Row(a!=null)",
exp: &Call{
Name: "Range",
Name: "Row",
Args: map[string]interface{}{
"a": &Condition{
Op: NEQ,
@ -495,9 +495,9 @@ func TestPQLDeepEquality(t *testing.T) {
}},
{
name: "RangeLTELT",
call: "Range(4 <= a < 9)",
call: "Row(4 <= a < 9)",
exp: &Call{
Name: "Range",
Name: "Row",
Args: map[string]interface{}{
"a": &Condition{
Op: BETWEEN,
@ -507,9 +507,9 @@ func TestPQLDeepEquality(t *testing.T) {
}},
{
name: "RangeLTLT",
call: "Range(4 < a < 9)",
call: "Row(4 < a < 9)",
exp: &Call{
Name: "Range",
Name: "Row",
Args: map[string]interface{}{
"a": &Condition{
Op: BETWEEN,
@ -519,9 +519,9 @@ func TestPQLDeepEquality(t *testing.T) {
}},
{
name: "RangeLTELTE",
call: "Range(4 <= a <= 9)",
call: "Row(4 <= a <= 9)",
exp: &Call{
Name: "Range",
Name: "Row",
Args: map[string]interface{}{
"a": &Condition{
Op: BETWEEN,
@ -531,9 +531,9 @@ func TestPQLDeepEquality(t *testing.T) {
}},
{
name: "RangeLTLTE",
call: "Range(4 < a <= 9)",
call: "Row(4 < a <= 9)",
exp: &Call{
Name: "Range",
Name: "Row",
Args: map[string]interface{}{
"a": &Condition{
Op: BETWEEN,