min/max documentation

This commit is contained in:
Travis Turner 2018-04-10 18:01:05 -05:00
parent 16539bbab1
commit ee4dbbf328
No known key found for this signature in database
GPG key ID: 7F08008DFD9314C9
5 changed files with 101 additions and 5 deletions

View file

@ -112,7 +112,7 @@ SetBit(frame="A", rowID=8, columnID=3, timestamp="2017-05-19T00:00")
#### BSI Range-Encoding
Bit-Sliced Indexing (BSI) is the storage method Pilosa uses to represent multi-bit integers in a bitmap index. Integers are stored as n-bit, range-encoded
bit-sliced indexes of base-2, along with an additional bitmap indicating "not null". This means that a 16-bit integer will require 17 bitmaps: one for each 0-bit of the 16 bit-slice components (the 1-bit does not need to be stored because with range-encoding the highest bit position is always 1) and one for the non-null bitmap. Pilosa can evaluate `Sum` and `Range` queries on these BSI integers.
bit-sliced indexes of base-2, along with an additional bitmap indicating "not null". This means that a 16-bit integer will require 17 bitmaps: one for each 0-bit of the 16 bit-slice components (the 1-bit does not need to be stored because with range-encoding the highest bit position is always 1) and one for the non-null bitmap. Pilosa can evaluate `Range`, `Min`, `Max`, and `Sum` queries on these BSI integers.
Internally Pilosa stores each BSI `field` as a `view` within a `frame`. The 'rowIDs' of the `view` are composed of the base-2 representation of the integer. Pilosa manages the base-2 offset and translation that efficiently packs the integer value within the minimum set of rows.

View file

@ -14,7 +14,7 @@ nav = []
<strong id="bitmap">[Bitmap](../data-model/#overview):</strong> The on-disk and in-memory representation of a [row](#row). Implemented with [Roaring](#roaring-bitmap). `Bitmap` is also the basic [PQL](#pql) query for reading a Bitmap.
<strong id="bsi">[BSI](../data-model/#bsi-range-encoding)</strong> Bit-sliced indexing is the method Pilosa uses to represent multi-bit integers. Integer values are stored in [fields](#field), and can be used for [Range](#range-bsi) and [Sum](#sum) queries.
<strong id="bsi">[BSI](../data-model/#bsi-range-encoding)</strong> Bit-sliced indexing is the method Pilosa uses to represent multi-bit integers. Integer values are stored in [fields](#field), and can be used for [Range](#range-bsi), [Min](#min), [Max](#max), and [Sum](#sum) queries.
<strong id="cluster">Cluster:</strong> A cluster consists of one or more [nodes](#node) which share a cluster configuration. The cluster also defines how data is [replicated](#replica) throughout and how internode communication is coordinated. Pilosa does not have a leader node, all data is evenly distributed, and any node can respond to queries.
@ -30,8 +30,12 @@ nav = []
<strong id="jump-consistent-hash">[Jump Consistent Hash](https://arxiv.org/pdf/1406.2294v1.pdf):</strong> A fast, minimal memory, consistent hash algorithm that evenly distributes the workload even when the number of buckets changes.
<strong id="max">[Max](../query-language/#max):</strong> A [PQL](#pql) query that returns the maximum integer value stored in [BSI](#bsi) [fields](#field).
<strong id="maxslice">MaxSlice:</strong> The total number of [slices](#slice) allocated to handle the current set of [columns](#column). This value is important for all [nodes](#node) to efficiently distribute queries.
<strong id="min">[Min](../query-language/#min):</strong> A [PQL](#pql) query that returns the minimum integer value stored in [BSI](#bsi) [fields](#field).
<strong id="node">Node:</strong> An individual running instance of Pilosa server which belongs to a [cluster](#cluster).
<strong id="partition">Partition:</strong> The [consistent hash](#jump-consistent-hash) maps keys to partitions (or locations on the unit circle), based on a preset maximum number of partitions. Partitions are then evenly mapped to physical [nodes](#node). To add nodes to the [cluster](#cluster), the partitions must be remapped, and data is then associated across the new cluster topology. `DefaultPartitionN` is 256. It can be modified, but only at compile time, and before ingesting any data.

View file

@ -523,6 +523,56 @@ Range(frame="stats", commitactivity >< [100, 200])
This is conceptually equivalent to the interval 100 <= commitactivity <= 200, but this chained comparison syntax is not currently supported. `BETWEEN` query syntax is restricted to greater-than-or-equal-to and less-than-or-equal-to, but any valid interval on the integers can be represented this way.
#### Min
**Spec:**
```
Min([BITMAP_CALL], <frame=STRING>, <field=STRING>)
```
**Description:**
Returns the minimum value of all BSI integer values in the `field` in this `frame`. If the optional `Bitmap` call is supplied, only columns with set bits are considered, otherwise all collumns are considered.
**Result Type:** object with the min and count of columns containing the min value.
**Examples:**
Query the size of all repositories.
```
Min(frame="stats", field="diskusage")
```
Return `{"min":4,"count":2}`
* Result is the smallest repository in kilobytes, plus the number of repositories of that size.
#### Max
**Spec:**
```
Max([BITMAP_CALL], <frame=STRING>, <field=STRING>)
```
**Description:**
Returns the maximum value of all BSI integer values in the `field` in this `frame`. If the optional `Bitmap` call is supplied, only columns with set bits are considered, otherwise all columns are considered.
**Result Type:** object with the max and count of columns containing the max value.
**Examples:**
Query the size of all repositories.
```
Max(frame="stats", field="diskusage")
```
Return `{"max":88,"count":13}`
* Result is the largest repository in kilobytes, plus the number of repositories of that size.
#### Sum
**Spec:**

View file

@ -247,7 +247,7 @@ Check out our [Administration Guide](https://www.pilosa.com/docs/latest/administ
#### Introduction
Pilosa can store integer values associated to the columns in an index, and those values are used to support `Range` and `Sum` queries. In this tutorial we will show how to set up integer fields, populate those fields with data, and query the fields. The example index we're going to create will represent fictional patients at a medical facility and various bits of information about those patients.
Pilosa can store integer values associated to the columns in an index, and those values are used to support `Range`, `Min`, `Max`, and `Sum` queries. In this tutorial we will show how to set up integer fields, populate those fields with data, and query the fields. The example index we're going to create will represent fictional patients at a medical facility and various bits of information about those patients.
First, create an index called `patients`:
``` request
@ -333,7 +333,7 @@ curl localhost:10101/index/patients/query \
```
The results you get from the `Sum` query contain the `sum` of all values as well as the `count` of columns with a value. To get the average you can just divide `sum` by `count`.
You can also provide a filter to the `Sum()` function, to find the average age of all patients over 40.
You can also provide a filter to the `Sum()` function to find the average age of all patients over 40.
``` request
curl localhost:10101/index/patients/query \
-X POST \
@ -344,6 +344,48 @@ curl localhost:10101/index/patients/query \
```
Notice in this case that the count is only `3` because of the `age > 40` filter applied to the query.
To find the minimum age of all patients, run a `Min` query:
``` request
curl localhost:10101/index/patients/query \
-X POST \
-d 'Min(frame="measurements", field="age")'
```
``` response
{"results":[{"min":19,"count":1}]}
```
The results you get from the `Min` query contain the `min` of all values as well as the `count` of columns with that value.
You can also provide a filter to the `Min()` function to find the minimum age of all patients over 40.
``` request
curl localhost:10101/index/patients/query \
-X POST \
-d 'Min(Range(frame="measurements", age > 40), frame="measurements", field="age")'
```
``` response
{"results":[{"min":57,"count":1}]}
```
To find the maximum age of all patients, run a `Max` query:
``` request
curl localhost:10101/index/patients/query \
-X POST \
-d 'Max(frame="measurements", field="age")'
```
``` response
{"results":[{"max":71,"count":1}]}
```
The results you get from the `Max` query contain the `max` of all values as well as the `count` of columns with that value.
You can also provide a filter to the `Max()` function to find the maximum age of all patients under 40.
``` request
curl localhost:10101/index/patients/query \
-X POST \
-d 'Max(Range(frame="measurements", age < 40), frame="measurements", field="age")'
```
``` response
{"results":[{"max":34,"count":1}]}
```
### Storing Row and Column Attributes
#### Introduction

View file

@ -256,7 +256,7 @@ func TestFragment_FieldSum(t *testing.T) {
})
}
// Ensure a fragment can find the max of field values.
// Ensure a fragment can find the min and max of field values.
func TestFragment_FieldMinMax(t *testing.T) {
const bitDepth = 16