diff --git a/docs/data-model.md b/docs/data-model.md index 6a57a96a1..655d1633e 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -112,7 +112,7 @@ SetBit(frame="A", rowID=8, columnID=3, timestamp="2017-05-19T00:00") #### BSI Range-Encoding Bit-Sliced Indexing (BSI) is the storage method Pilosa uses to represent multi-bit integers in a bitmap index. Integers are stored as n-bit, range-encoded -bit-sliced indexes of base-2, along with an additional bitmap indicating "not null". This means that a 16-bit integer will require 17 bitmaps: one for each 0-bit of the 16 bit-slice components (the 1-bit does not need to be stored because with range-encoding the highest bit position is always 1) and one for the non-null bitmap. Pilosa can evaluate `Sum` and `Range` queries on these BSI integers. +bit-sliced indexes of base-2, along with an additional bitmap indicating "not null". This means that a 16-bit integer will require 17 bitmaps: one for each 0-bit of the 16 bit-slice components (the 1-bit does not need to be stored because with range-encoding the highest bit position is always 1) and one for the non-null bitmap. Pilosa can evaluate `Range`, `Min`, `Max`, and `Sum` queries on these BSI integers. Internally Pilosa stores each BSI `field` as a `view` within a `frame`. The 'rowIDs' of the `view` are composed of the base-2 representation of the integer. Pilosa manages the base-2 offset and translation that efficiently packs the integer value within the minimum set of rows. diff --git a/docs/glossary.md b/docs/glossary.md index 612c65dd4..6157eba57 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -14,7 +14,7 @@ nav = [] [Bitmap](../data-model/#overview): The on-disk and in-memory representation of a [row](#row). Implemented with [Roaring](#roaring-bitmap). `Bitmap` is also the basic [PQL](#pql) query for reading a Bitmap. -[BSI](../data-model/#bsi-range-encoding) Bit-sliced indexing is the method Pilosa uses to represent multi-bit integers. Integer values are stored in [fields](#field), and can be used for [Range](#range-bsi) and [Sum](#sum) queries. +[BSI](../data-model/#bsi-range-encoding) Bit-sliced indexing is the method Pilosa uses to represent multi-bit integers. Integer values are stored in [fields](#field), and can be used for [Range](#range-bsi), [Min](#min), [Max](#max), and [Sum](#sum) queries. Cluster: A cluster consists of one or more [nodes](#node) which share a cluster configuration. The cluster also defines how data is [replicated](#replica) throughout and how internode communication is coordinated. Pilosa does not have a leader node, all data is evenly distributed, and any node can respond to queries. @@ -30,8 +30,12 @@ nav = [] [Jump Consistent Hash](https://arxiv.org/pdf/1406.2294v1.pdf): A fast, minimal memory, consistent hash algorithm that evenly distributes the workload even when the number of buckets changes. +[Max](../query-language/#max): A [PQL](#pql) query that returns the maximum integer value stored in [BSI](#bsi) [fields](#field). + MaxSlice: The total number of [slices](#slice) allocated to handle the current set of [columns](#column). This value is important for all [nodes](#node) to efficiently distribute queries. +[Min](../query-language/#min): A [PQL](#pql) query that returns the minimum integer value stored in [BSI](#bsi) [fields](#field). + Node: An individual running instance of Pilosa server which belongs to a [cluster](#cluster). Partition: The [consistent hash](#jump-consistent-hash) maps keys to partitions (or locations on the unit circle), based on a preset maximum number of partitions. Partitions are then evenly mapped to physical [nodes](#node). To add nodes to the [cluster](#cluster), the partitions must be remapped, and data is then associated across the new cluster topology. `DefaultPartitionN` is 256. It can be modified, but only at compile time, and before ingesting any data. diff --git a/docs/query-language.md b/docs/query-language.md index 6b0584320..efb143996 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -523,6 +523,56 @@ Range(frame="stats", commitactivity >< [100, 200]) This is conceptually equivalent to the interval 100 <= commitactivity <= 200, but this chained comparison syntax is not currently supported. `BETWEEN` query syntax is restricted to greater-than-or-equal-to and less-than-or-equal-to, but any valid interval on the integers can be represented this way. +#### Min + +**Spec:** + +``` +Min([BITMAP_CALL], , ) +``` + +**Description:** + +Returns the minimum value of all BSI integer values in the `field` in this `frame`. If the optional `Bitmap` call is supplied, only columns with set bits are considered, otherwise all collumns are considered. + +**Result Type:** object with the min and count of columns containing the min value. + +**Examples:** + +Query the size of all repositories. +``` +Min(frame="stats", field="diskusage") +``` + +Return `{"min":4,"count":2}` + +* Result is the smallest repository in kilobytes, plus the number of repositories of that size. + +#### Max + +**Spec:** + +``` +Max([BITMAP_CALL], , ) +``` + +**Description:** + +Returns the maximum value of all BSI integer values in the `field` in this `frame`. If the optional `Bitmap` call is supplied, only columns with set bits are considered, otherwise all columns are considered. + +**Result Type:** object with the max and count of columns containing the max value. + +**Examples:** + +Query the size of all repositories. +``` +Max(frame="stats", field="diskusage") +``` + +Return `{"max":88,"count":13}` + +* Result is the largest repository in kilobytes, plus the number of repositories of that size. + #### Sum **Spec:** diff --git a/docs/tutorials.md b/docs/tutorials.md index 2401cdfe4..6715e1cd1 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -247,7 +247,7 @@ Check out our [Administration Guide](https://www.pilosa.com/docs/latest/administ #### Introduction -Pilosa can store integer values associated to the columns in an index, and those values are used to support `Range` and `Sum` queries. In this tutorial we will show how to set up integer fields, populate those fields with data, and query the fields. The example index we're going to create will represent fictional patients at a medical facility and various bits of information about those patients. +Pilosa can store integer values associated to the columns in an index, and those values are used to support `Range`, `Min`, `Max`, and `Sum` queries. In this tutorial we will show how to set up integer fields, populate those fields with data, and query the fields. The example index we're going to create will represent fictional patients at a medical facility and various bits of information about those patients. First, create an index called `patients`: ``` request @@ -333,7 +333,7 @@ curl localhost:10101/index/patients/query \ ``` The results you get from the `Sum` query contain the `sum` of all values as well as the `count` of columns with a value. To get the average you can just divide `sum` by `count`. -You can also provide a filter to the `Sum()` function, to find the average age of all patients over 40. +You can also provide a filter to the `Sum()` function to find the average age of all patients over 40. ``` request curl localhost:10101/index/patients/query \ -X POST \ @@ -344,6 +344,48 @@ curl localhost:10101/index/patients/query \ ``` Notice in this case that the count is only `3` because of the `age > 40` filter applied to the query. +To find the minimum age of all patients, run a `Min` query: +``` request +curl localhost:10101/index/patients/query \ + -X POST \ + -d 'Min(frame="measurements", field="age")' +``` +``` response +{"results":[{"min":19,"count":1}]} +``` +The results you get from the `Min` query contain the `min` of all values as well as the `count` of columns with that value. + +You can also provide a filter to the `Min()` function to find the minimum age of all patients over 40. +``` request +curl localhost:10101/index/patients/query \ + -X POST \ + -d 'Min(Range(frame="measurements", age > 40), frame="measurements", field="age")' +``` +``` response +{"results":[{"min":57,"count":1}]} +``` + +To find the maximum age of all patients, run a `Max` query: +``` request +curl localhost:10101/index/patients/query \ + -X POST \ + -d 'Max(frame="measurements", field="age")' +``` +``` response +{"results":[{"max":71,"count":1}]} +``` +The results you get from the `Max` query contain the `max` of all values as well as the `count` of columns with that value. + +You can also provide a filter to the `Max()` function to find the maximum age of all patients under 40. +``` request +curl localhost:10101/index/patients/query \ + -X POST \ + -d 'Max(Range(frame="measurements", age < 40), frame="measurements", field="age")' +``` +``` response +{"results":[{"max":34,"count":1}]} +``` + ### Storing Row and Column Attributes #### Introduction diff --git a/fragment_test.go b/fragment_test.go index 6b49b5d75..d7c97a5cf 100644 --- a/fragment_test.go +++ b/fragment_test.go @@ -256,7 +256,7 @@ func TestFragment_FieldSum(t *testing.T) { }) } -// Ensure a fragment can find the max of field values. +// Ensure a fragment can find the min and max of field values. func TestFragment_FieldMinMax(t *testing.T) { const bitDepth = 16