diff --git a/docs/api-reference.md b/docs/api-reference.md index c11f2609f..df1bfd82a 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -161,6 +161,16 @@ The request payload is in JSON, and may contain the `options` field. The `option * `inverseEnabled` (boolean): Enables [the inverted view]({{< ref "data-model.md#inverse" >}}) for this frame if `true`. * `cacheType` (string): [ranked]({{< ref "data-model.md#ranked" >}}) or [LRU]({{< ref "data-model.md#lru" >}}) caching on this frame. Default is `lru`. * `cacheSize` (int): Number of rows to keep in the cache. Default 50,000. +* `rangeEnabled` (boolean): Enables range-encoded fields in this frame. +* `fields` (array): List of range-encoded fields. + +Each individual `field` contains the following: +* `name` (string): Field name. +* `type` (string): Field type, currently only "int" is supported. +* `min` (int): Minimum value allowed for this field. +* `max` (int): Maximum value allowed for this field. + +Integer fields are stored as n-bit range-encoded values. Pilosa supports 63-bit, signed integers with values between `min` and `max`. Request: ``` @@ -169,6 +179,12 @@ curl localhost:10101/index/user/frame/language \ -d '{"options": {"inverseEnabled": true}}' ``` +``` +curl localhost:10101/index/repository/frame/stats \ + -X POST \ + -d '{"rangeEnabled": true, "fields": [{"name": "pullrequests", "type": "int", "min": 0, "max": 1000000}]}' +``` + Response: ``` {} @@ -222,6 +238,29 @@ Response: {} ``` +### Create Field + +`POST /index//frame//field/` + +Creates a new field to store integer values in the given frame. + +The request payload is JSON, and it must contain the fields `type`, `min`, `max`. +* `type` (string): Field type, currently only "int" is supported. +* `min` (int): Minimum value allowed for this field. +* `max` (int): Maximum value allowed for this field. + +Request: +``` +curl localhost:10101/index/repository/frame/stats/field/pullrequests \ + -X POST \ + -d '{"type": "int", "min": 0, "max": 1000000}' +``` + +Response: +``` +{} +``` + ### Create input definition `POST /index//input-definition/` diff --git a/docs/data-model.md b/docs/data-model.md index 2e7ac03e8..6e54c4e0c 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -101,3 +101,23 @@ SetBit(frame="A", rowID=8, columnID=3, timestamp="2017-05-19T00:00") ``` ![time quantum frame diagram](/img/docs/frame-time-quantum.svg) + +#### BSI Range-Encoding + +Bit-Sliced Indexing (BSI) is the storage method Pilosa uses to represent multi-bit integers in a bitmap index. Integers are stored as n-bit, range-encoded +bit-sliced indexes of base-2, along with an additional bitmap indicating "not null". This means that a 16-bit integer will require 17 bitmaps: one for each 0-bit of the 16 bit-slice components (the 1-bit does not need to be stored because with range-encoding the highest bit position is always 1) and one for the non-null bitmap. Pilosa can evaluate, aggregate, and range queries on these BSI integers. + +Internally Pilosa stores each BSI `field` as a `view` within a `frame`. The 'rowIDs' of the `view` are composed of the base-2 representation of the integer. Pilosa manages the base-2 offset and translation that efficiently packs the integer value within the minimum set of rows. + +For example, the following `SetFieldValue()` queries will result in the data described in the illustration below: + +``` +SetFieldValue(col=1, frame="A", "field0"=1) +SetFieldValue(col=2, frame="A", "field0"=2) +SetFieldValue(col=3, frame="A", "field0"=3) +SetFieldValue(col=4, frame="A", "field0"=7) +SetFieldValue(col=2, frame="A", "field1"=1) +SetFieldValue(col=3, frame="A", "field1"=6) +``` + +![BSI diagram](/img/docs/frame-bsi.svg) diff --git a/docs/query-language.md b/docs/query-language.md index f866a1e27..9eb62215c 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -420,3 +420,83 @@ Range(frame="stargazer", rowID=1, start="2010-01-01T00:00", end="2017-03-02T03:0 Returns `{{"attrs":{},"bits":[10]}` * bits are repositories which were starred by user 1 from 2010-01-01 to 2017-03-02 + + +#### Range (BSI) + +**Spec:** + +``` +Range(, ) +``` + +**Description:** + +The `Range` query is overloaded to work on `field` values as well as `timestamp` values. +Returns bits that are true for the comparison operator. + +**Result Type:** object with attrs and bits + + +**Examples:** + +In our source data, commitactivity was counted over the last year. +The following Range query returns all repositories having more than 100 commits. + +``` +Range(frame="stats", commitactivity > 100) +``` + +Returns `{{"attrs":{},"bits":[10]}` + +* bits are repositories which had at least 100 commits in the last year. + + +#### Sum + +**Spec:** + +``` +Sum(, , [BITMAP_CALL]) +``` + +**Description:** + +Returns the count and computed sum of all bitmap encoded integer values across the `field` in this `frame`. The optional Bitmap call filters the bits used in this computation. + +**Result Type:** object with the computed sum and count of the bitmap field. + +**Examples:** + +Query the size of all repositories. +``` +Sum(frame="stats", field="diskusage") +``` + +Return `{"sum":10,"count":3}` + +* Result is the size of all repositories in kilobytes, plus the number of repositories. + + +#### SetFieldValue + +**Spec:** + +``` +SetFieldValue(, , ) +``` + +**Description:** + +`SetFieldValue` assigns an integer value with the specified field name to the `columnID` in the given `frame`. + +**Result Type:** null + +SetFieldValue returns `null` upon success. + +**Examples:** + +Set the number of pull requests of repository 10. +``` +SetFieldValue(col=10, frame="stats", "pullrequests"=2) +``` \ No newline at end of file