From 35d6e945a8f25946d224b125ded452fb45c19169 Mon Sep 17 00:00:00 2001 From: Michael Baird Date: Wed, 27 Sep 2017 16:51:16 -0500 Subject: [PATCH 1/4] Create BSI Frame and Field documentation --- docs/api-reference.md | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/docs/api-reference.md b/docs/api-reference.md index bf599fafe..115cf3383 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -162,6 +162,16 @@ The request payload is in JSON, and may contain the `options` field. The `option * `inverseEnabled` (boolean): Enables [the inverted view]({{< ref "data-model.md#inverse" >}}) for this frame if `true`. * `cacheType` (string): [ranked]({{< ref "data-model.md#ranked" >}}) or [LRU]({{< ref "data-model.md#lru" >}}) caching on this frame. Default is `lru`. * `cacheSize` (int): Number of rows to keep in the cache. Default 50,000. +* `rangeEnabled` (boolean): Enables range-encoded fields in this frame. +* `fields` (array): List of range-encoded fields. + +Each individual `field` contains the following: +* `name` (string): Field name. +* `type` (string): Field type, currently only "int" is supported. +* `min` (int): Minimum of the value range stored in this field. +* `max` (int): Maximum of the value range stored in this field. + +Integer fields are stored as n-bit range-encoded values. Pilosa can use up to a 64-bit integer with one bit reserved for the non-null bitmap leaving up to a 63-bit signed integer represented between `min` and `max`. Request: ``` @@ -170,6 +180,12 @@ curl localhost:10101/index/user/frame/language \ -d '{"options": {"rowLabel": "language_id"}}' ``` +``` +curl localhost:10101/index/repository/frame/stats \ + -X POST \ + -d '{"rangeEnabled": true, "fields": [{"name": "pullrequests", "type": "int", "min": 0, "max": 1000000}]}' +``` + Response: ``` {} @@ -223,6 +239,29 @@ Response: {} ``` +### Create Field + +`POST /index//frame//field/` + +Creates a new field to store integer values in the given frame. + +The request payload is JSON, and it must contain the fields `type`, `min`, `max`. +* `type` (string): Field type, currently only "int" is supported. +* `min` (int): Minimum of the value range stored in this field. +* `max` (int): Maximum of the value range stored in this field. + +Request: +``` +curl localhost:10101/index/repository/frame/stats/field/pullrequests \ + -X POST \ + -d '{"type": "int", "min": 0, "max": 1000000}' +``` + +Response: +``` +{} +``` + ### Create input definition `POST /index//input-definition/` From d2c052d4a94337acc072a318a83efc9bdc196a0f Mon Sep 17 00:00:00 2001 From: Michael Baird Date: Wed, 27 Sep 2017 16:53:02 -0500 Subject: [PATCH 2/4] BSI data model --- docs/data-model.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/docs/data-model.md b/docs/data-model.md index 2e7ac03e8..6e54c4e0c 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -101,3 +101,23 @@ SetBit(frame="A", rowID=8, columnID=3, timestamp="2017-05-19T00:00") ``` ![time quantum frame diagram](/img/docs/frame-time-quantum.svg) + +#### BSI Range-Encoding + +Bit-Sliced Indexing (BSI) is the storage method Pilosa uses to represent multi-bit integers in a bitmap index. Integers are stored as n-bit, range-encoded +bit-sliced indexes of base-2, along with an additional bitmap indicating "not null". This means that a 16-bit integer will require 17 bitmaps: one for each 0-bit of the 16 bit-slice components (the 1-bit does not need to be stored because with range-encoding the highest bit position is always 1) and one for the non-null bitmap. Pilosa can evaluate, aggregate, and range queries on these BSI integers. + +Internally Pilosa stores each BSI `field` as a `view` within a `frame`. The 'rowIDs' of the `view` are composed of the base-2 representation of the integer. Pilosa manages the base-2 offset and translation that efficiently packs the integer value within the minimum set of rows. + +For example, the following `SetFieldValue()` queries will result in the data described in the illustration below: + +``` +SetFieldValue(col=1, frame="A", "field0"=1) +SetFieldValue(col=2, frame="A", "field0"=2) +SetFieldValue(col=3, frame="A", "field0"=3) +SetFieldValue(col=4, frame="A", "field0"=7) +SetFieldValue(col=2, frame="A", "field1"=1) +SetFieldValue(col=3, frame="A", "field1"=6) +``` + +![BSI diagram](/img/docs/frame-bsi.svg) From a1d81fbf863ad8652a90ae7b6c4bd6eb7c16f673 Mon Sep 17 00:00:00 2001 From: Michael Baird Date: Wed, 27 Sep 2017 17:11:54 -0500 Subject: [PATCH 3/4] BSI queries --- docs/query-language.md | 80 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/docs/query-language.md b/docs/query-language.md index 7c54c3ba8..178eff562 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -420,3 +420,83 @@ Range(frame="stargazer", stargazer_id=1, start="2010-01-01T00:00", end="2017-03- Returns `{{"attrs":{},"bits":[10]}` * bits are repositories which were starred by user 1 from 2010-01-01 to 2017-03-02 + + +#### Range (BSI) + +**Spec:** + +``` +Range(, ) +``` + +**Description:** + +The to `Range` query is overloaded to also work on `field` values. +Returns bits that are true for the comparison operator + +**Result Type:** object with attrs and bits + + +**Examples:** + +In our source data commitactivity was counted over the last year. +This Range query finds all repositories that had that many commits. + +``` +Range(frame="stats", commitactivity > 100) +``` + +Returns `{{"attrs":{},"bits":[10]}` + +* bits are repositories which had at least 100 commits in the last year. + + +#### Sum + +**Spec:** + +``` +Sum(, ) +``` + +**Description:** + +Returns the computed sum of all `field` values across the bits in a `frame` plus the count of bitmaps with a field value. + +**Result Type:** object with sum and count. + +**Examples:** + +Query the size of all repositories. +``` +Sum(frame="stats", field="diskusage") +``` + +Return `{"sum":10,"count":3}` + +* Result is the size of all repositories in kilobytes, plus the number of repositories. + + +#### SetFieldValue + +**Spec:** + +``` +SetFieldValue(, , ) +``` + +**Description:** + +`SetFieldValue` assigns an integer value with the specified field name to the `columnID` in the given `frame`. The `field` values is encoded across the bits in a `frame`. + +**Result Type:** null + +SetFieldValue queries always return `null` upon success. + +**Examples:** + +Set the number of pull requests of repository 10. +``` +SetFieldValue(col=10, frame="stats", "pullrequests"=2) +``` \ No newline at end of file From 87f8d7be5f0af6b34c5ff98548b9d3ba9e192e79 Mon Sep 17 00:00:00 2001 From: Michael Baird Date: Mon, 2 Oct 2017 16:29:46 -0500 Subject: [PATCH 4/4] docs cleanup --- docs/api-reference.md | 10 +++++----- docs/query-language.md | 18 +++++++++--------- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/api-reference.md b/docs/api-reference.md index 115cf3383..eef3c4cbc 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -168,10 +168,10 @@ The request payload is in JSON, and may contain the `options` field. The `option Each individual `field` contains the following: * `name` (string): Field name. * `type` (string): Field type, currently only "int" is supported. -* `min` (int): Minimum of the value range stored in this field. -* `max` (int): Maximum of the value range stored in this field. +* `min` (int): Minimum value allowed for this field. +* `max` (int): Maximum value allowed for this field. -Integer fields are stored as n-bit range-encoded values. Pilosa can use up to a 64-bit integer with one bit reserved for the non-null bitmap leaving up to a 63-bit signed integer represented between `min` and `max`. +Integer fields are stored as n-bit range-encoded values. Pilosa supports 63-bit, signed integers with values between `min` and `max`. Request: ``` @@ -247,8 +247,8 @@ Creates a new field to store integer values in the given frame. The request payload is JSON, and it must contain the fields `type`, `min`, `max`. * `type` (string): Field type, currently only "int" is supported. -* `min` (int): Minimum of the value range stored in this field. -* `max` (int): Maximum of the value range stored in this field. +* `min` (int): Minimum value allowed for this field. +* `max` (int): Maximum value allowed for this field. Request: ``` diff --git a/docs/query-language.md b/docs/query-language.md index 178eff562..874a1e380 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -432,16 +432,16 @@ Range(, ) **Description:** -The to `Range` query is overloaded to also work on `field` values. -Returns bits that are true for the comparison operator +The `Range` query is overloaded to work on `field` values as well as `timestamp` values. +Returns bits that are true for the comparison operator. **Result Type:** object with attrs and bits **Examples:** -In our source data commitactivity was counted over the last year. -This Range query finds all repositories that had that many commits. +In our source data, commitactivity was counted over the last year. +The following Range query returns all repositories having more than 100 commits. ``` Range(frame="stats", commitactivity > 100) @@ -457,14 +457,14 @@ Returns `{{"attrs":{},"bits":[10]}` **Spec:** ``` -Sum(, ) +Sum(, , [BITMAP_CALL]) ``` **Description:** -Returns the computed sum of all `field` values across the bits in a `frame` plus the count of bitmaps with a field value. +Returns the count and computed sum of all bitmap encoded integer values across the `field` in this `frame`. The optional Bitmap call filters the bits used in this computation. -**Result Type:** object with sum and count. +**Result Type:** object with the computed sum and count of the bitmap field. **Examples:** @@ -488,11 +488,11 @@ SetFieldValue(, , ) **Description:** -`SetFieldValue` assigns an integer value with the specified field name to the `columnID` in the given `frame`. The `field` values is encoded across the bits in a `frame`. +`SetFieldValue` assigns an integer value with the specified field name to the `columnID` in the given `frame`. **Result Type:** null -SetFieldValue queries always return `null` upon success. +SetFieldValue returns `null` upon success. **Examples:**