Merge pull request #861 from raskle/740-bsi-doc-v2

740 bsi doc v2

docs/api-reference.md

@@ -161,6 +161,16 @@ The request payload is in JSON, and may contain the `options` field. The `option
 * `inverseEnabled` (boolean): Enables [the inverted view]({{< ref "data-model.md#inverse" >}}) for this frame if `true`.
 * `cacheType` (string): [ranked]({{< ref "data-model.md#ranked" >}}) or [LRU]({{< ref "data-model.md#lru" >}}) caching on this frame. Default is `lru`.
 * `cacheSize` (int): Number of rows to keep in the cache. Default 50,000.
+* `rangeEnabled` (boolean): Enables range-encoded fields in this frame.
+* `fields` (array): List of range-encoded fields.
+
+Each individual `field` contains the following:
+* `name` (string): Field name.
+* `type` (string): Field type, currently only "int" is supported.
+* `min` (int): Minimum value allowed for this field.
+* `max` (int): Maximum value allowed for this field.
+
+Integer fields are stored as n-bit range-encoded values. Pilosa supports 63-bit, signed integers with values between `min` and `max`.
 
 Request:
 ```
@@ -169,6 +179,12 @@ curl localhost:10101/index/user/frame/language \
      -d '{"options": {"inverseEnabled": true}}'
 ```
 
+```
+curl localhost:10101/index/repository/frame/stats \
+    -X POST \
+    -d '{"rangeEnabled": true, "fields": [{"name": "pullrequests", "type": "int", "min": 0, "max": 1000000}]}'
+```
+
 Response:
 ```
 {}
@@ -222,6 +238,29 @@ Response:
 {}
 ```
 
+### Create Field
+
+`POST /index/<index-name>/frame/<frame-name>/field/<field-name>`
+
+Creates a new field to store integer values in the given frame.
+
+The request payload is JSON, and it must contain the fields `type`, `min`, `max`.
+* `type` (string): Field type, currently only "int" is supported.
+* `min` (int): Minimum value allowed for this field.
+* `max` (int): Maximum value allowed for this field.
+
+Request:
+```
+curl localhost:10101/index/repository/frame/stats/field/pullrequests \
+    -X POST \
+    -d '{"type": "int", "min": 0, "max": 1000000}'
+```
+
+Response:
+```
+{}
+```
+
 ### Create input definition
 
 `POST /index/<index-name>/input-definition/<input-definition-name>`

docs/data-model.md

@@ -101,3 +101,23 @@ SetBit(frame="A", rowID=8, columnID=3, timestamp="2017-05-19T00:00")
 ```
 
 ![time quantum frame diagram](/img/docs/frame-time-quantum.svg)
+
+#### BSI Range-Encoding
+
+Bit-Sliced Indexing (BSI) is the storage method Pilosa uses to represent multi-bit integers in a bitmap index. Integers are stored as n-bit, range-encoded
+bit-sliced indexes of base-2, along with an additional bitmap indicating "not null". This means that a 16-bit integer will require 17 bitmaps: one for each 0-bit of the 16 bit-slice components (the 1-bit does not need to be stored because with range-encoding the highest bit position is always 1) and one for the non-null bitmap. Pilosa can evaluate, aggregate, and range queries on these BSI integers. 
+
+Internally Pilosa stores each BSI `field` as a `view` within a `frame`. The 'rowIDs' of the `view` are composed of the base-2 representation of the integer. Pilosa manages the base-2 offset and translation that efficiently packs the integer value within the minimum set of rows.
+
+For example, the following `SetFieldValue()` queries will result in the data described in the illustration below:
+
+```
+SetFieldValue(col=1, frame="A", "field0"=1)
+SetFieldValue(col=2, frame="A", "field0"=2)
+SetFieldValue(col=3, frame="A", "field0"=3)
+SetFieldValue(col=4, frame="A", "field0"=7)
+SetFieldValue(col=2, frame="A", "field1"=1)
+SetFieldValue(col=3, frame="A", "field1"=6)
+```
+
+![BSI diagram](/img/docs/frame-bsi.svg)

docs/query-language.md

@@ -420,3 +420,83 @@ Range(frame="stargazer", rowID=1, start="2010-01-01T00:00", end="2017-03-02T03:0
 Returns `{{"attrs":{},"bits":[10]}`
 
 * bits are repositories which were starred by user 1 from 2010-01-01 to 2017-03-02
+
+
+#### Range (BSI)
+
+**Spec:**
+
+```
+Range(<frame=STRING>, <FIELD_NAME, COMPARISON_OPERATOR, integer> )
+```
+
+**Description:**
+
+The `Range` query is overloaded to work on `field` values as well as `timestamp` values.
+Returns bits that are true for the comparison operator.
+
+**Result Type:** object with attrs and bits
+
+
+**Examples:**
+
+In our source data, commitactivity was counted over the last year.
+The following Range query returns all repositories having more than 100 commits.
+
+```
+Range(frame="stats", commitactivity > 100)
+```
+
+Returns `{{"attrs":{},"bits":[10]}`
+
+* bits are repositories which had at least 100 commits in the last year.
+
+
+#### Sum
+
+**Spec:**
+
+```
+Sum(<frame=STRING>, <field=STRING>, [BITMAP_CALL])
+```
+
+**Description:**
+
+Returns the count and computed sum of all bitmap encoded integer values across the `field` in this `frame`. The optional Bitmap call filters the bits used in this computation.
+
+**Result Type:** object with the computed sum and count of the bitmap field.
+
+**Examples:**
+
+Query the size of all repositories.
+```
+Sum(frame="stats", field="diskusage")
+```
+
+Return `{"sum":10,"count":3}`
+
+* Result is the size of all repositories in kilobytes, plus the number of repositories.
+
+
+#### SetFieldValue
+
+**Spec:**
+
+```
+SetFieldValue(<COL_LABEL=UINT>, <frame=STRING>, <FIELD_NAME=INT>)
+```
+
+**Description:**
+
+`SetFieldValue` assigns an integer value with the specified field name to the `columnID` in the given `frame`.
+
+**Result Type:** null
+
+SetFieldValue returns `null` upon success.
+
+**Examples:**
+
+Set the number of pull requests of repository 10.
+```
+SetFieldValue(col=10, frame="stats", "pullrequests"=2)
+```