From e4a7e2edbd1e22b050f8a584e8936b8044cbfb93 Mon Sep 17 00:00:00 2001 From: Travis Turner Date: Mon, 23 Apr 2018 16:47:22 -0500 Subject: [PATCH 1/2] remove references to row and column labels from the docs --- docs/api-reference.md | 4 +-- docs/data-model.md | 18 ++++++------- docs/pdk.md | 2 +- docs/query-language.md | 58 +++++++++++++++++++++--------------------- 4 files changed, 41 insertions(+), 41 deletions(-) diff --git a/docs/api-reference.md b/docs/api-reference.md index a2697f311..9fc232535 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -68,7 +68,7 @@ Sends a query to the Pilosa server with the given index. The request body is UTF ``` request curl localhost:10101/index/user/query \ -X POST \ - -d 'Bitmap(frame="language", rowID=5)' + -d 'Bitmap(frame="language", row=5)' ``` ``` response {"results":[{"attrs":{},"bits":[100]}]} @@ -83,7 +83,7 @@ The query is executed for all [slices](../data-model/#slice) by default. To use ``` request curl "localhost:10101/index/user/query?columnAttrs=true&slices=0,1" \ -X POST \ - -d 'Bitmap(frame="language", rowID=5)' + -d 'Bitmap(frame="language", row=5)' ``` ``` response { diff --git a/docs/data-model.md b/docs/data-model.md index dc8646a85..693b4c97d 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -88,8 +88,8 @@ The standard View contains the same Row/Column format as the input data. If a Frame has a time quantum, then Views are generated for each of the defined time segments. For example, for a frame with a time quantum of `YMD`, the following `SetBit()` queries will result in the data described in the illustration below: ``` -SetBit(frame="A", rowID=8, columnID=3, timestamp="2017-05-18T00:00") -SetBit(frame="A", rowID=8, columnID=3, timestamp="2017-05-19T00:00") +SetBit(frame="A", row=8, col=3, timestamp="2017-05-18T00:00") +SetBit(frame="A", row=8, col=3, timestamp="2017-05-19T00:00") ``` ![time quantum frame diagram](/img/docs/frame-time-quantum.svg) @@ -100,17 +100,17 @@ SetBit(frame="A", rowID=8, columnID=3, timestamp="2017-05-19T00:00") Bit-Sliced Indexing (BSI) is the storage method Pilosa uses to represent multi-bit integers in a bitmap index. Integers are stored as n-bit, range-encoded bit-sliced indexes of base-2, along with an additional bitmap indicating "not null". This means that a 16-bit integer will require 17 bitmaps: one for each 0-bit of the 16 bit-slice components (the 1-bit does not need to be stored because with range-encoding the highest bit position is always 1) and one for the non-null bitmap. Pilosa can evaluate `Sum` and `Range` queries on these BSI integers. -Internally Pilosa stores each BSI `field` as a `view` within a `frame`. The 'rowIDs' of the `view` are composed of the base-2 representation of the integer. Pilosa manages the base-2 offset and translation that efficiently packs the integer value within the minimum set of rows. +Internally Pilosa stores each BSI `field` as a `view` within a `frame`. The 'rows' of the `view` are composed of the base-2 representation of the integer. Pilosa manages the base-2 offset and translation that efficiently packs the integer value within the minimum set of rows. For example, the following `SetFieldValue()` queries will result in the data described in the illustration below: ``` -SetFieldValue(columnID=1, frame="A", field0=1) -SetFieldValue(columnID=2, frame="A", field0=2) -SetFieldValue(columnID=3, frame="A", field0=3) -SetFieldValue(columnID=4, frame="A", field0=7) -SetFieldValue(columnID=2, frame="A", field1=1) -SetFieldValue(columnID=3, frame="A", field1=6) +SetFieldValue(col=1, frame="A", field0=1) +SetFieldValue(col=2, frame="A", field0=2) +SetFieldValue(col=3, frame="A", field0=3) +SetFieldValue(col=4, frame="A", field0=7) +SetFieldValue(col=2, frame="A", field1=1) +SetFieldValue(col=3, frame="A", field1=6) ``` ![BSI frame diagram](/img/docs/frame-bsi.svg) diff --git a/docs/pdk.md b/docs/pdk.md index 8c02dc279..9be1d70be 100644 --- a/docs/pdk.md +++ b/docs/pdk.md @@ -34,7 +34,7 @@ With this definition available, the PDK tool can run the import, which consists - for each CSV record: - generate a columnID - apply all ParserMappers, generating a list of (frame, ID) pairs - - set the appropriate bit. schematically: SetBit(id=rowID, frame=frame, profileID=columnID) + - set the appropriate bit. schematically: SetBit(row=rowID, frame=frame, col=columnID) The process is summarized in this flowchart: diff --git a/docs/query-language.md b/docs/query-language.md index f26cf021d..83a10ab36 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -25,17 +25,17 @@ There will be one item in the `results` array for each PQL query in the request. * Angle Brackets `<>` denote required arguments * Square Brackets `[]` denote optional arguments -* UPPER_CASE denotes a descriptor that will need to be filled in with a concrete value (e.g. `ROW_LABEL`, `STRING`) +* UPPER_CASE denotes a descriptor that will need to be filled in with a concrete value (e.g. `ATTR_NAME`, `STRING`) ##### Examples Before running any of the example queries below, follow the instructions in the [Getting Started](../getting-started/) section to set up an index, frames, and populate them with some data. -The examples just show the PQL quer(ies) needed - to run the query `SetBit(frame="stargazer", columnID=10, rowID=1)` against a server using curl, you would: +The examples just show the PQL quer(ies) needed - to run the query `SetBit(frame="stargazer", col=10, row=1)` against a server using curl, you would: ``` request curl localhost:10101/index/repository/query \ -X POST \ - -d 'SetBit(frame="stargazer", columnID=10, rowID=1)' + -d 'SetBit(frame="stargazer", col=10, row=1)' ``` ``` response {"results":[true]} @@ -58,7 +58,7 @@ curl localhost:10101/index/repository/query \ **Spec:** ``` -SetBit(, , , +SetBit(, , , [timestamp=TIMESTAMP]) ``` @@ -76,26 +76,26 @@ A return value of `false` indicates that the bit was already set to 1 and nothin **Examples:** ``` -SetBit(frame="stargazer", columnID=10, rowID=1) +SetBit(frame="stargazer", col=10, row=1) ``` This query illustrates setting a bit in the stargazer frame. User with id=1 has starred repository with id=10. SetBit also supports providing a timestamp. To write the date that a user starred a repository. ``` -SetBit(frame="stargazer", columnID=10, rowID=1, timestamp="2016-01-01T00:00") +SetBit(frame="stargazer", col=10, row=1, timestamp="2016-01-01T00:00") ``` Setting multiple bits in a single request: ``` -SetBit(frame="stargazer", columnID=10, rowID=1) SetBit(frame="stargazer", columnID=10, rowID=2) SetBit(frame="stargazer", columnID=20, rowID=1) SetBit(frame="stargazer", columnID=30, rowID=2) +SetBit(frame="stargazer", col=10, row=1) SetBit(frame="stargazer", col=10, row=2) SetBit(frame="stargazer", col=20, row=1) SetBit(frame="stargazer", col=30, row=2) ``` #### SetRowAttrs **Spec:** ``` -SetRowAttrs(, , +SetRowAttrs(, , , [ATTR_NAME=ATTR_VALUE ...]) ``` @@ -111,13 +111,13 @@ SetRowAttrs queries always return `null` upon success. **Examples:** ``` -SetRowAttrs(frame="stargazer", rowID=10, username="mrpi", active=true) +SetRowAttrs(frame="stargazer", row=10, username="mrpi", active=true) ``` Set username value and active status for user 10. These are arbitrary key/value pairs which have no meaning to Pilosa. You can see the attributes you've set on a row with a [Bitmap](../query-language/#bitmap) query like so `Bitmap(frame="stargazer", stargazer_id=10)`. ``` -SetRowAttrs(frame="stargazer", rowID=10, username=null) +SetRowAttrs(frame="stargazer", row=10, username=null) ``` Delete username value for user 10. @@ -127,7 +127,7 @@ Delete username value for user 10. **Spec:** ``` -SetColumnAttrs(, , +SetColumnAttrs(, , , [ATTR_NAME=ATTR_VALUE ...]) ``` @@ -143,13 +143,13 @@ SetColumnAttrs queries always return `null` upon success. Setting a value of `nu **Examples:** ``` -SetColumnAttrs(columnID=10, stars=123, url="http://projects.pilosa.com/10", active=true) +SetColumnAttrs(col=10, stars=123, url="http://projects.pilosa.com/10", active=true) ``` -Set url value and active status for project 10. These are arbitrary key/value pairs which have no meaning to Pilosa. You can see the attributes you've set on a column with a [Bitmap](../query-language/#bitmap) query like so `Bitmap(frame="stargazer", columnID=10)`. +Set url value and active status for project 10. These are arbitrary key/value pairs which have no meaning to Pilosa. You can see the attributes you've set on a column with a [Bitmap](../query-language/#bitmap) query like so `Bitmap(frame="stargazer", col=10)`. ``` -SetColumnAttrs(columnID=10, url=null) +SetColumnAttrs(col=10, url=null) ``` Delete url value for repo 10. @@ -160,7 +160,7 @@ Delete url value for repo 10. **Spec:** ``` -SetBit(, , , +SetBit(, , , [timestamp=TIMESTAMP]) ``` @@ -177,7 +177,7 @@ A return value of `false` indicates that the bit was already set to 0 and nothin **Examples:** ``` -ClearBit(frame="stargazer", columnID=10, rowID=1) +ClearBit(frame="stargazer", col=10, row=1) ``` Remove relationship between the stargazer in row 1 and the repository in column 10 from the stargazer frame. @@ -188,12 +188,12 @@ Remove relationship between the stargazer in row 1 and the repository in column **Spec:** ``` -SetFieldValue(, , ) +SetFieldValue(, , ) ``` **Description:** -`SetFieldValue` assigns an integer value with the specified field name to the `columnID` in the given `frame`. +`SetFieldValue` assigns an integer value with the specified field name to the `col` in the given `frame`. **Result Type:** null @@ -203,7 +203,7 @@ SetFieldValue returns `null` upon success. Set the number of pull requests of repository 10. ``` -SetFieldValue(columnID=10, frame="stats", pullrequests=2) +SetFieldValue(col=10, frame="stats", pullrequests=2) ``` @@ -214,7 +214,7 @@ SetFieldValue(columnID=10, frame="stats", pullrequests=2) **Spec:** ``` -Bitmap(, ( | =UINT)) +Bitmap(, ( | =UINT)) ``` **Description:** @@ -229,7 +229,7 @@ e.g. `{"attrs":{"username":"mrpi","active":true},"bits":[10, 20]}` Query all repositories that user 1 has starred. ``` -Bitmap(frame="stargazer", rowID=1) +Bitmap(frame="stargazer", row=1) ``` Returns `{"attrs":{"username":"mrpi","active":true},"bits":[10, 20]}` @@ -286,7 +286,7 @@ attrs will always be empty Query repositories which have been starred by two users. ``` -Intersect(Bitmap(frame="stargazer", rowID=1), Bitmap(frame="stargazer", rowID=2)) +Intersect(Bitmap(frame="stargazer", row=1), Bitmap(frame="stargazer", row=2)) ``` Returns `{"attrs":{},"bits":[10]}`. @@ -313,7 +313,7 @@ attrs will always be empty Query repositories which have been starred by one user and not another. ``` -Difference(Bitmap(frame="stargazer", rowID=1), Bitmap( frame="stargazer", rowID=2)) +Difference(Bitmap(frame="stargazer", row=1), Bitmap( frame="stargazer", row=2)) ``` Return `{"results":[{"attrs":{},"bits":[20]}]}` @@ -321,7 +321,7 @@ Return `{"results":[{"attrs":{},"bits":[20]}]}` * bits are repositories that were starred by user 1 BUT NOT user 2 ``` -Difference(Bitmap(frame="stargazer", rowID=2), Bitmap( frame="stargazer", rowID=1)) +Difference(Bitmap(frame="stargazer", row=2), Bitmap( frame="stargazer", row=1)) ``` Return `{"attrs":{},"bits":[30]}` @@ -349,7 +349,7 @@ attrs will always be empty Query repositories which have been starred by two users. ``` -Xor(Bitmap(frame="stargazer", rowID=1), Bitmap(frame="stargazer", rowID=2)) +Xor(Bitmap(frame="stargazer", row=1), Bitmap(frame="stargazer", row=2)) ``` Returns `{"attrs":{},"bits":[30]}`. @@ -373,7 +373,7 @@ Returns the number of set bits in the `BITMAP_CALL` passed in. Query the number of repositories to which a user has contributed. ``` -Count(Bitmap(frame="stargazer", rowID=1)) +Count(Bitmap(frame="stargazer", row=1)) ``` Return `2` @@ -427,7 +427,7 @@ Returns `[{"key": 1, "count": 2}, {"key": 2, "count": 2}]` * Results are the top two users sorted by number of repositories they've starred in descending order. ``` -TopN(Bitmap(frame="language", rowID=1), frame="stargazer", n=2) +TopN(Bitmap(frame="language", row=1), frame="stargazer", n=2) ``` Returns `[{"key": 1, "count": 2}, {"key": 2, "count": 1}]` @@ -439,7 +439,7 @@ Returns `[{"key": 1, "count": 2}, {"key": 2, "count": 1}]` **Spec:** ``` -Range(, , +Range(, , , ) ``` @@ -455,7 +455,7 @@ between the given `start` and `end` timestamps. When you set timestamp using SetBit, you will able to query all repositories that a user has starred within a date range. ``` -Range(frame="stargazer", rowID=1, start="2010-01-01T00:00", end="2017-03-02T03:00") +Range(frame="stargazer", row=1, start="2010-01-01T00:00", end="2017-03-02T03:00") ``` Returns `{{"attrs":{},"bits":[10]}` From 7ecd2ec79468ecd89ce5f76aa3378494c888dce0 Mon Sep 17 00:00:00 2001 From: Travis Turner Date: Mon, 23 Apr 2018 17:10:48 -0500 Subject: [PATCH 2/2] adjust references to RowID and ColumnID in the docs --- docs/administration.md | 6 +++--- docs/data-model.md | 2 +- docs/glossary.md | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/administration.md b/docs/administration.md index c9d932df0..190b5ca1f 100644 --- a/docs/administration.md +++ b/docs/administration.md @@ -48,9 +48,9 @@ On Mac OS X, `ulimit` does not behave predictably. [This blog post](https://blog #### Importing -The import API expects a csv of RowID,ColumnID's. +The import API expects a csv of rowID,columnID's. -When importing large datasets remember it is much faster to pre sort the data by RowID and then by ColumnID in ascending order. You can use the `--sort` flag to do that. Also, avoid querying Pilosa until the import is complete, otherwise you will experience inconsistent results. +When importing large datasets remember it is much faster to pre sort the data by row ID and then by column ID in ascending order. You can use the `--sort` flag to do that. Also, avoid querying Pilosa until the import is complete, otherwise you will experience inconsistent results. ``` pilosa import --sort -i project -f stargazer project-stargazer.csv @@ -70,7 +70,7 @@ pilosa import -i project -f stargazer --field star_count project-stargazer-count #### Exporting -Exporting Data to csv can be performed on a live instance of Pilosa. You need to specify the Index, Frame, and View(default is standard). The API also expects the slice number, but the `pilosa export` sub command will export all slices within a Frame. The data will be in csv format RowID,ColumnID and sorted by column ID. +Exporting Data to csv can be performed on a live instance of Pilosa. You need to specify the Index, Frame, and View(default is standard). The API also expects the slice number, but the `pilosa export` sub command will export all slices within a Frame. The data will be in csv format rowID,columnID and sorted by columnID. ``` curl "http://localhost:10101/export?index=repository&frame=stargazer&slice=0&view=standard" \ --header "Accept: text/csv" diff --git a/docs/data-model.md b/docs/data-model.md index 693b4c97d..98750eda8 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -100,7 +100,7 @@ SetBit(frame="A", row=8, col=3, timestamp="2017-05-19T00:00") Bit-Sliced Indexing (BSI) is the storage method Pilosa uses to represent multi-bit integers in a bitmap index. Integers are stored as n-bit, range-encoded bit-sliced indexes of base-2, along with an additional bitmap indicating "not null". This means that a 16-bit integer will require 17 bitmaps: one for each 0-bit of the 16 bit-slice components (the 1-bit does not need to be stored because with range-encoding the highest bit position is always 1) and one for the non-null bitmap. Pilosa can evaluate `Sum` and `Range` queries on these BSI integers. -Internally Pilosa stores each BSI `field` as a `view` within a `frame`. The 'rows' of the `view` are composed of the base-2 representation of the integer. Pilosa manages the base-2 offset and translation that efficiently packs the integer value within the minimum set of rows. +Internally Pilosa stores each BSI `field` as a `view` within a `frame`. The rows of the `view` are composed of the base-2 representation of the integer. Pilosa manages the base-2 offset and translation that efficiently packs the integer value within the minimum set of rows. For example, the following `SetFieldValue()` queries will result in the data described in the illustration below: diff --git a/docs/glossary.md b/docs/glossary.md index d222cb511..bd9293c3d 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -24,7 +24,7 @@ nav = [] Fragment: A Fragment is the intersection of a [frame](#frame) and a [slice](#slice) in an [index](#index). -[Frame](../data-model/#frame): Frames are used to group [rows](#row) into different categories. `RowID`s are namespaced by frame such that the same `RowID` in a different frame refers to a different row. For [ranked](#topn) frames, rows are kept in sorted order within the frame. +[Frame](../data-model/#frame): Frames are used to group [rows](#row) into different categories. Row IDs are namespaced by frame such that the same row ID in a different frame refers to a different row. For [ranked](#topn) frames, rows are kept in sorted order within the frame. [Index](../data-model/#index): An Index is a top level container in Pilosa, analogous to a database in an RDBMS. Queries cannot operate across multiple indexes. @@ -62,6 +62,6 @@ nav = [] [TOML](https://github.com/toml-lang/toml): the language used for Pilosa's [configuration file](../configuration/). -[TopN](../query-language/#topn): A [PQL](#pql) query that returns a list of `RowID`s, sorted by the count of [bits](#bit) set in the [row](#row), within a specified [frame](#frame). +[TopN](../query-language/#topn): A [PQL](#pql) query that returns a list of row IDs, sorted by the count of [bits](#bit) set in the [row](#row), within a specified [frame](#frame). [View](../data-model/#view): Views separate the different data layouts within a [Frame](#frame). The primary view is standard, which represents the typical [row](#row)/[column](#column) data. Time based frame views are automatically generated for each [time quantum](#time-quantum). Views are internally managed by Pilosa, and never exposed directly via the API. This simplifies the functional interface by separating it from the physical data representation.