From 24d9fa0086c017d5ae2616ca3e42458136d2fde2 Mon Sep 17 00:00:00 2001 From: Yuce Tekol Date: Mon, 25 Sep 2017 17:22:21 +0300 Subject: [PATCH 1/3] deprecate labels --- docs/query-language.md | 40 ++++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/docs/query-language.md b/docs/query-language.md index 7c54c3ba8..f866a1e27 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -21,7 +21,7 @@ This section will provide a detailed reference and examples for the Pilosa Query There will be one item in the `results` array for each PQL query in the request. The type of each item in the array will depend on the type of query - each query in the reference below lists it's result type. -Row and Column labels are set and frame and index creation time respectively. When the specification of a query says *row_label* or *col_label*, one should use the labels that were set while creating the index and frame. The default row label is `id`, and the default column label is `columnID`. +The default row label is `rowID`, and the default column label is `columnID`. Changing these defaults is deprecated and this feature will be removed in a future release. #### Conventions @@ -33,18 +33,18 @@ Row and Column labels are set and frame and index creation time respectively. Wh Before running any of the example queries below, follow the instructions in the [Getting Started](../getting-started) section to set up an index, frames, and populate them with some data. -The examples just show the PQL quer(ies) needed - to run the query `SetBit(frame="stargazer", repo_id=10, stargazer_id=1)` against a server using curl, you would: +The examples just show the PQL quer(ies) needed - to run the query `SetBit(frame="stargazer", columnID=10, rowID=1)` against a server using curl, you would: ``` curl localhost:10101/index/repository/query \ -X POST \ - -d 'SetBit(frame="stargazer", repo_id=10, stargazer_id=1)' + -d 'SetBit(frame="stargazer", columnID=10, rowID=1)' ``` #### Arguments and Types * `frame` The frame specifies on which Pilosa [frame]({{< ref "glossary.md#frame" >}}) the query will operate. Valid frame names are lower case strings; they start with an alphanumeric character, and contain only alphanumeric characters and `_-`. They must be 64 characters or less in length. -* `ROW_LABEL` Pilosa allows users to set different row labels for each frame at frame creation time. The default row label is `rowID`, but one may set a more descriptive row label for their data (such as `stargazer_id`). -* `COL_LABEL` Pilosa allows users to set a different column label for each index at index creation time. The default column label is `columnID`. +* `ROW_LABEL` The default row label is `rowID`, changing the default is deprecated. +* `COL_LABEL` The default column label is `columnID`, changing the default is deprecated. * `TIMESTAMP` This is a timestamp in quotes with the following format `"YYYY-MM-DDTHH:MM"` (e.g. "2006-01-02T15:04") * `UINT` An unsigned integer (e.g. 42839) * `ATTR_NAME` Must be a valid identifier `[A-Za-z][A-Za-z0-9._-]*` @@ -77,19 +77,19 @@ A return value of `false` indicates that the bit was already set to 1 and nothin **Examples:** ``` -SetBit(frame="stargazer", repo_id=10, stargazer_id=1) +SetBit(frame="stargazer", repo_id=10, rowID=1) ``` This query illustrates setting a bit in the stargazer frame. User with id=1 has starred repository with id=10. SetBit also supports providing a timestamp. To write the date that a user starred a repository. ``` -SetBit(frame="stargazer", repo_id=10, stargazer_id=1, timestamp="2016-01-01T00:00") +SetBit(frame="stargazer", repo_id=10, rowID=1, timestamp="2016-01-01T00:00") ``` Setting multiple bits in a single request: ``` -SetBit(frame="stargazer", repo_id=10, stargazer_id=1) SetBit(frame="stargazer", repo_id=10, stargazer_id=2) SetBit(frame="stargazer", repo_id=20, stargazer_id=1) SetBit(frame="stargazer", repo_id=30, stargazer_id=2) +SetBit(frame="stargazer", columnID=10, rowID=1) SetBit(frame="stargazer", columnID=10, rowID=2) SetBit(frame="stargazer", columnID=20, rowID=1) SetBit(frame="stargazer", columnID=30, rowID=2) ``` #### SetRowAttrs @@ -112,13 +112,13 @@ SetRowAttrs queries always return `null` upon success. **Examples:** ``` -SetRowAttrs(frame="stargazer", stargazer_id=10, username="mrpi", active=true) +SetRowAttrs(frame="stargazer", rowID=10, username="mrpi", active=true) ``` Set username value and active status for user 10. These are arbitrary key/value pairs which have no meaning to Pilosa. You can see the attributes you've set on a row with a [Bitmap]({{< ref "query-language.md#bitmap" >}}) query like so `Bitmap(frame="stargazer", stargazer_id=10)`. ``` -SetRowAttrs(frame="stargazer", stargazer_id=10, username=null) +SetRowAttrs(frame="stargazer", rowID=10, username=null) ``` Delete username value for user 10. @@ -144,13 +144,13 @@ SetColumnAttrs queries always return `null` upon success. Setting a value of `nu **Examples:** ``` -SetColumnAttrs(repo_id=10, stars=123, url="http://projects.pilosa.com/10", active=true) +SetColumnAttrs(columnID=10, stars=123, url="http://projects.pilosa.com/10", active=true) ``` Set url value and active status for project 10. These are arbitrary key/value pairs which have no meaning to Pilosa. You can see the attributes you've set on a column with a [Bitmap]({{< ref "query-language.md#bitmap" >}}) query like so `Bitmap(frame="stargazer", repo_id=10)`. ``` -SetColumnAttrs(repo_id=10, url=null) +SetColumnAttrs(columnID=10, url=null) ``` Delete url value for repo 10. @@ -178,7 +178,7 @@ A return value of `false` indicates that the bit was already set to 0 and nothin **Examples:** ``` -ClearBit(frame="stargazer", repo_id=10, stargazer_id=1) +ClearBit(frame="stargazer", columnID=10, rowID=1) ``` Remove relationship between stargazer_id 1 and repo_id 10 from the stargazer frame. @@ -206,7 +206,7 @@ e.g. `{"attrs":{"username":"mrpi","active":true},"bits":[10, 20]}` Query all repositories that user 1 has starred. ``` -Bitmap(frame="stargazer", stargazer_id=1) +Bitmap(frame="stargazer", rowID=1) ``` Returns `{"attrs":{"username":"mrpi","active":true},"bits":[10, 20]}` @@ -263,7 +263,7 @@ attrs will always be empty Query repositories which have been starred by two users. ``` -Intersect(Bitmap(frame="stargazer", stargazer_id=1), Bitmap(frame="stargazer", stargazer_id=2)) +Intersect(Bitmap(frame="stargazer", rowID=1), Bitmap(frame="stargazer", rowID=2)) ``` Returns `{"attrs":{},"bits":[10]}`. @@ -290,7 +290,7 @@ attrs will always be empty Query repositories which have been starred by one user and not another. ``` -Difference(Bitmap(frame="stargazer", stargazer_id=1), Bitmap( frame="stargazer", stargazer_id=2)) +Difference(Bitmap(frame="stargazer", rowID=1), Bitmap( frame="stargazer", rowID=2)) ``` Return `{"results":[{"attrs":{},"bits":[20]}]}` @@ -298,7 +298,7 @@ Return `{"results":[{"attrs":{},"bits":[20]}]}` * bits are repositories that were starred by user 1 BUT NOT user 2 ``` -Difference(Bitmap(frame="stargazer", stargazer_id=2), Bitmap( frame="stargazer", stargazer_id=1)) +Difference(Bitmap(frame="stargazer", rowID=2), Bitmap( frame="stargazer", rowID=1)) ``` Return `{"attrs":{},"bits":[30]}` @@ -322,7 +322,7 @@ Returns the number of set bits in the `BITMAP_CALL` passed in. Query the number of repositories to which a user has contributed. ``` -Count(Bitmap(frame="stargazer", stargazer_id=1)) +Count(Bitmap(frame="stargazer", rowID=1)) ``` Return `2` @@ -386,7 +386,7 @@ Returns `[{"key": 1, "count": 2}, {"key": 2, "count": 2}]` * Results are the top two users sorted by number of repositories they've starred in descending order. ``` -TopN(Bitmap(frame="language", language_id=1), frame="stargazer", n=2) +TopN(Bitmap(frame="language", rowID=1), frame="stargazer", n=2) ``` Returns `[{"key": 1, "count": 2}, {"key": 2, "count": 1}]` @@ -414,7 +414,7 @@ between the given `start` and `end` timestamps. When you set timestamp using SetBit, you will able to query all repositories that a user has starred within a date range. ``` -Range(frame="stargazer", stargazer_id=1, start="2010-01-01T00:00", end="2017-03-02T03:00") +Range(frame="stargazer", rowID=1, start="2010-01-01T00:00", end="2017-03-02T03:00") ``` Returns `{{"attrs":{},"bits":[10]}` From 89ecd5c0e8bccc94810283342947db14328efe0d Mon Sep 17 00:00:00 2001 From: Yuce Tekol Date: Wed, 27 Sep 2017 16:24:38 +0300 Subject: [PATCH 2/3] more updates --- docs/api-reference.md | 16 +++++++--------- docs/client-libraries.md | 31 ++++++++++++++++++------------- docs/getting-started.md | 20 ++++++++------------ docs/input-definition.md | 14 +++++--------- docs/webui.md | 6 +++--- 5 files changed, 41 insertions(+), 46 deletions(-) diff --git a/docs/api-reference.md b/docs/api-reference.md index bf599fafe..c11f2609f 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -47,13 +47,13 @@ Creates an index with the given name. The request payload is in JSON, and may contain the `options` field. The `options` field is a JSON object which may contain the following fields: -* `columnLabel` (string): column label of the index. +* `timeQuantum` (string): time quantum of the index. Request: ``` curl localhost:10101/index/user \ -X POST \ - -d '{"options": {"columnLabel": "user_id"}}' + -d '{"options": {"timeQuantum": "YMDH"}}' ``` Response: @@ -87,7 +87,7 @@ Request: ``` curl localhost:10101/index/user/query \ -X POST \ - -d 'Bitmap(frame="language", id=5)' + -d 'Bitmap(frame="language", rowID=5)' ``` Response: @@ -105,7 +105,7 @@ Request: ``` curl "localhost:10101/index/user/query?columnAttrs=true&slices=0,1" \ -X POST \ - -d 'Bitmap(frame="language", id=5)' + -d 'Bitmap(frame="language", rowID=5)' ``` Response: ``` @@ -157,7 +157,6 @@ Creates a frame in the given index with the given name. The request payload is in JSON, and may contain the `options` field. The `options` field is a JSON object which may contain the following fields: -* `rowLabel` (string): Row label of the frame. * `timeQuantum` (string): [Time Quantum]({{< ref "data-model.md#time-quantum" >}}) for this frame. * `inverseEnabled` (boolean): Enables [the inverted view]({{< ref "data-model.md#inverse" >}}) for this frame if `true`. * `cacheType` (string): [ranked]({{< ref "data-model.md#ranked" >}}) or [LRU]({{< ref "data-model.md#lru" >}}) caching on this frame. Default is `lru`. @@ -167,7 +166,7 @@ Request: ``` curl localhost:10101/index/user/frame/language \ -X POST \ - -d '{"options": {"rowLabel": "language_id"}}' + -d '{"options": {"inverseEnabled": true}}' ``` Response: @@ -231,7 +230,6 @@ Creates an input definition in the given index with the given name. The request payload is JSON, and it must contain the fields `frames` and `fields`. `frames` is an array of frames used within this input definition. Each frame must contain a `name` and may contain the following options: -* `rowLabel` (string): Row label of the frame. * `timeQuantum` (string): [Time Quantum]({{< ref "data-model.md#time-quantum" >}}) for this frame. * `inverseEnabled` (boolean): Enables [the inverted view]({{< ref "data-model.md#inverse" >}}) for this frame if `true`. * `cacheType` (string): [ranked]({{< ref "data-model.md#ranked" >}}) or [LRU]({{< ref "data-model.md#lru" >}}) caching on this frame. Default is `lru`. @@ -260,7 +258,7 @@ curl localhost:10101/index/user/input-definition/stargazer-input \ "frames":[ { "name": "language", - "options": {"rowLabel": "language_id"} + "options": {"inverseEnabled": true} } ], "fields":[ @@ -304,7 +302,7 @@ curl -XGET localhost:10101/index/user/input-definition/stargazer-input Response: ``` -{"frames":[{"name":"language","options":{"rowLabel":"language_id"}}],"fields":[{"name":"repo_id","primaryKey":true},{"name":"language_id","actions":[{"frame":"language","valueDestination":"mapping","valueMap":{"Go":5,"Python":17,"C++":10}}]}]} +{"frames":[{"name":"language","options":{"inverseEnabled":true}}],"fields":[{"name":"repo_id","primaryKey":true},{"name":"language_id","actions":[{"frame":"language","valueDestination":"mapping","valueMap":{"Go":5,"Python":17,"C++":10}}]}]} ``` ### Remove input definition diff --git a/docs/client-libraries.md b/docs/client-libraries.md index e896fbab4..4c776a04a 100644 --- a/docs/client-libraries.md +++ b/docs/client-libraries.md @@ -31,13 +31,17 @@ import ( func main() { // Let's create Index and Frame objects, which will contain the settings // for the corresponding indexes and frames. - repositoryOptions, := &pilosa.ColumnOptions{ColumnLabel: "repo_id"} - repository, _ := pilosa.NewIndex("repository", repositoryOptions) + repository, _ := pilosa.NewIndex("repository", nil) - stargazerOptions := &pilosa.RowOptions{RowLabel: "stargazer_id"} + stargazerOptions := &pilosa.RowOptions{ + TimeQuantum: pilosa.TimeQuantumYearMonthDay, + InverseEnabled: true, + } stargazer, _ := repository.Frame("stargazer", stargazerOptions) - languageOptions := &pilosa.RowOptions{RowLabel: "language_id"} + languageOptions := &pilosa.RowOptions{ + InverseEnabled: true, + } language, _ := repository.Frame("language", languageOptions) // We will just use the default client which assumes the server is at http://localhost:10101 @@ -99,13 +103,15 @@ We are going to use the index you have created in the [Getting Started](../getti Error handling has been omitted in the example below for brevity. ```python -from pilosa import Index, Client, PilosaError +from pilosa import Index, Client, PilosaError, TimeQuantum # Let's create Index and Frame objects, which will contain the settings # for the corresponding indexes and frames. -repository = Index("repository", column_label="repo_id") -stargazer = repository.frame("stargazer", row_label="stargazer_id") -language = repository.frame("language", row_label="language_id") +repository = Index("repository") +stargazer = repository.frame("stargazer", + time_quantum=TimeQuantum.YEAR_MONTH_DAY, + inverse_enabled=True) +language = repository.frame("language", inverse_enabled=True) # We will just use the default client which assumes the server is at http://localhost:10101 client = Client() @@ -161,18 +167,17 @@ public class StarTrace { public static void main(String[] args) { // Let's create Index and Frame objects, which will contain the settings // for the corresponding indexes and frames. - IndexOptions repositoryOptions = IndexOptions.builder() - .setColumnLabel("repo_id") - .build(); + IndexOptions repositoryOptions = IndexOptions.withDefaults(); Index repository = Index.withName("repository", repositoryOptions); FrameOptions stargazerOptions = FrameOptions.builder() - .setRowLabel("stargazer_id") + .setTimeQuantum(TimeQuantum.YEAR_MONTH_DAY) + .setInverseEnabled(true) .build(); Frame stargazer = repository.frame("stargazer", stargazerOptions); FrameOptions languageOptions = FrameOptions.builder() - .setRowLabel("language_id") + .setInverseEnabled(true) .build(); Frame language = repository.frame("language", languageOptions); diff --git a/docs/getting-started.md b/docs/getting-started.md index 534e733b4..c1eef3c08 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -54,9 +54,7 @@ curl localhost:10101/schema Before we can import data or run queries, we need to create our indexes and the frames within them. Let's create the repository index first: ``` -curl localhost:10101/index/repository \ - -X POST \ - -d '{"options": {"columnLabel": "repo_id"}}' +curl localhost:10101/index/repository -X POST ``` Repository IDs are the main focus of the `repository` index, so we chose `repo_id` as the column label. @@ -65,8 +63,7 @@ Let's create the `stargazer` frame which has user IDs of stargazers as its rows: ``` curl localhost:10101/index/repository/frame/stargazer \ -X POST \ - -d '{"options": {"rowLabel": "stargazer_id", - "timeQuantum": "YMD", + -d '{"options": {"timeQuantum": "YMD", "inverseEnabled": true}}' ``` @@ -78,8 +75,7 @@ Next up is the `language` frame, which will contain IDs for programming language ``` curl localhost:10101/index/repository/frame/language \ -X POST \ - -d '{"options": {"rowLabel": "language_id", - "inverseEnabled": true}}' + -d '{"options": {"inverseEnabled": true}}' ``` #### Import Data From CSV Files @@ -121,7 +117,7 @@ Which repositories did user 14 star: ``` curl localhost:10101/index/repository/query \ -X POST \ - -d 'Bitmap(frame="stargazer", stargazer_id=14)' + -d 'Bitmap(frame="stargazer", rowID=14)' ``` What are the top 5 languages in the sample data: @@ -135,28 +131,28 @@ Which repositories were starred by user 14 and 19: ``` curl localhost:10101/index/repository/query \ -X POST \ - -d 'Intersect(Bitmap(frame="stargazer", stargazer_id=14), Bitmap(frame="stargazer", stargazer_id=19))' + -d 'Intersect(Bitmap(frame="stargazer", rowID=14), Bitmap(frame="stargazer", rowID=19))' ``` Which repositories were starred by user 14 or 19: ``` curl localhost:10101/index/repository/query \ -X POST \ - -d 'Union(Bitmap(frame="stargazer", stargazer_id=14), Bitmap(frame="stargazer", stargazer_id=19))' + -d 'Union(Bitmap(frame="stargazer", rowID=14), Bitmap(frame="stargazer", rowID=19))' ``` Which repositories were starred by user 14 and 19 and also were written in language 1: ``` curl localhost:10101/index/repository/query \ -X POST \ - -d 'Intersect(Bitmap(frame="stargazer", stargazer_id=14), Bitmap(frame="stargazer", stargazer_id=19), Bitmap(frame="language", language_id=1))' + -d 'Intersect(Bitmap(frame="stargazer", rowID=14), Bitmap(frame="stargazer", rowID=19), Bitmap(frame="language", rowID=1))' ``` Set user 99999 as a stargazer for repository 77777: ``` curl localhost:10101/index/repository/query \ -X POST \ - -d 'SetBit(frame="stargazer", repo_id=77777, stargazer_id=99999)' + -d 'SetBit(frame="stargazer", columnID=77777, rowID=99999)' ``` ### What's Next? diff --git a/docs/input-definition.md b/docs/input-definition.md index c0c34ff7f..da59ee391 100644 --- a/docs/input-definition.md +++ b/docs/input-definition.md @@ -18,9 +18,7 @@ Input definitions allow users to define a schema based on their data and to prov Before creating a schema, let's create the repository index first: ``` -curl localhost:10101/index/repository \ - -X POST \ - -d '{"options": {"columnLabel": "repo_id"}}' +curl localhost:10101/index/repository -X POST ``` Then we can send the following input definition as JSON to Pilosa. The sample input defintion schema for the "Star Trace" project is at [Pilosa Getting Started repository](https://github.com/pilosa/getting-started), `input-definition.json` file @@ -32,7 +30,6 @@ curl localhost:10101/index/repository/input-definition/stargazer \ { "name": "language", "options": { - "rowLabel": "language_id", "inverseEnabled": true, "timeQuantum": "YMD" } @@ -40,7 +37,6 @@ curl localhost:10101/index/repository/input-definition/stargazer \ { "name": "stargazer", "options": { - "rowLabel": "stargazer_id", "inverseEnabled": true, "timeQuantum": "YMD" } @@ -127,9 +123,9 @@ The data input above is equivalent to the following `SetBit()` operations: ``` curl localhost:10101/index/repository/query \ -X POST \ - -d 'SetBit(frame="stargazer", repo_id=91720568, stargazer_id=513114) - SetBit(frame="stargazer", repo_id=91720568, stargazer_id=513114, timestamp="2017-05-18T20:40") - SetBit(frame="language", repo_id=91720568, language_id=5) - SetBit(frame="language", repo_id=95122322, language_id=17) + -d 'SetBit(frame="stargazer", columnID=91720568, rowID=513114) + SetBit(frame="stargazer", columnID=91720568, rowID=513114, timestamp="2017-05-18T20:40") + SetBit(frame="language", columnID=91720568, rowID=5) + SetBit(frame="language", columnID=95122322, rowID=17) ' ``` diff --git a/docs/webui.md b/docs/webui.md index 1e123393f..a348f59a6 100644 --- a/docs/webui.md +++ b/docs/webui.md @@ -30,10 +30,10 @@ In addition to standard PQL, the console supports a few special commands, prefix - `:create frame ` - `:delete frame ` -Index and frame creation also supports options like `columnLabel`,`rowLabel` or `inverseEnabled`. When creating new index or new frame, add options by using the keys documented in [API reference](../api-reference). +Index and frame creation also supports options like `timeQuantum` or `inverseEnabled`. When creating new index or new frame, add options by using the keys documented in [API reference](../api-reference). -- `:create index columnLabel=col_id` -- `:create frame rowLabel=row_id inverseEnabled=true cacheSize=10000` +- `:create index timeQuantum=YM` +- `:create frame inverseEnabled=true cacheSize=10000` ### Cluster Admin From a20adffd85d0d206fcd77691e31d11306cefc456 Mon Sep 17 00:00:00 2001 From: Yuce Tekol Date: Thu, 28 Sep 2017 19:06:56 +0300 Subject: [PATCH 3/3] zap remaining label ref --- docs/getting-started.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index c1eef3c08..84c5fd0bf 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -57,8 +57,6 @@ Before we can import data or run queries, we need to create our indexes and the curl localhost:10101/index/repository -X POST ``` -Repository IDs are the main focus of the `repository` index, so we chose `repo_id` as the column label. - Let's create the `stargazer` frame which has user IDs of stargazers as its rows: ``` curl localhost:10101/index/repository/frame/stargazer \