diff --git a/docs/api-reference.md b/docs/api-reference.md index 206c30f55..52ea2c369 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -13,13 +13,10 @@ nav = [] Returns the schema of all indexes in JSON. -Request: -``` +``` request curl -XGET localhost:10101/index ``` - -Response: -``` +``` response {"indexes":[{"name":"user","frames":[{"name":"collab"}]}]} ``` @@ -29,13 +26,10 @@ Response: Returns the schema of the specified index in JSON. -Request: -``` +``` request curl -XGET localhost:10101/index/user ``` - -Response: -``` +``` response {"index":{"name":"user"}, "frames":[{"name":"collab"}]}]} ``` @@ -45,13 +39,10 @@ Response: Creates an index with the given name. -Request: -``` +``` request curl -XPOST localhost:10101/index/user ``` - -Response: -``` +``` response {} ``` @@ -61,13 +52,10 @@ Response: Removes the given index. -Request: -``` +``` request curl -XDELETE localhost:10101/index/user ``` - -Response: -``` +``` response {} ``` @@ -77,15 +65,12 @@ Response: Sends a query to the Pilosa server with the given index. The request body is UTF-8 encoded text and response body is in JSON by default. -Request: -``` +``` request curl localhost:10101/index/user/query \ -X POST \ -d 'Bitmap(frame="language", rowID=5)' ``` - -Response: -``` +``` response {"results":[{"attrs":{},"bits":[100]}]} ``` @@ -95,14 +80,12 @@ The response doesn't include column attributes by default. To return them, set t The query is executed for all [slices](../data-model#slice) by default. To use specified slices only, set the `slices` query argument to a comma-separated list of slice indices. -Request: -``` +``` request curl "localhost:10101/index/user/query?columnAttrs=true&slices=0,1" \ -X POST \ -d 'Bitmap(frame="language", rowID=5)' ``` -Response: -``` +``` response { "results":[{"attrs":{},"bits":[100]}], "columnAttrs":[{"id":100,"attrs":{"name":"Klingon"}}] @@ -134,21 +117,21 @@ Each individual `field` contains the following: Integer fields are stored as n-bit range-encoded values. Pilosa supports 63-bit, signed integers with values between `min` and `max`. -Request: -``` +``` request curl localhost:10101/index/user/frame/language \ -X POST \ -d '{"options": {"inverseEnabled": true}}' ``` - +``` response +{} ``` + +``` request curl localhost:10101/index/repository/frame/stats \ -X POST \ -d '{"rangeEnabled": true, "fields": [{"name": "pullrequests", "type": "int", "min": 0, "max": 1000000}]}' ``` - -Response: -``` +``` response {} ``` @@ -158,13 +141,10 @@ Response: Removes the given frame. -Request: -``` +``` request curl -XDELETE localhost:10101/index/user/frame/language ``` - -Response: -``` +``` response {} ``` @@ -188,15 +168,12 @@ The payload is in JSON with the format: `{"timeQuantum": "${TIME_QUANTUM}"}`. Va * MDH: month, day and hour * YMDH: year, month, day and hour -Request: -``` +``` request curl localhost:10101/index/user/frame/language/time-quantum \ -X POST \ -d '{"timeQuantum": "YM"}' ``` - -Response: -``` +``` response {} ``` @@ -211,15 +188,12 @@ The request payload is JSON, and it must contain the fields `type`, `min`, `max` * `min` (int): Minimum value allowed for this field. * `max` (int): Maximum value allowed for this field. -Request: -``` +``` request curl localhost:10101/index/repository/frame/stats/field/pullrequests \ -X POST \ -d '{"type": "int", "min": 0, "max": 1000000}' ``` - -Response: -``` +``` response {} ``` @@ -251,8 +225,7 @@ The `action` describes how the field value will be processed. Each `action` may - `mapping`: Map the value to a RowID in the `valueMap`. * `valueMap` (object): string and integer pairs used to map field values to RowID's. -Request: -``` +``` request curl localhost:10101/index/user/input-definition/stargazer-input \ -X POST \ -d '{ @@ -284,9 +257,7 @@ curl localhost:10101/index/user/input-definition/stargazer-input \ ] }' ``` - -Response: -``` +``` response {} ``` @@ -296,13 +267,10 @@ Response: Returns the given input definition as JSON. -Request: -``` +``` request curl -XGET localhost:10101/index/user/input-definition/stargazer-input ``` - -Response: -``` +``` response {"frames":[{"name":"language","options":{"inverseEnabled":true}}],"fields":[{"name":"repo_id","primaryKey":true},{"name":"language_id","actions":[{"frame":"language","valueDestination":"mapping","valueMap":{"Go":5,"Python":17,"C++":10}}]}]} ``` @@ -312,13 +280,10 @@ Response: Removes the given input definition. -Request: -``` +``` request curl -XDELETE localhost:10101/index/user/input-definition/stargazer-input ``` - -Response: -``` +``` response {} ``` @@ -330,15 +295,12 @@ Processes the JSON payload using the given input definition. The request payload is a JSON array of objects containing one field for the primary key that corresponds to the column label, and additional fields that will be handled by corresponding actions in the input definition. -Request: -``` +``` request curl localhost:10101/index/user/input/stargazer-input \ -X POST \ -d '[{"language_id": "Go", "repo_id": 92274475}]' ``` - -Response: -``` +``` response {} ``` @@ -348,13 +310,10 @@ Response: Returns the hosts in the cluster. -Request: -``` +``` request curl -XGET localhost:10101/hosts ``` - -Response: -``` +``` response [{"host":":10101"}] ``` @@ -364,13 +323,10 @@ Response: Returns the version of the Pilosa server. -Request: -``` +``` request curl -XGET localhost:10101/version ``` - -Response: -``` +``` response {"version":"v0.6.0"} ``` diff --git a/docs/getting-started.md b/docs/getting-started.md index 35042d59d..d4802e8ee 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -31,11 +31,12 @@ docker run -it --rm --name pilosa -p 10101:10101 pilosa/pilosa:latest ``` Let's make sure Pilosa is running: -``` +``` request curl localhost:10101/status ``` - -Which should output: `{"status":{"Nodes":[{"Host":":10101","State":"UP"}]}}` +``` response +{"status":{"Nodes":[{"Host":":10101","State":"UP"}]}} +``` ### Sample Project @@ -45,30 +46,41 @@ Although Pilosa doesn't keep the data in a tabular format, we still use the term #### Create the Schema -Note: The queries in this section which are used to set up the indexes in Pilosa just the empty object on success: `{}` - if you would like to verify that a query worked as you expected, you can request the schema as follows: -``` +``` request curl localhost:10101/schema +``` +``` response {"indexes":null} ``` Before we can import data or run queries, we need to create our indexes and the frames within them. Let's create the repository index first: -``` +``` request curl localhost:10101/index/repository -X POST ``` +``` response +{} +``` Let's create the `stargazer` frame which has user IDs of stargazers as its rows: -``` +``` request curl localhost:10101/index/repository/frame/stargazer \ -X POST \ -d '{"options": {"timeQuantum": "YMD"}}' ``` +``` response +{} +``` Since our data contains time stamps for the time users starred repos, we set the *time quantum* for the `stargazer` frame in the options as well. Time quantum is the resolution of the time we want to use, and we set it to `YMD` (year, month, day) for `stargazer`. Next up is the `language` frame, which will contain IDs for programming languages: +``` request +curl localhost:10101/index/repository/frame/language \ + -X POST ``` -curl localhost:10101/index/repository/frame/language -X POST +``` response +{} ``` #### Import Data From CSV Files @@ -107,46 +119,112 @@ Alternatively Pilosa can import JSON data using an [Input Definition](../input-d Which repositories did user 14 star: -``` +``` request curl localhost:10101/index/repository/query \ -X POST \ -d 'Bitmap(frame="stargazer", rowID=14)' ``` +``` response +{ + "results":[ + { + "attrs":{}, + "bits":[1,2,3,362,368,391,396,409,416,430,436,450,454,460,461,464,466,469,470,483,484,486,490,491,503,504,514] + } + ] +} +``` What are the top 5 languages in the sample data: -``` +``` request curl localhost:10101/index/repository/query \ -X POST \ -d 'TopN(frame="language", n=5)' ``` +``` response +{ + "results":[ + [ + {"id":5,"count":119}, + {"id":1,"count":50}, + {"id":4,"count":48}, + {"id":9,"count":31}, + {"id":13,"count":25} + ] + ] +} +``` Which repositories were starred by user 14 and 19: -``` +``` request curl localhost:10101/index/repository/query \ -X POST \ - -d 'Intersect(Bitmap(frame="stargazer", rowID=14), Bitmap(frame="stargazer", rowID=19))' + -d 'Intersect( + Bitmap(frame="stargazer", rowID=14), + Bitmap(frame="stargazer", rowID=19) + )' +``` +``` response +{ + "results":[ + { + "attrs":{}, + "bits":[2,3,362,396,416,461,464,466,470,486] + } + ] +} ``` Which repositories were starred by user 14 or 19: -``` +``` request curl localhost:10101/index/repository/query \ -X POST \ - -d 'Union(Bitmap(frame="stargazer", rowID=14), Bitmap(frame="stargazer", rowID=19))' + -d 'Union( + Bitmap(frame="stargazer", rowID=14), + Bitmap(frame="stargazer", rowID=19) + )' +``` +``` response +{ + "results":[ + { + "attrs":{}, + "bits":[1,2,3,361,362,368,376,377,378,382,386,388,391,396,398,400,409,411,412,416,426,428,430,435,436,450,452,453,454,456,460,461,464,465,466,469,470,483,484,486,487,489,490,491,500,503,504,505,512,514] + } + ] +} ``` Which repositories were starred by user 14 and 19 and also were written in language 1: -``` +``` request curl localhost:10101/index/repository/query \ -X POST \ - -d 'Intersect(Bitmap(frame="stargazer", rowID=14), Bitmap(frame="stargazer", rowID=19), Bitmap(frame="language", rowID=1))' + -d 'Intersect( + Bitmap(frame="stargazer", rowID=14), + Bitmap(frame="stargazer", rowID=19), + Bitmap(frame="language", rowID=1) + )' +``` +``` response +{ + "results":[ + { + "attrs":{}, + "bits":[2,362,416,461] + } + ] +} ``` Set user 99999 as a stargazer for repository 77777: -``` +``` request curl localhost:10101/index/repository/query \ -X POST \ -d 'SetBit(frame="stargazer", columnID=77777, rowID=99999)' ``` +``` response +{"results":[true]} +``` ### What's Next? diff --git a/docs/query-language.md b/docs/query-language.md index f28257fe6..0e02351ab 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -34,11 +34,14 @@ The default row label is `rowID`, and the default column label is `columnID`. Ch Before running any of the example queries below, follow the instructions in the [Getting Started](../getting-started) section to set up an index, frames, and populate them with some data. The examples just show the PQL quer(ies) needed - to run the query `SetBit(frame="stargazer", columnID=10, rowID=1)` against a server using curl, you would: -``` +``` request curl localhost:10101/index/repository/query \ -X POST \ -d 'SetBit(frame="stargazer", columnID=10, rowID=1)' ``` +``` response +{"results":[true]} +``` #### Arguments and Types diff --git a/docs/tutorials.md b/docs/tutorials.md index 9c19f3a67..4d787577c 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -178,38 +178,53 @@ curl -k --ipv4 https://01.pilosa.local:10501/status The `-k` flag is used to tell curl that it shouldn't bother with checking the certificate the server provides and `--ipv4` workarounds an issue on MacOS where the curl requests take a long time if the address resolves to `127.0.0.1`. You can leave it out on Linux and WSL. All nodes should be in the `UP` state: -``` +``` response {"status":{"Nodes":[{"Host":"01.pilosa.local:10501","State":"UP"},{"Host":"02.pilosa.local:10502","State":"UP"},{"Host":"03.pilosa.local:10503","State":"UP"}]}} ``` #### Running Queries Having confirmed that our cluster is running OK, let's run a few queries. But before that, we need to create an index and a frame: -``` +``` request curl -k --ipv4 https://01.pilosa.local:10501/index/sample-index -d '' ``` +``` response +{} +``` This will create index `sample-index` with default options. Let's create the frame now: -``` +``` request curl -k --ipv4 https://01.pilosa.local:10501/index/sample-index/frame/sample-frame -d '' ``` +``` response +{} +``` We just created frame `sample-frame` with default options. Let's run a `SetBit` query: -``` +``` request curl -k --ipv4 https://01.pilosa.local:10501/index/sample-index/query -d 'SetBit(frame="sample-frame", rowID=1, columnID=100)' ``` +``` response +{"results":[true]} +``` Confirm that the bit was indeed set: -``` +``` request curl -k --ipv4 https://01.pilosa.local:10501/index/sample-index/query -d 'Bitmap(frame="sample-frame", rowID=1)' ``` +``` response +{"results":[{"attrs":{},"bits":[100]}]} +``` The same response should be returned when querying other nodes in the cluster: -``` +``` request curl -k --ipv4 https://02.pilosa.local:10502/index/sample-index/query -d 'Bitmap(frame="sample-frame", rowID=1)' ``` +``` response +{"results":[{"attrs":{},"bits":[100]}]} +``` #### What's Next? @@ -223,20 +238,16 @@ Check out our [Administration Guide](https://www.pilosa.com/docs/latest/administ Pilosa can store integer values associated to the columns in an index, and those values are used to support range and aggregate queries. In this tutorial we will show how to set up integer fields, populate those fields with data, and query the fields. The example index we're going to create will represent fictional patients at a medical facility and various bits of information about those patients. First, create an index called `patients`: -``` +``` request curl localhost:10101/index/patients \ -X POST ``` - -Next, create a frame in the `patients` index called `measurements` which will represent information gathered about each patient. -``` -curl localhost:10101/index/patients/frame/measurements \ - -X POST \ - -d '{"options":{"rangeEnabled": true}}' +``` response +{} ``` In addition to storing rows of bits, a frame can also contain fields that store integer values. The next step creates three fields (`age`, `weight`, `tcells`) in the `measurements` frame. -``` +``` request curl localhost:10101/index/patients/frame/measurements \ -X POST \ -d '{"options":{ @@ -248,17 +259,23 @@ curl localhost:10101/index/patients/frame/measurements \ ] }}' ``` +``` response +{} +``` If you need to, you can add fields to an existing frame by posting to the [Create Field endpoint](../api-reference/#create-field). Next, let's populate our fields with data. There are two ways to get data into fields: use the `SetFieldValue()` PQL function to set fields individually, or use the `pilosa import` command to import many values at once. First, let's set some field data using PQL. This query sets the age, weight, and t-cell count for the patient with ID `1` in our system: -``` +``` request curl localhost:10101/index/patients/query \ -X POST \ -d 'SetFieldValue(columnID=1, frame="measurements", age=34, weight=128, tcells=1145)' ``` +``` response +{"results":[null]} +``` In the case where we need to load a lot of data at once, we can use the `pilosa import` command. This method lets us import data into Pilosa from a CSV file. @@ -282,39 +299,38 @@ pilosa import -i patients -f measurements --field age ages.csv Now that we have some data in our index, let's run a few queries to demonstrate how to use that data. In order to find all patients over the age of 40, then simply run a `Range` query against the `age` field. -``` +``` request curl localhost:10101/index/patients/query \ -X POST \ -d 'Range(frame="measurements", age > 40)' ``` -You should get the following results: -``` +``` response {"results":[{"attrs":{},"bits":[2,6,9]}]} ``` You can find a list of supported range operators in the [Range Query](../query-language/#range-bsi) documentation. To find the average age of all patients, run a `Sum` query: -``` +``` request curl localhost:10101/index/patients/query \ -X POST \ -d 'Sum(frame="measurements", field="age")' ``` -The results you get from the `Sum` query contain the `sum` of all values as well as the `count` of columns with a value. To get the average you can just divide `sum` by `count`. -``` +``` response {"results":[{"sum":377,"count":9}]} ``` +The results you get from the `Sum` query contain the `sum` of all values as well as the `count` of columns with a value. To get the average you can just divide `sum` by `count`. You can also provide a filter to the `Sum()` function, to find the average age of all patients over 40. -``` +``` request curl localhost:10101/index/patients/query \ -X POST \ -d 'Sum(Range(frame="measurements", age > 40), frame="measurements", field="age")' ``` -Notice in this case that the count is only `3` because of the `age > 40` filter applied to the query. -``` +``` response {"results":[{"sum":191,"count":3}]} ``` +Notice in this case that the count is only `3` because of the `age > 40` filter applied to the query. ### Storing Row and Column Attributes @@ -323,20 +339,26 @@ Notice in this case that the count is only `3` because of the `age > 40` filter Pilosa can store arbitrary values associated to any row or column. In Pilosa, these are referred to as `attributes`, and they can be of type `string`, `integer`, `boolean`, or `float`. In this tutorial we will store some attribute data and then run some queries that return that data. First, create an index called `books` to use for this tutorial: -``` +``` request curl localhost:10101/index/books \ -X POST ``` +``` response +{} +``` Next, create a frame in the `books` index called `members` which will represent library members who have read books. -``` +``` request curl localhost:10101/index/books/frame/members \ -X POST \ -d '{}' ``` +``` response +{} +``` Now, let's add some books to our index. -``` +``` request curl localhost:10101/index/books/query \ -X POST \ -d 'SetColumnAttrs(columnID=1, name="To Kill a Mockingbird", year=1960) @@ -345,9 +367,12 @@ curl localhost:10101/index/books/query \ SetColumnAttrs(columnID=4, name="Out Stealing Horses", year=2003) SetColumnAttrs(columnID=5, name="The Forever War", year=2008)' ``` +``` response +{"results":[null,null,null,null,null]} +``` And add some members. -``` +``` request curl localhost:10101/index/books/query \ -X POST \ -d 'SetRowAttrs(frame="members", rowID=10001, fullName="John Smith") @@ -356,20 +381,22 @@ curl localhost:10101/index/books/query \ SetRowAttrs(frame="members", rowID=10004, fullName="Pedro Vazquez") SetRowAttrs(frame="members", rowID=10005, fullName="Pat Washington")' ``` +``` response +{"results":[null,null,null,null,null]} +``` At this point we can query one of the `member` records by querying that row. -``` +``` request curl localhost:10101/index/books/query \ -X POST \ -d 'Bitmap(frame="members", rowID=10002)' ``` -You should get the following result set: -``` +``` response {"results":[{"attrs":{"fullName":"Sue Perkins"},"bits":[]}]} ``` Now let's add some data to the matrix such that each pair represents a member who has read that book. -``` +``` request curl localhost:10101/index/books/query \ -X POST \ -d 'SetBit(frame="members", rowID=10001, columnID=3) @@ -390,28 +417,28 @@ curl localhost:10101/index/books/query \ SetBit(frame="members", rowID=10005, columnID=4) SetBit(frame="members", rowID=10005, columnID=5)' ``` +``` response +{"results":[true,true,true,true,true,true,true,true,true,true,true,true,true]} +``` Now pull the record for `Sue Perkins` again. -``` +``` request curl localhost:10101/index/books/query \ -X POST \ -d 'Bitmap(frame="members", rowID=10002)' ``` -Notice that the result set now contains a list of integers in the `bits` attribute. These integers match the column IDs of the books that Sue has read. -``` +``` response {"results":[{"attrs":{"fullName":"Sue Perkins"},"bits":[1,2,4]}]} ``` +Notice that the result set now contains a list of integers in the `bits` attribute. These integers match the column IDs of the books that Sue has read. In order to retrieve the attribute information that we stored for each book, we need to add a URL parameter `columnAttrs=true` to the query. -``` +``` request curl localhost:10101/index/books/query?columnAttrs=true \ -X POST \ -d 'Bitmap(frame="members", rowID=10002)' ``` - -Here, the `book` attributes will be included in the result set at the `columnAttrs` attribute. - -``` +``` response { "results":[{"attrs":{"fullName":"Sue Perkins"},"bits":[1,2,4]}], "columnAttrs":[ @@ -421,15 +448,15 @@ Here, the `book` attributes will be included in the result set at the `columnAtt ] } ``` +The `book` attributes are included in the result set at the `columnAttrs` attribute. Finally, if we want to find out which books were read by both `Sue` and `Pedro`, we just perform an `Intersect` query on those two members: -``` +``` request curl localhost:10101/index/books/query?columnAttrs=true \ -X POST \ -d 'Intersect(Bitmap(frame="members", rowID=10002), Bitmap(frame="members", rowID=10004))' ``` - -``` +``` response { "results":[{"attrs":{},"bits":[4]}], "columnAttrs":[