diff --git a/docs/getting-started.md b/docs/getting-started.md index 8535a4ac2..9c069cbd0 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -75,129 +75,6 @@ curl localhost:10101/index/repository/frame/language \ "inverseEnabled": true}}' ``` -#### Create the Schema Using an Input Definition - -Input definitions allow users to define a schema based on their data and to provide data to Pilosa in -a more standard format like JSON. Once an input definition is created, we can send data to Pilosa as JSON, and as long as the data adheres to the definition, Pilosa will internally perform all -of the appropriate mutations. - -Before creating a schema, let's create the repository index first: - -``` -curl localhost:10101/index/repository \ - -X POST \ - -d '{"options": {"columnLabel": "repo_id"}}' -``` -Then we can send the following input definition as JSON to Pilosa. The sample input defintion schema for the "Star Trace" project is at [Pilosa Getting Started repository](https://github.com/pilosa/getting-started), `input-definition.json` file - -``` -curl localhost:10101/index/repository/input-definition/stargazer \ - -X POST \ - -d '{ - "frames": [ - { - "name": "language", - "options": { - "inverseEnabled": true, - "timeQuantum": "YMD" - } - }, - { - "name": "stargazer", - "options": { - "inverseEnabled": true, - "timeQuantum": "YMD" - } - } - ], - "fields": [ - { - "name": "repo_id", - "primaryKey": true - }, - { - "actions": [ - { - "frame": "language", - "valueDestination": "mapping", - "valueMap": { - "C": 7, - "C#": 27, - "Go": 5, - "Java": 21, - "JavaScript": 13, - "Python": 17, - } - } - ], - "name": "language_id" - }, - { - "actions": [ - { - "frame": "stargazer", - "valueDestination": "value-to-row" - } - ], - "name": "stargazer_id" - }, - { - "actions": [ - { - "frame": "stargazer", - "valueDestination": "set-timestamp" - } - ], - "name": "time_value - } - ] - }' -``` - -Instead of creating a `stargazer` frame and a `language` frame individually like above, we can create multiple frames in one input definition. -We can also set `repo_id` for multiple frames at the same time by providing field actions. There are three options for valueDestination: - - - value-to-row: The value for this field is used as the `rowID`. - - single-row-boolean: The value must be a boolean, and this specifies `SetBit()` or `ClearBit()`, a `rowID` must be specified for this destination type. - - mapping: The value for this field is used to lookup a `rowID` in a map. A valueMap is required for this destination type. - - set-timestamp: The value for this field is used to lookup timestamp and set timestamp for the whole frame - -#### Import Data Using an Input Definition - -The sample data for the "Star Trace" project is at [Pilosa Getting Started repository](https://github.com/pilosa/getting-started). - -If you import data using an input definition, download the `json-input.json` file in that repo, then run the following request using the input definition created above: - -``` -curl localhost:10101/index/repository/input/stargazer \ - -X POST \ - -d '[ - { - "language_id": "Go", - "repo_id": 91720568, - "stargazer_id": 513114 - "time_value": "2017-05-18T20:40" - }, - { - "language_id": "Python", - "repo_id": 95122322 - }' - ] -``` - -As defined in the input definition, field name `language_id` maps language to a corresponding id defined in `valueMap` and sets the appropriate bit in the `language` frame. The value corresponding to field name `stargazer_id` is added to the `stargazer` frame as rowID. -The data input above is equivalent to the following `SetBit()` operations: - -``` -curl localhost:10101/index/repository/query \ - -X POST \ - -d 'SetBit(frame="stargazer", repo_id=91720568, stargazer_id=513114) - 'SetBit(frame="stargazer", repo_id=91720568, stargazer_id=513114, timestamp="2017-05-18T20:40") - SetBit(frame="language", repo_id=91720568, language_id=5) - SetBit(frame="language", repo_id=95122322, language_id=17) - ' -``` - #### Import Data From CSV Files If you import data using csv files and without input defintion, download the `stargazer.csv` and `language.csv` files in that repo. @@ -224,6 +101,10 @@ docker exec -it pilosa /pilosa import -i repository -f language /language.csv Note that, both the user IDs and the repository IDs were remapped to sequential integers in the data files, they don't correspond to actual Github IDs anymore. You can check out `language.txt` to see the mapping for languages. +### Input Definition +Alternatively Pilosa can import JSON data using a definition file describing the schema and ETL rules to process the data. +[Input Definition](../input-definition/) + #### Make Some Queries
diff --git a/docs/input-definition.md b/docs/input-definition.md new file mode 100644 index 000000000..4b73c4991 --- /dev/null +++ b/docs/input-definition.md @@ -0,0 +1,128 @@ ++++ +title = "Input Definition" ++++ + +## Input Definition +This document builds on the data import concepts introduced in [Getting Started](../getting-started/). +Here we will demonstrate creating the index's schema and data definition. Then using this definition to import JSON data. + +#### Create the Schema Using an Input Definition + +Input definitions allow users to define a schema based on their data and to provide data to Pilosa in a more standard format like JSON. Once an input definition is created, we can send data to Pilosa as JSON, and as long as the data adheres to the definition, Pilosa will internally perform all of the appropriate mutations. + +Before creating a schema, let's create the repository index first: + +``` +curl localhost:10101/index/repository \ + -X POST \ + -d '{"options": {"columnLabel": "repo_id"}}' +``` +Then we can send the following input definition as JSON to Pilosa. The sample input defintion schema for the "Star Trace" project is at [Pilosa Getting Started repository](https://github.com/pilosa/getting-started), `input-definition.json` file + +``` +curl localhost:10101/index/repository/input-definition/stargazer \ + -X POST \ + -d '{ + "frames": [ + { + "name": "language", + "options": { + "inverseEnabled": true, + "timeQuantum": "YMD" + } + }, + { + "name": "stargazer", + "options": { + "inverseEnabled": true, + "timeQuantum": "YMD" + } + } + ], + "fields": [ + { + "name": "repo_id", + "primaryKey": true + }, + { + "actions": [ + { + "frame": "language", + "valueDestination": "mapping", + "valueMap": { + "C": 7, + "C#": 27, + "Go": 5, + "Java": 21, + "JavaScript": 13, + "Python": 17, + } + } + ], + "name": "language_id" + }, + { + "actions": [ + { + "frame": "stargazer", + "valueDestination": "value-to-row" + } + ], + "name": "stargazer_id" + }, + { + "actions": [ + { + "frame": "stargazer", + "valueDestination": "set-timestamp" + } + ], + "name": "time_value + } + ] + }' +``` + +Instead of creating a `stargazer` frame and a `language` frame individually like in [Getting Started](../getting-started/), we can create multiple frames in one input definition. +We can also set `repo_id` for multiple frames at the same time by providing field actions. There are three options for valueDestination: + + - value-to-row: The value for this field is used as the `rowID`. + - single-row-boolean: The value must be a boolean, and this specifies `SetBit()` or `ClearBit()`, a `rowID` must be specified for this destination type. + - mapping: The value for this field is used to lookup a `rowID` in a map. A valueMap is required for this destination type. + - set-timestamp: The value for this field is used to lookup timestamp and set timestamp for the whole frame + +#### Import Data Using an Input Definition + +The sample data for the "Star Trace" project is at [Pilosa Getting Started repository](https://github.com/pilosa/getting-started). + +If you import data using an input definition, download the `json-input.json` file in that repo, then run the following request using the input definition created above: + +``` +curl localhost:10101/index/repository/input/stargazer \ + -X POST \ + -d '[ + { + "language_id": "Go", + "repo_id": 91720568, + "stargazer_id": 513114 + "time_value": "2017-05-18T20:40" + }, + { + "language_id": "Python", + "repo_id": 95122322 + }' + ] +``` + +As defined in the input definition, field name `language_id` maps language to a corresponding id defined in `valueMap` and sets the appropriate bit in the `language` frame. The value corresponding to field name `stargazer_id` is added to the `stargazer` frame as rowID. +The data input above is equivalent to the following `SetBit()` operations: + +``` +curl localhost:10101/index/repository/query \ + -X POST \ + -d 'SetBit(frame="stargazer", repo_id=91720568, stargazer_id=513114) + 'SetBit(frame="stargazer", repo_id=91720568, stargazer_id=513114, timestamp="2017-05-18T20:40") + SetBit(frame="language", repo_id=91720568, language_id=5) + SetBit(frame="language", repo_id=95122322, language_id=17) + ' +``` \ No newline at end of file