From 59e94b451c8a3930111a7b1021a433c488f07ff1 Mon Sep 17 00:00:00 2001 From: Ashley Svetlik Date: Tue, 25 Jun 2019 13:34:54 -0500 Subject: [PATCH 01/17] Removed pilosa import and added go --- docs/getting-started.md | 299 +++++++++++++++++++++++++--------------- 1 file changed, 189 insertions(+), 110 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index c7dd54a8c..2e0104591 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -44,6 +44,49 @@ In order to better understand Pilosa's capabilities, we will create a sample pro Although Pilosa doesn't keep the data in a tabular format, we still use the terms "columns" and "rows" when describing the data model. We put the primary objects in columns, and the properties of those objects in rows. For example, the Star Trace project will contain an index called "repository" which contains columns representing Github repositories, and rows representing properties like programming languages and tags. We can better organize the rows by grouping them into sets called Fields. So the "repository" index might have a "languages" field as well as a "tags" field. You can learn more about indexes and fields in the [Data Model](../data-model/) section of the documentation. +#### Create the Environment + +While we can create indexes and query directly in the terminal, it is more conventional to do so in a client library. Pilosa supports Go, Java, and Python, though you will have to install the library for compatibility. + +For Go users, open a terminal (one other than the one running pilosa) and download the library in your `GOPATH` using: +``` +go get github.com/pilosa/go-pilosa +``` + +For Java users, add the following dependency in your `pom.xml`: +``` + + + com.pilosa + pilosa-client + 1.3.1 + + +``` + +For Python users, open a terminal (one other than the one running pilosa) and install the library using: +``` +pip install pilosa +``` + +For simplicity, we reccomend that you create a separate folder for this project. In the terminal, create a new folder as follows: +``` +mkdir GettingStarted +cd GettingStarted +``` + +In this folder, we will download two CSV files to provide data to our fields later on. Download the stargazer.csv and language.csv files here: +``` +curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/stargazer.csv +curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/language.csv +``` + +We will also create a file called StarTrace.go (for Go users), StarTrace.java (for Java users), or StarTrace.py (for Python users) as follows: +``` +touch StarTrace.go +``` +This file will be used in the following section. + #### Create the Schema Note: @@ -55,184 +98,220 @@ curl localhost:10101/schema ``` response {"indexes":null} ``` +##### Go Users -Before we can import data or run queries, we need to create our indexes and the fields within them. Let's create the repository index first: -``` request -curl localhost:10101/index/repository -X POST +Before we can import data or run queries, we need to create our indexes and the fields within them. Let's create the repository index first. Copy the following into the StarTrace.go file: +``` +package main + +import ( + "bytes" + "fmt" + "github.com/pilosa/go-pilosa" + "github.com/pilosa/go-pilosa/csv" + "io/ioutil" + "log" +) + +func main() { + // Create the Schema + client := pilosa.DefaultClient() + schema, _ := client.Schema() + repository := schema.Index("repository") + // This is where the field will go later + err := client.SyncSchema(schema) + if err != nil { + log.Fatal(err) + } +} ``` -``` response -{"success":true} -``` -The index name must be 64 characters or less, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. +The index name must be 64 characters or less, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. The same goes for field names. Let's create the `stargazer` field which has user IDs of stargazers as its rows: -``` request -curl localhost:10101/index/repository/field/stargazer \ - -X POST \ - -d '{"options": {"type": "time", "timeQuantum": "YMD"}}' ``` -``` response -{"success":true} + stargazer := repository.Field("stargazer") ``` -Since our data contains time stamps which represent the time users starred repos, we set the field type to `time`. Time quantum is the resolution of the time we want to use, and we set it to `YMD` (year, month, day) for `stargazer`. - Next up is the `language` field, which will contain IDs for programming languages: -``` request -curl localhost:10101/index/repository/field/language \ - -X POST ``` -``` response -{"success":true} + language := repository.Field("language") ``` -The `language` is a `set` field, but since the default field type is `set`, we didn't specify it in field options. +Your StarTrace.go file should look like: +``` +package main + +import ( + "bytes" + "fmt" + "github.com/pilosa/go-pilosa" + "github.com/pilosa/go-pilosa/csv" + "io/ioutil" + "log" +) + +func main() { + // Create the Schema + client := pilosa.DefaultClient() + schema, _ := client.Schema() + repository := schema.Index("repository") + stargazer := repository.Field("stargazer") + language := repository.Field("language") + err := client.SyncSchema(schema) + if err != nil { + log.Fatal(err) + } +} +``` + +##### Java and Python Users + +
+

Java and Python support will be uploaded shortly. +

#### Import Data From CSV Files +Now that we have our index and our fields, we can import the data we downloaded earlier and soon be making our own queries. + +##### Go Users + +First, we will load our data into the `stargazer` field: +``` + stargazerFile, err := ioutil.ReadFile("stargazer.csv") + if err != nil { + log.Fatal(err) + } + format := "2006-01-02T15:04" + iterator = csv.NewColumnIteratorWithTimestampFormat(csv.RowIDColumnID, bytes.NewReader(stargazerFile), format) + err = client.ImportField(stargazer, iterator) + if err != nil { + log.Fatal(err) + } +``` +Since our `stargazer` data contains time stamps, which represent the time users starred repos, we will be using the `csv.NewColumnIterator` function that is built into the go-pilosa import. For more information on imports in go-pilosa, please see the go-pilosa [site](https://github.com/pilosa/go-pilosa/blob/master/docs/imports-exports.md). Time quantum is the resolution of the time we want to use and is defined by the `format` variable. + +Next, we will load our data into the `language` field: +``` + languageFile, err := ioutil.ReadFile("language.csv") + if err != nil { + log.Fatal(err) + } + iterator := csv.NewColumnIterator(csv.RowIDColumnID, bytes.NewReader(languageFile)) + err = client.ImportField(language, iterator) + if err != nil { + log.Fatal(err) + } +``` +The `language` is a `set` field, but since the default field type is `set`, we didn't need to specify it. + +##### Java and Python Users +
-

For demonstration purposes, we're using Pilosa's built in utility to import specially formatted CSV files. For more general usage, see how the various client libraries expose the bulk import functionality in Go, Java, and Python.

+

Java and Python support will be uploaded shortly.

- -Download the `stargazer.csv` and `language.csv` files here: - -``` -curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/stargazer.csv -curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/language.csv -``` - -Run the following commands to import the data into Pilosa: - -``` -pilosa import -i repository -f stargazer stargazer.csv -pilosa import -i repository -f language language.csv -``` - -If you are using a Docker container for Pilosa (with name `pilosa`), you should instead copy the `*.csv` file into the container and then import them: +
+

If you are using a Docker container for Pilosa (with name `pilosa`), you should instead copy the `*.csv` file into the container and then import them: ``` docker cp stargazer.csv pilosa:/stargazer.csv docker exec -it pilosa /pilosa import -i repository -f stargazer /stargazer.csv docker cp language.csv pilosa:/language.csv docker exec -it pilosa /pilosa import -i repository -f language /language.csv ``` +

Note that both the user IDs and the repository IDs were remapped to sequential integers in the data files, they don't correspond to actual Github IDs anymore. You can check out [languages.txt](https://github.com/pilosa/getting-started/blob/master/languages.txt) to see the mapping for languages. #### Make Some Queries +Now that we have a working schema, we can query it. + +##### Go Users + Which repositories did user 14 star: ``` request -curl localhost:10101/index/repository/query \ - -X POST \ - -d 'Row(stargazer=14)' + response, err := client.Query(stargazer.Row(14)) + if err != nil { + log.Fatal(err) + } + fmt.Println("Row Query: ", response.Result().Row().Columns) ``` ``` response -{ - "results":[ - { - "attrs":{}, - "columns":[1,2,3,362,368,391,396,409,416,430,436,450,454,460,461,464,466,469,470,483,484,486,490,491,503,504,514] - } - ] -} +Row Query: [1 2 3 362 368 391 396 409 416 430 436 450 454 460 461 464 466 469 470 483 484 486 490 491 503 504 514] ``` What are the top 5 languages in the sample data: ``` request -curl localhost:10101/index/repository/query \ - -X POST \ - -d 'TopN(language, n=5)' + response, err = client.Query(language.TopN(5)) + if err != nil { + log.Fatal(err) + } + fmt.Println("TopN Query: ", response.Result().CountItems()) ``` ``` response -{ - "results":[ - [ - {"id":5,"count":119}, - {"id":1,"count":50}, - {"id":4,"count":48}, - {"id":9,"count":31}, - {"id":13,"count":25} - ] - ] -} +TopN Query: [{5 119} {1 50} {4 48} {9 31} {13 25}] ``` Which repositories were starred by user 14 and 19: ``` request -curl localhost:10101/index/repository/query \ - -X POST \ - -d 'Intersect( - Row(stargazer=14), - Row(stargazer=19) - )' + response, err = client.Query(repository.Intersect(stargazer.Row(14), stargazer.Row(19))) + if err != nil { + log.Fatal(err) + } + fmt.Println("Stargazer Intersect Query: ", response.Result().Row().Columns) ``` ``` response -{ - "results":[ - { - "attrs":{}, - "columns":[2,3,362,396,416,461,464,466,470,486] - } - ] -} +Stargazer Intersect Query: [2 3 362 396 416 461 464 466 470 486] ``` Which repositories were starred by user 14 or 19: ``` request -curl localhost:10101/index/repository/query \ - -X POST \ - -d 'Union( - Row(stargazer=14), - Row(stargazer=19) - )' + response, err = client.Query(repository.Union(stargazer.Row(14), stargazer.Row(19))) + if err != nil { + log.Fatal(err) + } + fmt.Println("Union Query: ", response.Result().Row().Columns) ``` ``` response -{ - "results":[ - { - "attrs":{}, - "columns":[1,2,3,361,362,368,376,377,378,382,386,388,391,396,398,400,409,411,412,416,426,428,430,435,436,450,452,453,454,456,460,461,464,465,466,469,470,483,484,486,487,489,490,491,500,503,504,505,512,514] - } - ] -} +Union Query: [1 2 3 361 362 368 376 377 378 382 386 388 391 396 398 400 409 411 412 416 426 428 430 435 436 450 452 453 454 456 460 461 464 465 466 469 470 483 484 486 487 489 490 491 500 503 504 505 512 514] ``` Which repositories were starred by user 14 and 19 and also were written in language 1: ``` request -curl localhost:10101/index/repository/query \ - -X POST \ - -d 'Intersect( - Row(stargazer=14), - Row(stargazer=19), - Row(language=1) - )' + response, err = client.Query(repository.Intersect(stargazer.Row(14), stargazer.Row(19), language.Row(1))) + if err != nil { + log.Fatal(err) + } + fmt.Println("Stargazer and Language Intersect Query: ", response.Result().Row().Columns) ``` ``` response -{ - "results":[ - { - "attrs":{}, - "columns":[2,362,416,461] - } - ] -} +Stargazer and Language Intersect Query: [2 362 416 461] ``` Set user 99999 as a stargazer for repository 77777: ``` request -curl localhost:10101/index/repository/query \ - -X POST \ - -d 'Set(77777, stargazer=99999)' + client.Query(stargazer.Set(99999, 77777)) + response, err = client.Query(stargazer.Row(99999)) + if err != nil { + log.Fatal(err) + } + fmt.Println("Set Query: ", response.Result().Row().Columns) ``` ``` response -{"results":[true]} +Set Query: [77777] ``` Please note that while user ID 99999 may not be sequential with the other column IDs, it is still a relatively low number. Don't try to use arbitrary 64-bit integers as column or row IDs in Pilosa - this will lead to problems such as poor performance and out of memory errors. +For more information about Query Language, please see [Data Model and Queries](https://github.com/pilosa/go-pilosa/blob/master/docs/data-model-queries.md) and [Server Interaction](https://github.com/pilosa/go-pilosa/blob/master/docs/server-interaction.md) +##### Java and Python Users + +
+

Java and Python support will be uploaded shortly. +

### What's Next? From 7e9fed87e71578d03f521ce98d6779ddf797a541 Mon Sep 17 00:00:00 2001 From: Ashley Svetlik Date: Wed, 26 Jun 2019 16:04:17 -0500 Subject: [PATCH 02/17] Reformatted and added HTTP --- docs/getting-started.md | 330 ++++++++++++++++++++++++++++------------ 1 file changed, 229 insertions(+), 101 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index 2e0104591..73971895e 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -11,7 +11,7 @@ nav = [ ## Getting Started Pilosa supports an HTTP interface which uses JSON by default. -Any HTTP tool can be used to interact with the Pilosa server. The examples in this documentation will use [curl](https://curl.haxx.se/) which is available by default on many UNIX-like systems including Linux and MacOS. Windows users can download curl [here](https://curl.haxx.se/download.html). +Any HTTP tool can be used to interact with the Pilosa server. The examples in this documentation will use curl which is available by default on many UNIX-like systems including Linux and MacOS. However, the best way to interface with the Pilosa server is through one of our three client libraries. Pilosa currently supports go, java, and python.

Note that Pilosa server requires a high limit for open files. Check the documentation of your system to see how to increase it in case you hit that limit. See Open File Limits for more details.

@@ -44,31 +44,196 @@ In order to better understand Pilosa's capabilities, we will create a sample pro Although Pilosa doesn't keep the data in a tabular format, we still use the terms "columns" and "rows" when describing the data model. We put the primary objects in columns, and the properties of those objects in rows. For example, the Star Trace project will contain an index called "repository" which contains columns representing Github repositories, and rows representing properties like programming languages and tags. We can better organize the rows by grouping them into sets called Fields. So the "repository" index might have a "languages" field as well as a "tags" field. You can learn more about indexes and fields in the [Data Model](../data-model/) section of the documentation. -#### Create the Environment +Note: +If at any time you want to verify the data structure, you can request the schema as follows: -While we can create indexes and query directly in the terminal, it is more conventional to do so in a client library. Pilosa supports Go, Java, and Python, though you will have to install the library for compatibility. +``` request +curl localhost:10101/schema +``` +``` response +{"indexes":null} +``` -For Go users, open a terminal (one other than the one running pilosa) and download the library in your `GOPATH` using: +#### Using HTTP + +Note: This is not the recommended way to interact with Pilosa, but it is the fastest way to see the efficiency of Pilosa. + +##### Creating the Schema + +Before we can import data or run queries, we need to create our indexes and the fields within them. Let's create the repository index first: +``` request +curl localhost:10101/index/repository -X POST +``` +``` response +{"success":true} +``` +The index name must be 64 characters or less, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. + +Let's create the `stargazer` field which has user IDs of stargazers as its rows: +``` request +curl localhost:10101/index/repository/field/stargazer \ + -X POST \ + -d '{"options": {"type": "time", "timeQuantum": "YMD"}}' +``` +``` response +{"success":true} +``` + +Since our data contains time stamps which represent the time users starred repos, we set the field type to `time`. Time quantum is the resolution of the time we want to use, and we set it to `YMD` (year, month, day) for `stargazer`. + +Next up is the `language` field, which will contain IDs for programming languages: +``` request +curl localhost:10101/index/repository/field/language \ + -X POST +``` +``` response +{"success":true} +``` + +The `language` is a `set` field, but since the default field type is `set`, we didn't specify it in field options. + +##### Import Data From CSV Files + +Download the `stargazer.csv` and `language.csv` files here: + +``` +curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/stargazer.csv +curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/language.csv +``` + +Run the following commands to import the data into Pilosa: + +``` +pilosa import -i repository -f stargazer stargazer.csv +pilosa import -i repository -f language language.csv +``` + +Note that both the user IDs and the repository IDs were remapped to sequential integers in the data files, they don't correspond to actual Github IDs anymore. You can check out [languages.txt](https://github.com/pilosa/getting-started/blob/master/languages.txt) to see the mapping for languages. + +##### Make Some Queries + +Which repositories did user 14 star: +``` request +curl localhost:10101/index/repository/query \ + -X POST \ + -d 'Row(stargazer=14)' +``` +``` response +{ + "results":[ + { + "attrs":{}, + "columns":[1,2,3,362,368,391,396,409,416,430,436,450,454,460,461,464,466,469,470,483,484,486,490,491,503,504,514] + } + ] +} +``` + +What are the top 5 languages in the sample data: +``` request +curl localhost:10101/index/repository/query \ + -X POST \ + -d 'TopN(language, n=5)' +``` +``` response +{ + "results":[ + [ + {"id":5,"count":119}, + {"id":1,"count":50}, + {"id":4,"count":48}, + {"id":9,"count":31}, + {"id":13,"count":25} + ] + ] +} +``` + +Which repositories were starred by user 14 and 19: +``` request +curl localhost:10101/index/repository/query \ + -X POST \ + -d 'Intersect( + Row(stargazer=14), + Row(stargazer=19) + )' +``` +``` response +{ + "results":[ + { + "attrs":{}, + "columns":[2,3,362,396,416,461,464,466,470,486] + } + ] +} +``` + +Which repositories were starred by user 14 or 19: +``` request +curl localhost:10101/index/repository/query \ + -X POST \ + -d 'Union( + Row(stargazer=14), + Row(stargazer=19) + )' +``` +``` response +{ + "results":[ + { + "attrs":{}, + "columns":[1,2,3,361,362,368,376,377,378,382,386,388,391,396,398,400,409,411,412,416,426,428,430,435,436,450,452,453,454,456,460,461,464,465,466,469,470,483,484,486,487,489,490,491,500,503,504,505,512,514] + } + ] +} +``` + +Which repositories were starred by user 14 and 19 and also were written in language 1: +``` request +curl localhost:10101/index/repository/query \ + -X POST \ + -d 'Intersect( + Row(stargazer=14), + Row(stargazer=19), + Row(language=1) + )' +``` +``` response +{ + "results":[ + { + "attrs":{}, + "columns":[2,362,416,461] + } + ] +} +``` + +Set user 99999 as a stargazer for repository 77777: +``` request +curl localhost:10101/index/repository/query \ + -X POST \ + -d 'Set(77777, stargazer=99999)' +``` +``` response +{"results":[true]} +``` + +Please note that while user ID 99999 may not be sequential with the other column IDs, it is still a relatively low number. +Don't try to use arbitrary 64-bit integers as column or row IDs in Pilosa - this will lead to problems such as poor performance and out of memory errors. + +#### Using Go + +Pilosa requires Go 1.12 or higher. It is also recommended that you have a code editor downloaded. + +##### Create the Environment + +In order to communicate with Pilosa through your go code, you must have a "translator," which is go-pilosa. To install go-pilsa, open a terminal (one other than the one running pilosa) and download the library in your `GOPATH` using: ``` go get github.com/pilosa/go-pilosa ``` -For Java users, add the following dependency in your `pom.xml`: -``` - - - com.pilosa - pilosa-client - 1.3.1 - - -``` - -For Python users, open a terminal (one other than the one running pilosa) and install the library using: -``` -pip install pilosa -``` - For simplicity, we reccomend that you create a separate folder for this project. In the terminal, create a new folder as follows: ``` mkdir GettingStarted @@ -81,24 +246,13 @@ curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/stargaze curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/language.csv ``` -We will also create a file called StarTrace.go (for Go users), StarTrace.java (for Java users), or StarTrace.py (for Python users) as follows: +We will also create a file called StarTrace.go as follows: ``` touch StarTrace.go ``` -This file will be used in the following section. +This file will be used in the following sections. -#### Create the Schema - -Note: -If at any time you want to verify the data structure, you can request the schema as follows: - -``` request -curl localhost:10101/schema -``` -``` response -{"indexes":null} -``` -##### Go Users +##### Create the Schema Before we can import data or run queries, we need to create our indexes and the fields within them. Let's create the repository index first. Copy the following into the StarTrace.go file: ``` @@ -164,18 +318,10 @@ func main() { } ``` -##### Java and Python Users - -
-

Java and Python support will be uploaded shortly. -

- -#### Import Data From CSV Files +##### Import Data From CSV Files Now that we have our index and our fields, we can import the data we downloaded earlier and soon be making our own queries. -##### Go Users - First, we will load our data into the `stargazer` field: ``` stargazerFile, err := ioutil.ReadFile("stargazer.csv") @@ -205,109 +351,91 @@ Next, we will load our data into the `language` field: ``` The `language` is a `set` field, but since the default field type is `set`, we didn't need to specify it. -##### Java and Python Users - -
-

Java and Python support will be uploaded shortly. -

- -
-

If you are using a Docker container for Pilosa (with name `pilosa`), you should instead copy the `*.csv` file into the container and then import them: -``` -docker cp stargazer.csv pilosa:/stargazer.csv -docker exec -it pilosa /pilosa import -i repository -f stargazer /stargazer.csv -docker cp language.csv pilosa:/language.csv -docker exec -it pilosa /pilosa import -i repository -f language /language.csv -``` -

- Note that both the user IDs and the repository IDs were remapped to sequential integers in the data files, they don't correspond to actual Github IDs anymore. You can check out [languages.txt](https://github.com/pilosa/getting-started/blob/master/languages.txt) to see the mapping for languages. -#### Make Some Queries +##### Make Some Queries Now that we have a working schema, we can query it. -##### Go Users - Which repositories did user 14 star: ``` request - response, err := client.Query(stargazer.Row(14)) - if err != nil { - log.Fatal(err) - } - fmt.Println("Row Query: ", response.Result().Row().Columns) +response, err := client.Query(stargazer.Row(14)) +if err != nil { + log.Fatal(err) +} +fmt.Println("User 14 starred: ", response.Result().Row().Columns) ``` ``` response -Row Query: [1 2 3 362 368 391 396 409 416 430 436 450 454 460 461 464 466 469 470 483 484 486 490 491 503 504 514] +User 14 starred: [1 2 3 362 368 391 396 409 416 430 436 450 454 460 461 464 466 469 470 483 484 486 490 491 503 504 514] ``` What are the top 5 languages in the sample data: ``` request - response, err = client.Query(language.TopN(5)) - if err != nil { - log.Fatal(err) - } - fmt.Println("TopN Query: ", response.Result().CountItems()) +response, err = client.Query(language.TopN(5)) +if err != nil { + log.Fatal(err) +} +fmt.Println("Top Languages: ", response.Result().CountItems()) ``` ``` response -TopN Query: [{5 119} {1 50} {4 48} {9 31} {13 25}] +Top Languages: [{5 119} {1 50} {4 48} {9 31} {13 25}] ``` Which repositories were starred by user 14 and 19: ``` request - response, err = client.Query(repository.Intersect(stargazer.Row(14), stargazer.Row(19))) - if err != nil { - log.Fatal(err) - } - fmt.Println("Stargazer Intersect Query: ", response.Result().Row().Columns) +response, err = client.Query(repository.Intersect(stargazer.Row(14), stargazer.Row(19))) +if err != nil { + log.Fatal(err) +} +fmt.Println("Both user 14 and 19 starred: ", response.Result().Row().Columns) ``` ``` response -Stargazer Intersect Query: [2 3 362 396 416 461 464 466 470 486] +Both user 14 and 19 starred: [2 3 362 396 416 461 464 466 470 486] ``` Which repositories were starred by user 14 or 19: ``` request - response, err = client.Query(repository.Union(stargazer.Row(14), stargazer.Row(19))) - if err != nil { - log.Fatal(err) - } - fmt.Println("Union Query: ", response.Result().Row().Columns) +response, err = client.Query(repository.Union(stargazer.Row(14), stargazer.Row(19))) +if err != nil { + log.Fatal(err) +} +fmt.Println("User 14 or 19 starred: ", response.Result().Row().Columns) ``` ``` response -Union Query: [1 2 3 361 362 368 376 377 378 382 386 388 391 396 398 400 409 411 412 416 426 428 430 435 436 450 452 453 454 456 460 461 464 465 466 469 470 483 484 486 487 489 490 491 500 503 504 505 512 514] +User 14 or 19 starred: [1 2 3 361 362 368 376 377 378 382 386 388 391 396 398 400 409 411 412 416 426 428 430 435 436 450 452 453 454 456 460 461 464 465 466 469 470 483 484 486 487 489 490 491 500 503 504 505 512 514] ``` Which repositories were starred by user 14 and 19 and also were written in language 1: ``` request - response, err = client.Query(repository.Intersect(stargazer.Row(14), stargazer.Row(19), language.Row(1))) - if err != nil { - log.Fatal(err) - } - fmt.Println("Stargazer and Language Intersect Query: ", response.Result().Row().Columns) +response, err = client.Query(repository.Intersect(stargazer.Row(14), stargazer.Row(19), language.Row(1))) +if err != nil { + log.Fatal(err) +} +fmt.Println("User 14 or 19 starred, written in language 1: ", response.Result().Row().Columns) ``` ``` response -Stargazer and Language Intersect Query: [2 362 416 461] +User 14 or 19 starred, written in language 1: [2 362 416 461] ``` Set user 99999 as a stargazer for repository 77777: ``` request - client.Query(stargazer.Set(99999, 77777)) - response, err = client.Query(stargazer.Row(99999)) - if err != nil { - log.Fatal(err) - } - fmt.Println("Set Query: ", response.Result().Row().Columns) +client.Query(stargazer.Set(99999, 77777)) +response, err = client.Query(stargazer.Row(99999)) +if err != nil { + log.Fatal(err) +} +fmt.Println("Set user 99999 as a stargazer for repository 77777") ``` ``` response -Set Query: [77777] +Set user 99999 as a stargazer for repository 77777 ``` Please note that while user ID 99999 may not be sequential with the other column IDs, it is still a relatively low number. Don't try to use arbitrary 64-bit integers as column or row IDs in Pilosa - this will lead to problems such as poor performance and out of memory errors. -For more information about Query Language, please see [Data Model and Queries](https://github.com/pilosa/go-pilosa/blob/master/docs/data-model-queries.md) and [Server Interaction](https://github.com/pilosa/go-pilosa/blob/master/docs/server-interaction.md) +For more information about go-pilosa, please see our Go client library for [go-pilosa](https://github.com/pilosa/go-pilosa) -##### Java and Python Users +#### Java and Python Users

Java and Python support will be uploaded shortly. From 66867b0cb8056d6955d9b1419c6f5c2ca5a59bf4 Mon Sep 17 00:00:00 2001 From: Ashley Svetlik Date: Wed, 26 Jun 2019 16:31:24 -0500 Subject: [PATCH 03/17] Fixed typos --- docs/getting-started.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index 73971895e..afc76d371 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -40,9 +40,9 @@ curl localhost:10101/status ### Sample Project -In order to better understand Pilosa's capabilities, we will create a sample project called "Star Trace" containing information about 1,000 popular Github repositories which have "go" in their name. The Star Trace index will include data points such as programming language, tags, and stargazers—people who have starred a project. +In order to better understand Pilosa's capabilities, we will create a sample project called "Star Trace" containing information about 1,000 popular Github repositories which have "go" in their name. The Star Trace index will include data points such as programming language and stargazers—people who have starred a project. -Although Pilosa doesn't keep the data in a tabular format, we still use the terms "columns" and "rows" when describing the data model. We put the primary objects in columns, and the properties of those objects in rows. For example, the Star Trace project will contain an index called "repository" which contains columns representing Github repositories, and rows representing properties like programming languages and tags. We can better organize the rows by grouping them into sets called Fields. So the "repository" index might have a "languages" field as well as a "tags" field. You can learn more about indexes and fields in the [Data Model](../data-model/) section of the documentation. +Although Pilosa doesn't keep the data in a tabular format, we still use the terms "columns" and "rows" when describing the data model. We put the primary objects in columns, and the properties of those objects in rows. For example, the Star Trace project will contain an index called "repository" which contains columns representing Github repositories, and rows representing properties like programming languages and stargazers. We can better organize the rows by grouping them into sets called Fields. So the "repository" index might have a "languages" field as well as a "stargazers" field. You can learn more about indexes and fields in the [Data Model](../data-model/) section of the documentation. Note: If at any time you want to verify the data structure, you can request the schema as follows: @@ -58,7 +58,7 @@ curl localhost:10101/schema Note: This is not the recommended way to interact with Pilosa, but it is the fastest way to see the efficiency of Pilosa. -##### Creating the Schema +##### Create the Schema Before we can import data or run queries, we need to create our indexes and the fields within them. Let's create the repository index first: ``` request @@ -67,7 +67,7 @@ curl localhost:10101/index/repository -X POST ``` response {"success":true} ``` -The index name must be 64 characters or less, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. +The index name must be 64 characters or less, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. The same goes for field names. Let's create the `stargazer` field which has user IDs of stargazers as its rows: ``` request @@ -229,7 +229,7 @@ Pilosa requires Go 1.12 or higher. It is also recommended that you have a code e ##### Create the Environment -In order to communicate with Pilosa through your go code, you must have a "translator," which is go-pilosa. To install go-pilsa, open a terminal (one other than the one running pilosa) and download the library in your `GOPATH` using: +In order to communicate with Pilosa through your go code, you must have a "translator," which is go-pilosa. To install go-pilosa, open a terminal (one other than the one running pilosa) and download the library in your `GOPATH` using: ``` go get github.com/pilosa/go-pilosa ``` @@ -411,10 +411,10 @@ response, err = client.Query(repository.Intersect(stargazer.Row(14), stargazer.R if err != nil { log.Fatal(err) } -fmt.Println("User 14 or 19 starred, written in language 1: ", response.Result().Row().Columns) +fmt.Println("Both user 14 and 19 starred and were written in language 1: ", response.Result().Row().Columns) ``` ``` response -User 14 or 19 starred, written in language 1: [2 362 416 461] +Both user 14 and 19 starred and were written in language 1: [2 362 416 461] ``` Set user 99999 as a stargazer for repository 77777: From 93da23d902a186ec96a8fd75ed06483747d30d53 Mon Sep 17 00:00:00 2001 From: Ashley Svetlik Date: Thu, 27 Jun 2019 10:17:50 -0500 Subject: [PATCH 04/17] Added Java and Fixed Typos --- docs/getting-started.md | 238 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 230 insertions(+), 8 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index afc76d371..5e91e3bde 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -54,7 +54,7 @@ curl localhost:10101/schema {"indexes":null} ``` -#### Using HTTP +#### Using Curl Note: This is not the recommended way to interact with Pilosa, but it is the fastest way to see the efficiency of Pilosa. @@ -236,11 +236,10 @@ go get github.com/pilosa/go-pilosa For simplicity, we reccomend that you create a separate folder for this project. In the terminal, create a new folder as follows: ``` -mkdir GettingStarted -cd GettingStarted +mkdir GettingStarted && cd GettingStarted ``` -In this folder, we will download two CSV files to provide data to our fields later on. Download the stargazer.csv and language.csv files here: +In this folder, we will download two CSV files to provide data to our fields later on. Download the `stargazer.csv` and `language.csv` files here: ``` curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/stargazer.csv curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/language.csv @@ -272,7 +271,7 @@ func main() { client := pilosa.DefaultClient() schema, _ := client.Schema() repository := schema.Index("repository") - // This is where the field will go later + // This is where the fields will go later err := client.SyncSchema(schema) if err != nil { log.Fatal(err) @@ -291,7 +290,7 @@ Next up is the `language` field, which will contain IDs for programming language language := repository.Field("language") ``` -Your StarTrace.go file should look like: +Your `StarTrace.go` file should look like: ``` package main @@ -335,7 +334,7 @@ First, we will load our data into the `stargazer` field: log.Fatal(err) } ``` -Since our `stargazer` data contains time stamps, which represent the time users starred repos, we will be using the `csv.NewColumnIterator` function that is built into the go-pilosa import. For more information on imports in go-pilosa, please see the go-pilosa [site](https://github.com/pilosa/go-pilosa/blob/master/docs/imports-exports.md). Time quantum is the resolution of the time we want to use and is defined by the `format` variable. +Since our `stargazer` data contains time stamps, which represent the time users starred repos, we will be using the `csv.NewColumnIterator` function that is built into the go-pilosa import. Time quantum is the resolution of the time we want to use and is defined by the `format` variable. Next, we will load our data into the `language` field: ``` @@ -351,6 +350,8 @@ Next, we will load our data into the `language` field: ``` The `language` is a `set` field, but since the default field type is `set`, we didn't need to specify it. +For more information on imports in go-pilosa, please see the go-pilosa [site](https://github.com/pilosa/go-pilosa/blob/master/docs/imports-exports.md). + Note that both the user IDs and the repository IDs were remapped to sequential integers in the data files, they don't correspond to actual Github IDs anymore. You can check out [languages.txt](https://github.com/pilosa/getting-started/blob/master/languages.txt) to see the mapping for languages. ##### Make Some Queries @@ -435,7 +436,228 @@ Don't try to use arbitrary 64-bit integers as column or row IDs in Pilosa - this For more information about go-pilosa, please see our Go client library for [go-pilosa](https://github.com/pilosa/go-pilosa) -#### Java and Python Users +#### Using Java + +Pilosa requires Java 8 or higher and Maven 3 or higher. It is also recommended that you have a code editor downloaded. + +##### Create the Environment + +To contain the Getting Started project in one place, we will create a new folder as follows: +``` +mkdir GettingStarted && cd GettingStarted +``` + +In this folder, we will download two CSV files to provide data to our fields later on. Download the `stargazer.csv` and `language.csv` files here: +``` +curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/stargazer.csv +curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/language.csv +``` + +We will now create the java directory that will contain our `pom.xml` file and import the `pom.xml` file: +``` +mkdir startrace && cd startrace +curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/java/startrace/pom.xml +``` + +For this specific project, the `pom.xml` file needs to be edited. The file can be edited by typing `nano pom.xml` directly into the terminal or simply using your code editing software. The following needs to be changed: +``` + + + com.pilosa + pilosa-client + 1.3.1 + + + + + + org.apache.maven.plugins + maven-jar-plugin + 3.0.2 + + + + true + lib/ + main.java.StarTrace + + + + +``` + +We will now create the java directory that will contain our `StarTrace.java` file and create the `StarTrace.jave file: +``` +mkdir src && cd src +mkdir main && cd main +mkdir java && cd java +touch StarTrace.go +``` + +This file will be used in the following sections. + +##### Create the Schema + +Before we can import data or run queries, we need to create our indexes and the fields within them. Let's create the repository index first. Copy the following into the StarTrace.java file: +``` +package main.java; + +import com.pilosa.client.PilosaClient; +import com.pilosa.client.QueryResponse; +import com.pilosa.client.exceptions.PilosaException; +import com.pilosa.client.orm.*; +import com.pilosa.client.csv.FileRecordIterator; +import com.pilosa.client.TimeQuantum; + +import java.io.IOException; +import java.text.SimpleDateFormat; + +public class StarTrace { + public static void main(String []args) throws IOException { + // Create the Schema + PilosaClient client = PilosaClient.defaultClient(); + Schema schema = client.readSchema(); + Index repository = schema.index("repository"); + // This is were the fields will go later + client.syncSchema(schema); + } +} +``` +The index name must be 64 characters or less, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. The same goes for field names. + +Let's create the `stargazer` field which has user IDs of stargazers as its rows: +``` + FieldOptions stargazerOptions = FieldOptions.builder() + .fieldTime(TimeQuantum.YEAR_MONTH_DAY) + .build(); + Field stargazer = repository.field("stargazer", stargazerOptions); +``` +Since our data contains time stamps which represent the time users starred repos, we set the field type to `time` using `fieldTime()`. Time quantum is the resolution of the time we want to use, and we set it to `YEAR_MONTH-DAY` for `stargazer`. + +Next up is the `language` field, which will contain IDs for programming languages: +``` + Field language = repository.field("language"); +``` + +Your `StarTrace.java` file should look like: +``` +package main.java; + +import com.pilosa.client.PilosaClient; +import com.pilosa.client.QueryResponse; +import com.pilosa.client.exceptions.PilosaException; +import com.pilosa.client.orm.*; +import com.pilosa.client.csv.FileRecordIterator; +import com.pilosa.client.TimeQuantum; + +import java.io.IOException; +import java.text.SimpleDateFormat; + +public class StarTrace { + public static void main(String []args) throws IOException { + // Create the Schema + PilosaClient client = PilosaClient.defaultClient(); + Schema schema = client.readSchema(); + Index repository = schema.index("repository"); + + FieldOptions stargazerOptions = FieldOptions.builder() + .fieldTime(TimeQuantum.YEAR_MONTH_DAY) + .build(); + Field stargazer = repository.field("stargazer", stargazerOptions); + + Field language = repository.field("language"); + client.syncSchema(schema); + } +} +``` + +##### Import Data From CSV Files + +Now that we have our index and our fields, we can import the data we downloaded earlier and soon be making our own queries. + +First, we will load our data into the `stargazer` field: +``` + SimpleDateFormat timestampFormat = new SimpleDateFormat("yyyy-MM-dd'T'hh:mm"); + FileRecordIterator iterator = FileRecordIterator.fromPath("stargazer.csv", stargazer, timestampFormat); + client.importField(stargazer, iterator); +``` +Due to the time aspect of the `stargazer` field, we have to specify the format of the time stamps using the `SimpleDateFormat() function. + +Next, we will load our data into the `language` field: +``` + iterator = FileRecordIterator.fromPath("language.csv", language); + client.importField(language, iterator); +``` +The `language` is a `set` field, but since the default field type is `set`, we didn't need to specify it. + +For more information on imports in java-pilosa, please see the java-pilosa [site](https://github.com/pilosa/java-pilosa/blob/master/docs/imports.md). + +Note that both the user IDs and the repository IDs were remapped to sequential integers in the data files, they don't correspond to actual Github IDs anymore. You can check out [languages.txt](https://github.com/pilosa/getting-started/blob/master/languages.txt) to see the mapping for languages. + +##### Make Some Queries + +Now that we have a working schema, we can query it. + +Which repositories did user 14 star: +``` request +QueryResponse response = client.query(stargazer.row(14)); +System.out.println("User 14 starred: " + response.getResult().getRow().getColumns()); +``` +``` response +User 14 starred: [1, 2, 3, 362, 368, 391, 396, 409, 416, 430, 436, 450, 454, 460, 461, 464, 466, 469, 470, 483, 484, 486, 490, 491, 503, 504, 514] +``` + +What are the top 5 languages in the sample data: +``` request +response = client.query(language.topN(5)); +System.out.println("Top Languages: " + response.getResult().getCountItems()); +``` +``` response +Top Languages: [CountResultItem(id=5, count=119), CountResultItem(id=1, count=50), CountResultItem(id=4, count=48), CountResultItem(id=9, count=31), CountResultItem(id=13, count=25)] +``` + +Which repositories were starred by user 14 and 19: +``` request +response = client.query(repository.intersect(stargazer.row(14), stargazer.row(19))); +System.out.println("Both user 14 and 19 starred: " + response.getResult().getRow().getColumns()); +``` +``` response +Both user 14 and 19 starred: [2, 3, 362, 396, 416, 461, 464, 466, 470, 486] +``` + +Which repositories were starred by user 14 or 19: +``` request +response = client.query(repository.union(stargazer.row(14), stargazer.row(19))); +System.out.println("User 14 or 19 starred: " + response.getResult().getRow().getColumns()); +``` +``` response +User 14 or 19 starred: [1, 2, 3, 361, 362, 368, 376, 377, 378, 382, 386, 388, 391, 396, 398, 400, 409, 411, 412, 416, 426, 428, 430, 435, 436, 450, 452, 453, 454, 456, 460, 461, 464, 465, 466, 469, 470, 483, 484, 486, 487, 489, 490, 491, 500, 503, 504, 505, 512, 514] +``` + +Which repositories were starred by user 14 and 19 and also were written in language 1: +``` request +response = client.query(repository.intersect(stargazer.row(14), stargazer.row(19), language.row(1))); +System.out.println("Both user 14 and 19 starred and were written in language 1: " + response.getResult().getRow().getColumns()); +``` +``` response +Both user 14 and 19 starred and were written in language 1: [2, 362, 416, 461] +``` + +Set user 99999 as a stargazer for repository 77777: +``` request +client.query(stargazer.set(99999, 77777)); +System.out.println("Set user 99999 as a stargazer for repository 77777"); +``` +``` response +Set user 99999 as a stargazer for repository 77777 +``` + +Please note that while user ID 99999 may not be sequential with the other column IDs, it is still a relatively low number. +Don't try to use arbitrary 64-bit integers as column or row IDs in Pilosa - this will lead to problems such as poor performance and out of memory errors. + +For more information about java-pilosa, please see our Java client library for [java-pilosa](https://github.com/pilosa/java-pilosa) + +#### Python Users

Java and Python support will be uploaded shortly. From 74cfa79bb0e7cdef83e991da3d6d61f3ed7a2d55 Mon Sep 17 00:00:00 2001 From: Ashley Svetlik Date: Thu, 27 Jun 2019 16:06:18 -0500 Subject: [PATCH 05/17] Added Python --- docs/getting-started.md | 201 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 197 insertions(+), 4 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index 5e91e3bde..2b40b8e8d 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -581,7 +581,7 @@ First, we will load our data into the `stargazer` field: FileRecordIterator iterator = FileRecordIterator.fromPath("stargazer.csv", stargazer, timestampFormat); client.importField(stargazer, iterator); ``` -Due to the time aspect of the `stargazer` field, we have to specify the format of the time stamps using the `SimpleDateFormat() function. +Due to the time aspect of the `stargazer` field, we have to specify the format of the time stamps using the `SimpleDateFormat()` function. Next, we will load our data into the `language` field: ``` @@ -659,9 +659,202 @@ For more information about java-pilosa, please see our Java client library for [ #### Python Users -

-

Java and Python support will be uploaded shortly. -

+Pilosa requires Python 2.7 or higher or Python 3.4 or higher. It is also recommended that you have a code editor downloaded. + +##### Create the Environment + +To contain the Getting Started project in one place, we will create a new folder as follows: +``` +mkdir GettingStarted && cd GettingStarted +``` +In this folder, we will download two CSV files to provide data to our fields later on. Download the `stargazer.csv` and `language.csv` files here: +``` +curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/stargazer.csv +curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/language.csv +``` +We will also download two `.txt` files. One is the `requirements.txt` that will install python-pilosa and the other is `languages.txt` which will provide context to the `language` field. +``` +curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/python/requirements.txt +curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/language.txt +``` +We will now create the python environment: +``` +python3 -m venv startrace +``` + +Next, we activate the python environment we created and install the requirements: +``` +source startrace/bin/activate +pip install -r requirements.txt +``` +We will also create a file called StarTrace.py as follows: +``` +touch StarTrace.py +``` +This file will be used in the following sections. + +##### Create the Schema + +Before we can import data or run queries, we need to create our indexes and the fields within them. Let's create the repository index first. Copy the following into the StarTrace.py file: +``` +from __future__ import print_function + +import os +import sys +import time +import pilosa + +from pilosa import Client, Index, TimeQuantum +from pilosa.imports import csv_column_reader, csv_row_id_column_id + +from io import StringIO + +# Create the Schema +client = pilosa.Client() +schema = client.schema() +repository = schema.index("repository") +# This is where the fields will go later +client.sync_schema(schema) +``` +The index name must be 64 characters or less, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. The same goes for field names. + +Let's create the `stargazer` field which has user IDs of stargazers as its rows: +``` +stargazer = repository.field("stargazer", time_quantum=pilosa.TimeQuantum.YEAR_MONTH_DAY) +``` +Since our data contains time stamps which represent the time users starred repos, we set the field type to `time` using `time_quantum`. Time quantum is the resolution of the time we want to use, and we set it to `YEAR_MONTH-DAY` for `stargazer`. + +Next up is the `language` field, which will contain IDs for programming languages: +``` +language = repository.field("language") +``` +Your `StarTrace.py` file should look like: +``` +from __future__ import print_function + +import os +import sys +import time +import pilosa + +from pilosa import Client, Index, TimeQuantum +from pilosa.imports import csv_column_reader, csv_row_id_column_id + +from io import StringIO + +# Create the Schema +client = pilosa.Client() +schema = client.schema() +repository = schema.index("repository") +stargazer = repository.field("stargazer", time_quantum=pilosa.TimeQuantum.YEAR_MONTH_DAY) +language = repository.field("language") +client.sync_schema(schema) +``` + +##### Import Data From CSV Files + +Now that we have our index and our fields, we can import the data we downloaded earlier and soon be making our own queries. + +First, we will load our data into the `stargazer` field: +``` +time_func = lambda s: int(time.mktime(time.strptime(s, "%Y-%m-%dT%H:%M"))) +with open("stargazer.csv") as f: + stargazer_reader = csv_column_reader(f, timefunc=time_func) + client.import_field(stargazer, stargazer_reader) +``` +Due to the time aspect of the `stargazer` field, we have to specify the format of the time stamps using the `time_func` variable. + +Next, we will load our data into the `language` field: +``` +with open("language.csv") as f: + language_reader = csv_column_reader(f, csv_row_id_column_id) + client.import_field(language, language_reader) +``` + +The `language` is a `set` field, but since the default field type is `set`, we didn't need to specify it. + +For more information on imports in python-pilosa, please see the python-pilosa [site](https://github.com/pilosa/python-pilosa/blob/master/docs/imports.md). + +Note that both the user IDs and the repository IDs were remapped to sequential integers in the data files, they don't correspond to actual Github IDs anymore. You can check out [languages.txt](https://github.com/pilosa/getting-started/blob/master/languages.txt) to see the mapping for languages. + +##### Make Some Queries + +Now that we have a working schema, we can query it. + +Which repositories did user 14 star: +``` request +response = client.query(stargazer.row(14)) +print("User 14 starred: ", response.result.row.columns) +``` +``` response +User 14 starred: [1, 2, 3, 362, 368, 391, 396, 409, 416, 430, 436, 450, 454, 460, 461, 464, 466, 469, 470, 483, 484, 486, 490, 491, 503, 504, 514] +``` + +What are the top 5 languages in the sample data: +``` request +def load_language_names(): + with open("languages.txt") as f: + return [line.strip() for line in f] + +def print_topn(items): + lines = ["\t{i}. {s[0]}: {s[1]} stars".format(s=s, i=i + 1) for i, s in enumerate(items)] + print("\n".join(lines)) + +language_names = load_language_names() +top_languages = client.query(language.topn(5)).result.count_items +language_items = [(language_names[item.id], item.count) for item in top_languages] +print("Top languages: ") +print_topn(language_items) +``` +``` response +Top languages: + 1. Go: 119 stars + 2. Shell: 50 stars + 3. Makefile: 48 stars + 4. HTML: 31 stars + 5. JavaScript: 25 stars +``` + +Which repositories were starred by user 14 and 19: +``` request +repsonse = client.query(repository.intersect(stargazer.row(14), stargazer.row(19))) +print("Both user 14 and 19 starred: ", response.result.row.columns) +``` +``` resposne +Both user 14 and 19 starred: [1, 2, 3, 362, 368, 391, 396, 409, 416, 430, 436, 450, 454, 460, 461, 464, 466, 469, 470, 483, 484, 486, 490, 491, 503, 504, 514] +``` + +Which repositories were starred by user 14 or 19: +``` request +response = client.query(repository.union(stargazer.row(14), stargazer.row(19))) +print("User 14 or 19 starred: ", response.result.row.columns) +``` +``` response +User 14 or 19 starred: [1, 2, 3, 361, 362, 368, 376, 377, 378, 382, 386, 388, 391, 396, 398, 400, 409, 411, 412, 416, 426, 428, 430, 435, 436, 450, 452, 453, 454, 456, 460, 461, 464, 465, 466, 469, 470, 483, 484, 486, 487, 489, 490, 491, 500, 503, 504, 505, 512, 514] +``` + +Which repositories were starred by user 14 and 19 and also were written in language 1: +``` request +response = client.query(repository.intersect(stargazer.row(14), stargazer.row(19), language.row(1))) +print("Both user 14 and 19 starred and were written in language 1: ", response.result.row.columns) +``` +``` response +Both user 14 and 19 starred and were written in language 1: [2, 362, 416, 461] +``` + +Set user 99999 as a stargazer for repository 77777: +``` request +client.query(stargazer.set(99999, 77777)) +print("Set user 99999 as a stargazer for repository 77777") +``` +``` response +Set user 99999 as a stargazer for repository 77777 +``` + +Please note that while user ID 99999 may not be sequential with the other column IDs, it is still a relatively low number. +Don't try to use arbitrary 64-bit integers as column or row IDs in Pilosa - this will lead to problems such as poor performance and out of memory errors. + +For more information about python-pilosa, please see our Python client library for [python-pilosa](https://github.com/pilosa/python-pilosa). ### What's Next? From 680b0119a95a5d1c33e9493272d82ec1dd4b176a Mon Sep 17 00:00:00 2001 From: Ashley Svetlik Date: Fri, 28 Jun 2019 11:44:52 -0500 Subject: [PATCH 06/17] Added explanation and fixed typos --- docs/getting-started.md | 125 ++++++++++++++++++++++++++-------------- 1 file changed, 83 insertions(+), 42 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index 2b40b8e8d..9075f5b8f 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -11,7 +11,7 @@ nav = [ ## Getting Started Pilosa supports an HTTP interface which uses JSON by default. -Any HTTP tool can be used to interact with the Pilosa server. The examples in this documentation will use curl which is available by default on many UNIX-like systems including Linux and MacOS. However, the best way to interface with the Pilosa server is through one of our three client libraries. Pilosa currently supports go, java, and python. +Any HTTP tool can be used to interact with the Pilosa server. The examples in this documentation will use curl which is available by default on many UNIX-like systems including Linux and MacOS. However, the best way to interface with the Pilosa server is through one of our three client libraries. Pilosa currently supports [Go](https://github.com/pilosa/go-pilosa), [Java](https://github.com/pilosa/java-pilosa), and [Python](https://github.com/pilosa/python-pilosa).

Note that Pilosa server requires a high limit for open files. Check the documentation of your system to see how to increase it in case you hit that limit. See Open File Limits for more details.

@@ -24,10 +24,6 @@ Execute the following in a terminal to run Pilosa with the default configuration ``` pilosa server ``` -If you are using the Docker image, you can run an ephemeral Pilosa container on the default address using the following command: -``` -docker run -it --rm --name pilosa -p 10101:10101 pilosa/pilosa:latest -``` Let's make sure Pilosa is running: ``` request @@ -44,8 +40,9 @@ In order to better understand Pilosa's capabilities, we will create a sample pro Although Pilosa doesn't keep the data in a tabular format, we still use the terms "columns" and "rows" when describing the data model. We put the primary objects in columns, and the properties of those objects in rows. For example, the Star Trace project will contain an index called "repository" which contains columns representing Github repositories, and rows representing properties like programming languages and stargazers. We can better organize the rows by grouping them into sets called Fields. So the "repository" index might have a "languages" field as well as a "stargazers" field. You can learn more about indexes and fields in the [Data Model](../data-model/) section of the documentation. -Note: -If at any time you want to verify the data structure, you can request the schema as follows: +Pilosa supports curl (or any HTTP tool), Go, Java, and Python. In this project, we will walk you through how to use each one to best communicate with the Pilosa server. + +Note: If at any time you want to verify the data structure, you can request the schema as follows: ``` request curl localhost:10101/schema @@ -60,7 +57,7 @@ Note: This is not the recommended way to interact with Pilosa, but it is the fas ##### Create the Schema -Before we can import data or run queries, we need to create our indexes and the fields within them. Let's create the repository index first: +Before we can import data or run queries, we need to create our indexes and the fields within them. Let's create the `repository` index first: ``` request curl localhost:10101/index/repository -X POST ``` @@ -229,12 +226,12 @@ Pilosa requires Go 1.12 or higher. It is also recommended that you have a code e ##### Create the Environment -In order to communicate with Pilosa through your go code, you must have a "translator," which is go-pilosa. To install go-pilosa, open a terminal (one other than the one running pilosa) and download the library in your `GOPATH` using: +In order to communicate with Pilosa through your Go code, you must have a "translator," which is go-pilosa. To install go-pilosa, open a terminal (one other than the one running Pilosa) and download the library in your `GOPATH` using: ``` go get github.com/pilosa/go-pilosa ``` -For simplicity, we reccomend that you create a separate folder for this project. In the terminal, create a new folder as follows: +To contain the Getting Started project in one place, we will create a new folder as follows: ``` mkdir GettingStarted && cd GettingStarted ``` @@ -245,7 +242,7 @@ curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/stargaze curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/language.csv ``` -We will also create a file called StarTrace.go as follows: +We will also create a file called `StarTrace.go` as follows: ``` touch StarTrace.go ``` @@ -253,7 +250,7 @@ This file will be used in the following sections. ##### Create the Schema -Before we can import data or run queries, we need to create our indexes and the fields within them. Let's create the repository index first. Copy the following into the StarTrace.go file: +Before we can import data or run queries, we need to create our schema. Go-pilosa is implemented by importing `github.com/pilosa/go-pilosa` and its ability to read csv files is implemented by importing 'github.com/pilosa/go-pilosa/csv`. The first steps to creating the schema are creating a client which will communicate our schema to Pilosa, creating a schema which will contain our indexes and fields, and syncing with Pilosa. This is all done in the `StarTrace.go` file: ``` package main @@ -270,7 +267,7 @@ func main() { // Create the Schema client := pilosa.DefaultClient() schema, _ := client.Schema() - repository := schema.Index("repository") + // This is where the index will go later // This is where the fields will go later err := client.SyncSchema(schema) if err != nil { @@ -278,6 +275,12 @@ func main() { } } ``` + +Next, let's create the `repository` index: +``` + repository := schema.Index("repository") +``` + The index name must be 64 characters or less, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. The same goes for field names. Let's create the `stargazer` field which has user IDs of stargazers as its rows: @@ -319,7 +322,7 @@ func main() { ##### Import Data From CSV Files -Now that we have our index and our fields, we can import the data we downloaded earlier and soon be making our own queries. +Now that we have our index and our fields, we can import the data we downloaded earlier and be on our way to making our own queries. First, we will load our data into the `stargazer` field: ``` @@ -334,7 +337,7 @@ First, we will load our data into the `stargazer` field: log.Fatal(err) } ``` -Since our `stargazer` data contains time stamps, which represent the time users starred repos, we will be using the `csv.NewColumnIterator` function that is built into the go-pilosa import. Time quantum is the resolution of the time we want to use and is defined by the `format` variable. +Since our `stargazer` data contains time stamps, which represent the time users starred repos, we will be using the `csv.NewColumnIteratorWithTimeStampFormat` function that is built into the go-pilosa import. This function takes the format of the csv files (`csv.RowIDColumnID`), an `io.Reader` (`bytes.NewReader(stargazerFile)`), and the time quantum format (`format`) and translates the csv file into a format Pilosa can read. Time quantum is the resolution of the time we want to use. Next, we will load our data into the `language` field: ``` @@ -348,12 +351,12 @@ Next, we will load our data into the `language` field: log.Fatal(err) } ``` -The `language` is a `set` field, but since the default field type is `set`, we didn't need to specify it. - -For more information on imports in go-pilosa, please see the go-pilosa [site](https://github.com/pilosa/go-pilosa/blob/master/docs/imports-exports.md). +Since our `language` data doesn't contain time stamps, we will use the `csv.NewColumnIterator` function in place of `csv.NewColumnIteratorWithTimeStampFormat`. Note that both the user IDs and the repository IDs were remapped to sequential integers in the data files, they don't correspond to actual Github IDs anymore. You can check out [languages.txt](https://github.com/pilosa/getting-started/blob/master/languages.txt) to see the mapping for languages. +For more information on imports in go-pilosa, please see the go-pilosa [site](https://github.com/pilosa/go-pilosa/blob/master/docs/imports-exports.md). + ##### Make Some Queries Now that we have a working schema, we can query it. @@ -434,7 +437,7 @@ Set user 99999 as a stargazer for repository 77777 Please note that while user ID 99999 may not be sequential with the other column IDs, it is still a relatively low number. Don't try to use arbitrary 64-bit integers as column or row IDs in Pilosa - this will lead to problems such as poor performance and out of memory errors. -For more information about go-pilosa, please see our Go client library for [go-pilosa](https://github.com/pilosa/go-pilosa) +For more information about go-pilosa, please see our Go client library at [go-pilosa](https://github.com/pilosa/go-pilosa) or checkout the go-pilosa [Data Model and Queries](https://github.com/pilosa/go-pilosa/blob/master/docs/data-model-queries.md) section for more query options. #### Using Java @@ -453,7 +456,7 @@ curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/stargaze curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/language.csv ``` -We will now create the java directory that will contain our `pom.xml` file and import the `pom.xml` file: +We will now create the java directory that will contain our `pom.xml` file and then import the `pom.xml` file: ``` mkdir startrace && cd startrace curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/java/startrace/pom.xml @@ -465,7 +468,7 @@ For this specific project, the `pom.xml` file needs to be edited. The file can b com.pilosa pilosa-client - 1.3.1 + **1.3.1** @@ -479,14 +482,14 @@ For this specific project, the `pom.xml` file needs to be edited. The file can b true lib/ - main.java.StarTrace + **main.java.StarTrace** ``` -We will now create the java directory that will contain our `StarTrace.java` file and create the `StarTrace.jave file: +We will now create the java directory that will contain our `StarTrace.java` file and create the `StarTrace.java` file: ``` mkdir src && cd src mkdir main && cd main @@ -498,7 +501,16 @@ This file will be used in the following sections. ##### Create the Schema -Before we can import data or run queries, we need to create our indexes and the fields within them. Let's create the repository index first. Copy the following into the StarTrace.java file: +Before we can import data or run queries, we need to create our schema. The following imports implement the java-pilosa: +``` +import com.pilosa.client.PilosaClient; +import com.pilosa.client.QueryResponse; +import com.pilosa.client.exceptions.PilosaException; +import com.pilosa.client.orm.*; +import com.pilosa.client.csv.FileRecordIterator; +import com.pilosa.client.TimeQuantum; +``` +The first steps to creating the schema are creating a client which will communicate our schema to Pilosa, creating a schema which will contain our indexes and fields, and syncing with Pilosa. This is all done in the `StarTrace.java` file: ``` package main.java; @@ -517,12 +529,17 @@ public class StarTrace { // Create the Schema PilosaClient client = PilosaClient.defaultClient(); Schema schema = client.readSchema(); - Index repository = schema.index("repository"); + // This is were the index will go later // This is were the fields will go later client.syncSchema(schema); } } ``` + +Next, let's create the `repository` index: +``` + Index repository = schema.index("repository"); +``` The index name must be 64 characters or less, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. The same goes for field names. Let's create the `stargazer` field which has user IDs of stargazers as its rows: @@ -538,6 +555,7 @@ Next up is the `language` field, which will contain IDs for programming language ``` Field language = repository.field("language"); ``` +The `language` field is a `set` field, but since the default field type is `set`, we don't need to specify it Your `StarTrace.java` file should look like: ``` @@ -573,7 +591,7 @@ public class StarTrace { ##### Import Data From CSV Files -Now that we have our index and our fields, we can import the data we downloaded earlier and soon be making our own queries. +Now that we have our index and our fields, we can import the data we downloaded earlier and be on our way to making our own queries. First, we will load our data into the `stargazer` field: ``` @@ -581,19 +599,19 @@ First, we will load our data into the `stargazer` field: FileRecordIterator iterator = FileRecordIterator.fromPath("stargazer.csv", stargazer, timestampFormat); client.importField(stargazer, iterator); ``` -Due to the time aspect of the `stargazer` field, we have to specify the format of the time stamps using the `SimpleDateFormat()` function. +Due to the time aspect of the `stargazer` csv file, we have to specify the time stamp format in the `fromPath` function. We set the variable `timestampFormat` to the format present in the csv file using the function `SimpleDateFormat()` and call the variable in the `fromPath` function, which will take the csv file name, the field name, and the time stamp format and translate the csv file into a format Pilosa can read. Next, we will load our data into the `language` field: ``` iterator = FileRecordIterator.fromPath("language.csv", language); client.importField(language, iterator); ``` -The `language` is a `set` field, but since the default field type is `set`, we didn't need to specify it. - -For more information on imports in java-pilosa, please see the java-pilosa [site](https://github.com/pilosa/java-pilosa/blob/master/docs/imports.md). +Since our `language` data doesn't have a time aspect, the time stamp format doesn't need to be specified. Note that both the user IDs and the repository IDs were remapped to sequential integers in the data files, they don't correspond to actual Github IDs anymore. You can check out [languages.txt](https://github.com/pilosa/getting-started/blob/master/languages.txt) to see the mapping for languages. +For more information on imports in java-pilosa, please see the java-pilosa [site](https://github.com/pilosa/java-pilosa/blob/master/docs/imports.md). + ##### Make Some Queries Now that we have a working schema, we can query it. @@ -655,7 +673,7 @@ Set user 99999 as a stargazer for repository 77777 Please note that while user ID 99999 may not be sequential with the other column IDs, it is still a relatively low number. Don't try to use arbitrary 64-bit integers as column or row IDs in Pilosa - this will lead to problems such as poor performance and out of memory errors. -For more information about java-pilosa, please see our Java client library for [java-pilosa](https://github.com/pilosa/java-pilosa) +For more information about java-pilosa, please see our Java client library at [java-pilosa](https://github.com/pilosa/java-pilosa) or checkout the java-pilosa [Data Model and Queries](https://github.com/pilosa/java-pilosa/blob/master/docs/data-model-queries.md) section for more query options. #### Python Users @@ -672,7 +690,7 @@ In this folder, we will download two CSV files to provide data to our fields lat curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/stargazer.csv curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/language.csv ``` -We will also download two `.txt` files. One is the `requirements.txt` that will install python-pilosa and the other is `languages.txt` which will provide context to the `language` field. +We will also download two text files. One is the `requirements.txt` that will install python-pilosa later on and the other is `languages.txt` which will provide context to the `language` field. ``` curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/python/requirements.txt curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/language.txt @@ -682,12 +700,12 @@ We will now create the python environment: python3 -m venv startrace ``` -Next, we activate the python environment we created and install the requirements: +Next, we activate the python environment we created and install the requirements (and python-pilosa): ``` source startrace/bin/activate pip install -r requirements.txt ``` -We will also create a file called StarTrace.py as follows: +We will also create a file called `StarTrace.py` as follows: ``` touch StarTrace.py ``` @@ -695,7 +713,14 @@ This file will be used in the following sections. ##### Create the Schema -Before we can import data or run queries, we need to create our indexes and the fields within them. Let's create the repository index first. Copy the following into the StarTrace.py file: +Before we can import data or run queries, we need to create our schema. The following imports implement the python-pilosa. This is all done in the `StarTrace.py` file: +``` +import pilosa +from pilosa import Client, Index, TimeQuantum +from pilosa.imports import csv_column_reader, csv_row_id_column_id +``` + +The first steps to creating the schema are creating a client which will communicate our schema to Pilosa, creating a schema which will contain our indexes and fields, and syncing with Pilosa: ``` from __future__ import print_function @@ -707,27 +732,38 @@ import pilosa from pilosa import Client, Index, TimeQuantum from pilosa.imports import csv_column_reader, csv_row_id_column_id -from io import StringIO +try: + # Python 2.7 and 3 + from io import StringIO +except ImportError: + # Python 2.6 and 2.7 + from StringIO import StringIO # Create the Schema client = pilosa.Client() schema = client.schema() -repository = schema.index("repository") +# This is where the index will go later # This is where the fields will go later client.sync_schema(schema) ``` +Next, let's create the `repository` index: +``` +repository = schema.index("repository") +``` The index name must be 64 characters or less, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. The same goes for field names. Let's create the `stargazer` field which has user IDs of stargazers as its rows: ``` stargazer = repository.field("stargazer", time_quantum=pilosa.TimeQuantum.YEAR_MONTH_DAY) ``` -Since our data contains time stamps which represent the time users starred repos, we set the field type to `time` using `time_quantum`. Time quantum is the resolution of the time we want to use, and we set it to `YEAR_MONTH-DAY` for `stargazer`. +Since our data contains time stamps which represent the time users starred repos, we establish the time aspect by using `time_quantum`. Time quantum is the resolution of the time we want to use, and we set it to `YEAR_MONTH-DAY` for `stargazer`. Next up is the `language` field, which will contain IDs for programming languages: ``` language = repository.field("language") ``` +The `language` field is a `set` field, but since the defualt field is `set`, we didn't need to specify any options. + Your `StarTrace.py` file should look like: ``` from __future__ import print_function @@ -740,7 +776,12 @@ import pilosa from pilosa import Client, Index, TimeQuantum from pilosa.imports import csv_column_reader, csv_row_id_column_id -from io import StringIO +try: + # Python 2.7 and 3 + from io import StringIO +except ImportError: + # Python 2.6 and 2.7 + from StringIO import StringIO # Create the Schema client = pilosa.Client() @@ -753,7 +794,7 @@ client.sync_schema(schema) ##### Import Data From CSV Files -Now that we have our index and our fields, we can import the data we downloaded earlier and soon be making our own queries. +Now that we have our index and our fields, we can import the data we downloaded earlier and be on our way to making our own queries. First, we will load our data into the `stargazer` field: ``` @@ -762,7 +803,7 @@ with open("stargazer.csv") as f: stargazer_reader = csv_column_reader(f, timefunc=time_func) client.import_field(stargazer, stargazer_reader) ``` -Due to the time aspect of the `stargazer` field, we have to specify the format of the time stamps using the `time_func` variable. +Due to the time aspect of the `stargazer` csv file, we have to specify the time stamp format in the `csv_column_reader` function. We set the variable `time_func` to the format present in the csv file and call it in the `csv_column_reader` function, which will take the csv file and the time stamp format and translate the csv file into a format Pilosa can read Next, we will load our data into the `language` field: ``` @@ -854,7 +895,7 @@ Set user 99999 as a stargazer for repository 77777 Please note that while user ID 99999 may not be sequential with the other column IDs, it is still a relatively low number. Don't try to use arbitrary 64-bit integers as column or row IDs in Pilosa - this will lead to problems such as poor performance and out of memory errors. -For more information about python-pilosa, please see our Python client library for [python-pilosa](https://github.com/pilosa/python-pilosa). +For more information about python-pilosa, please see our Python client library at [python-pilosa](https://github.com/pilosa/python-pilosa) or checkout the python-pilosa [Data Model and Queries](https://github.com/pilosa/python-pilosa/blob/master/docs/data-model-queries.md) section for more query options. ### What's Next? From 9d7273e0616334606c157d52be8102e487c767f6 Mon Sep 17 00:00:00 2001 From: Ashley Svetlik Date: Fri, 28 Jun 2019 12:39:14 -0500 Subject: [PATCH 07/17] Added the Sample Project subsections to left nav area --- docs/getting-started.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/getting-started.md b/docs/getting-started.md index 9075f5b8f..4a6d9c3de 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -4,6 +4,10 @@ weight = 3 nav = [ "Starting Pilosa", "Sample Project", + "Using Curl", + "Using Go", + "Using Java", + "Using Python", "What's Next?", ] +++ From c8e68c845677791eb50a3acc8082ca7144aed55b Mon Sep 17 00:00:00 2001 From: Ashley Svetlik Date: Mon, 1 Jul 2019 08:43:22 -0500 Subject: [PATCH 08/17] Revised to include review comments --- docs/getting-started.md | 57 +++++++++++++++++++++-------------------- 1 file changed, 29 insertions(+), 28 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index 4a6d9c3de..48f8fc7c8 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -15,7 +15,7 @@ nav = [ ## Getting Started Pilosa supports an HTTP interface which uses JSON by default. -Any HTTP tool can be used to interact with the Pilosa server. The examples in this documentation will use curl which is available by default on many UNIX-like systems including Linux and MacOS. However, the best way to interface with the Pilosa server is through one of our three client libraries. Pilosa currently supports [Go](https://github.com/pilosa/go-pilosa), [Java](https://github.com/pilosa/java-pilosa), and [Python](https://github.com/pilosa/python-pilosa). +Any HTTP tool can be used to interact with the Pilosa server. The examples in this documentation will use curl which is available by default on many UNIX-like systems including Linux and MacOS. However, the best way to interface with the Pilosa server is through one of our three official client libraries. Pilosa currently supports [Go](https://github.com/pilosa/go-pilosa), [Java](https://github.com/pilosa/java-pilosa), and [Python](https://github.com/pilosa/python-pilosa).

Note that Pilosa server requires a high limit for open files. Check the documentation of your system to see how to increase it in case you hit that limit. See Open File Limits for more details.

@@ -44,7 +44,7 @@ In order to better understand Pilosa's capabilities, we will create a sample pro Although Pilosa doesn't keep the data in a tabular format, we still use the terms "columns" and "rows" when describing the data model. We put the primary objects in columns, and the properties of those objects in rows. For example, the Star Trace project will contain an index called "repository" which contains columns representing Github repositories, and rows representing properties like programming languages and stargazers. We can better organize the rows by grouping them into sets called Fields. So the "repository" index might have a "languages" field as well as a "stargazers" field. You can learn more about indexes and fields in the [Data Model](../data-model/) section of the documentation. -Pilosa supports curl (or any HTTP tool), Go, Java, and Python. In this project, we will walk you through how to use each one to best communicate with the Pilosa server. +Pilosa as an organization supports curl (or any HTTP tool), Go, Java, and Python. However, Pilosa as a server will support any client that can send requests to it. In this project, we will walk you through how to use each one to best communicate with the Pilosa server. Note: If at any time you want to verify the data structure, you can request the schema as follows: @@ -52,8 +52,9 @@ Note: If at any time you want to verify the data structure, you can request the curl localhost:10101/schema ``` ``` response -{"indexes":null} +{"indexes":[{"name":"repository","options":{"keys":false,"trackExistence":true},"fields":[{"name":"language","options":{"type":"set","cacheType":"ranked","cacheSize":50000,"keys":false}},{"name":"stargazer","options":{"type":"time","timeQuantum":"YMDH","keys":false,"noStandardView":false}}],"shardWidth":1048576}} ``` +Note: This is the response you should recieve once completing this project. #### Using Curl @@ -226,18 +227,18 @@ Don't try to use arbitrary 64-bit integers as column or row IDs in Pilosa - this #### Using Go -Pilosa requires Go 1.12 or higher. It is also recommended that you have a code editor downloaded. +Pilosa requires Go 1.12 or higher. ##### Create the Environment -In order to communicate with Pilosa through your Go code, you must have a "translator," which is go-pilosa. To install go-pilosa, open a terminal (one other than the one running Pilosa) and download the library in your `GOPATH` using: +In order to communicate with Pilosa through your Go code, you must have a client, which is go-pilosa. To install go-pilosa, open a terminal (one other than the one running Pilosa) and download the library to your `GOPATH` using: ``` go get github.com/pilosa/go-pilosa ``` To contain the Getting Started project in one place, we will create a new folder as follows: ``` -mkdir GettingStarted && cd GettingStarted +mkdir getting_started && cd getting_started ``` In this folder, we will download two CSV files to provide data to our fields later on. Download the `stargazer.csv` and `language.csv` files here: @@ -246,15 +247,15 @@ curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/stargaze curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/language.csv ``` -We will also create a file called `StarTrace.go` as follows: +We will also create a file called `startrace.go` as follows: ``` -touch StarTrace.go +touch startrace.go ``` This file will be used in the following sections. ##### Create the Schema -Before we can import data or run queries, we need to create our schema. Go-pilosa is implemented by importing `github.com/pilosa/go-pilosa` and its ability to read csv files is implemented by importing 'github.com/pilosa/go-pilosa/csv`. The first steps to creating the schema are creating a client which will communicate our schema to Pilosa, creating a schema which will contain our indexes and fields, and syncing with Pilosa. This is all done in the `StarTrace.go` file: +Before we can import data or run queries, we need to create our schema. You can see two imports from the go-pilosa repo, go-pilosa for the client, and csv for the CSV reader. Create the schema by creating a client which will communicate our schema to Pilosa, creating a schema which will contain our indexes and fields, and syncing with Pilosa. This is all done in the `startrace.go` file: ``` package main @@ -297,7 +298,7 @@ Next up is the `language` field, which will contain IDs for programming language language := repository.Field("language") ``` -Your `StarTrace.go` file should look like: +Your `startrace.go` file should look like: ``` package main @@ -341,7 +342,7 @@ First, we will load our data into the `stargazer` field: log.Fatal(err) } ``` -Since our `stargazer` data contains time stamps, which represent the time users starred repos, we will be using the `csv.NewColumnIteratorWithTimeStampFormat` function that is built into the go-pilosa import. This function takes the format of the csv files (`csv.RowIDColumnID`), an `io.Reader` (`bytes.NewReader(stargazerFile)`), and the time quantum format (`format`) and translates the csv file into a format Pilosa can read. Time quantum is the resolution of the time we want to use. +Since our `stargazer` data contains time stamps, which represent the time users starred repos, we will be using the `csv.NewColumnIteratorWithTimeStampFormat` function from the go-pilosa/csv package. This function takes the format of the csv files (`csv.RowIDColumnID`), an `io.Reader` (`bytes.NewReader(stargazerFile)`), and the time quantum format (`format`) and translates the csv file into a format Pilosa can read. Time quantum is the resolution of the time we want to use. Next, we will load our data into the `language` field: ``` @@ -445,13 +446,13 @@ For more information about go-pilosa, please see our Go client library at [go-pi #### Using Java -Pilosa requires Java 8 or higher and Maven 3 or higher. It is also recommended that you have a code editor downloaded. +Pilosa requires Java 8 or higher and Maven 3 or higher. ##### Create the Environment To contain the Getting Started project in one place, we will create a new folder as follows: ``` -mkdir GettingStarted && cd GettingStarted +mkdir getting_started && cd getting_started ``` In this folder, we will download two CSV files to provide data to our fields later on. Download the `stargazer.csv` and `language.csv` files here: @@ -486,26 +487,26 @@ For this specific project, the `pom.xml` file needs to be edited. The file can b true lib/ - **main.java.StarTrace** + **main.java.startrace** ``` -We will now create the java directory that will contain our `StarTrace.java` file and create the `StarTrace.java` file: +We will now create the java directory that will contain our `startrace.java` file and create the `startrace.java` file: ``` mkdir src && cd src mkdir main && cd main mkdir java && cd java -touch StarTrace.go +touch startrace.go ``` This file will be used in the following sections. ##### Create the Schema -Before we can import data or run queries, we need to create our schema. The following imports implement the java-pilosa: +Before we can import data or run queries, we need to create our schema. The following imports can be seen from the java-pilosa repo: ``` import com.pilosa.client.PilosaClient; import com.pilosa.client.QueryResponse; @@ -514,7 +515,7 @@ import com.pilosa.client.orm.*; import com.pilosa.client.csv.FileRecordIterator; import com.pilosa.client.TimeQuantum; ``` -The first steps to creating the schema are creating a client which will communicate our schema to Pilosa, creating a schema which will contain our indexes and fields, and syncing with Pilosa. This is all done in the `StarTrace.java` file: +Create the schema by creating a client which will communicate our schema to Pilosa, creating a schema which will contain our indexes and fields, and syncing with Pilosa. This is all done in the `startrace.java` file: ``` package main.java; @@ -528,7 +529,7 @@ import com.pilosa.client.TimeQuantum; import java.io.IOException; import java.text.SimpleDateFormat; -public class StarTrace { +public class startrace { public static void main(String []args) throws IOException { // Create the Schema PilosaClient client = PilosaClient.defaultClient(); @@ -561,7 +562,7 @@ Next up is the `language` field, which will contain IDs for programming language ``` The `language` field is a `set` field, but since the default field type is `set`, we don't need to specify it -Your `StarTrace.java` file should look like: +Your `startrace.java` file should look like: ``` package main.java; @@ -575,7 +576,7 @@ import com.pilosa.client.TimeQuantum; import java.io.IOException; import java.text.SimpleDateFormat; -public class StarTrace { +public class startrace { public static void main(String []args) throws IOException { // Create the Schema PilosaClient client = PilosaClient.defaultClient(); @@ -681,13 +682,13 @@ For more information about java-pilosa, please see our Java client library at [j #### Python Users -Pilosa requires Python 2.7 or higher or Python 3.4 or higher. It is also recommended that you have a code editor downloaded. +Pilosa requires Python 2.7 or higher or Python 3.4 or higher. ##### Create the Environment To contain the Getting Started project in one place, we will create a new folder as follows: ``` -mkdir GettingStarted && cd GettingStarted +mkdir getting_started && cd getting_started ``` In this folder, we will download two CSV files to provide data to our fields later on. Download the `stargazer.csv` and `language.csv` files here: ``` @@ -709,22 +710,22 @@ Next, we activate the python environment we created and install the requirements source startrace/bin/activate pip install -r requirements.txt ``` -We will also create a file called `StarTrace.py` as follows: +We will also create a file called `startrace.py` as follows: ``` -touch StarTrace.py +touch startrace.py ``` This file will be used in the following sections. ##### Create the Schema -Before we can import data or run queries, we need to create our schema. The following imports implement the python-pilosa. This is all done in the `StarTrace.py` file: +Before we can import data or run queries, we need to create our schema. The following imports can be seen from the python-pilosa repo. This is all done in the `startrace.py` file: ``` import pilosa from pilosa import Client, Index, TimeQuantum from pilosa.imports import csv_column_reader, csv_row_id_column_id ``` -The first steps to creating the schema are creating a client which will communicate our schema to Pilosa, creating a schema which will contain our indexes and fields, and syncing with Pilosa: +Create the schema by creating a client which will communicate our schema to Pilosa, creating a schema which will contain our indexes and fields, and syncing with Pilosa: ``` from __future__ import print_function @@ -760,7 +761,7 @@ Let's create the `stargazer` field which has user IDs of stargazers as its rows: ``` stargazer = repository.field("stargazer", time_quantum=pilosa.TimeQuantum.YEAR_MONTH_DAY) ``` -Since our data contains time stamps which represent the time users starred repos, we establish the time aspect by using `time_quantum`. Time quantum is the resolution of the time we want to use, and we set it to `YEAR_MONTH-DAY` for `stargazer`. +Since our data contains time stamps which represent the time users starred repos, we establish the time aspect by using `time_quantum`. Time quantum is the resolution of the time we want to use, and we set it to `YEAR_MONTH_DAY` for `stargazer`. Next up is the `language` field, which will contain IDs for programming languages: ``` From 636c7d2966a50f5aa7784d563ef35315ca08cf88 Mon Sep 17 00:00:00 2001 From: Ashley Svetlik Date: Mon, 1 Jul 2019 16:42:46 -0500 Subject: [PATCH 09/17] Made syntax, format, and wording corrections --- docs/getting-started.md | 191 ++++++++++++++++++++++++++-------------- 1 file changed, 127 insertions(+), 64 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index 48f8fc7c8..b9881aa99 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -44,7 +44,7 @@ In order to better understand Pilosa's capabilities, we will create a sample pro Although Pilosa doesn't keep the data in a tabular format, we still use the terms "columns" and "rows" when describing the data model. We put the primary objects in columns, and the properties of those objects in rows. For example, the Star Trace project will contain an index called "repository" which contains columns representing Github repositories, and rows representing properties like programming languages and stargazers. We can better organize the rows by grouping them into sets called Fields. So the "repository" index might have a "languages" field as well as a "stargazers" field. You can learn more about indexes and fields in the [Data Model](../data-model/) section of the documentation. -Pilosa as an organization supports curl (or any HTTP tool), Go, Java, and Python. However, Pilosa as a server will support any client that can send requests to it. In this project, we will walk you through how to use each one to best communicate with the Pilosa server. +Pilosa officially supports curl (or any HTTP tool), Go, Java, and Python, however it will accept any client that can send requests to it. In this project, we will walk you through how to use each one to best communicate with the Pilosa server. Note: If at any time you want to verify the data structure, you can request the schema as follows: @@ -52,9 +52,40 @@ Note: If at any time you want to verify the data structure, you can request the curl localhost:10101/schema ``` ``` response -{"indexes":[{"name":"repository","options":{"keys":false,"trackExistence":true},"fields":[{"name":"language","options":{"type":"set","cacheType":"ranked","cacheSize":50000,"keys":false}},{"name":"stargazer","options":{"type":"time","timeQuantum":"YMDH","keys":false,"noStandardView":false}}],"shardWidth":1048576}} +{ + "indexes": [ + { + "name": "repository", + "options": { + "keys": false, + "trackExistence": true + }, + "fields": [ + { + "name": "language", + "options": { + "type": "set", + "cacheType": "ranked", + "cacheSize": 50000, + "keys": false + } + }, + { + "name": "stargazer", + "options": { + "type": "time", + "timeQuantum": "YMDH", + "keys": false, + "noStandardView": false + } + } + ], + "shardWidth": 1048576 + } + ] +} ``` -Note: This is the response you should recieve once completing this project. +Note: This is the response you should receive once completing this project. It has also been formatted using `jq`. #### Using Curl @@ -110,6 +141,14 @@ pilosa import -i repository -f stargazer stargazer.csv pilosa import -i repository -f language language.csv ``` +If you are using a Docker container for Pilosa (with name `pilosa`), you should instead copy the `*.csv` file into the container and then import them: +``` +docker cp stargazer.csv pilosa:/stargazer.csv +docker exec -it pilosa /pilosa import -i repository -f stargazer /stargazer.csv +docker cp language.csv pilosa:/language.csv +docker exec -it pilosa /pilosa import -i repository -f language /language.csv +``` + Note that both the user IDs and the repository IDs were remapped to sequential integers in the data files, they don't correspond to actual Github IDs anymore. You can check out [languages.txt](https://github.com/pilosa/getting-started/blob/master/languages.txt) to see the mapping for languages. ##### Make Some Queries @@ -227,18 +266,18 @@ Don't try to use arbitrary 64-bit integers as column or row IDs in Pilosa - this #### Using Go -Pilosa requires Go 1.12 or higher. +Pilosa supports the two most recent versions of Go. ##### Create the Environment -In order to communicate with Pilosa through your Go code, you must have a client, which is go-pilosa. To install go-pilosa, open a terminal (one other than the one running Pilosa) and download the library to your `GOPATH` using: +Interacting with Pilosa in your go program is best accomplished using our client, go-pilosa. To install go-pilosa, open a new terminal and download the library to your `GOPATH` using: ``` go get github.com/pilosa/go-pilosa ``` -To contain the Getting Started project in one place, we will create a new folder as follows: +Create a project folder: ``` -mkdir getting_started && cd getting_started +mkdir getting-started && cd getting-started ``` In this folder, we will download two CSV files to provide data to our fields later on. Download the `stargazer.csv` and `language.csv` files here: @@ -255,7 +294,7 @@ This file will be used in the following sections. ##### Create the Schema -Before we can import data or run queries, we need to create our schema. You can see two imports from the go-pilosa repo, go-pilosa for the client, and csv for the CSV reader. Create the schema by creating a client which will communicate our schema to Pilosa, creating a schema which will contain our indexes and fields, and syncing with Pilosa. This is all done in the `startrace.go` file: +Before we can import data or run queries, we need to create our schema. You can see two imports from the go-pilosa repo, go-pilosa for the client, and csv for the CSV reader. Create the schema by creating a client (which will communicate our schema to Pilosa), creating a schema locally (which will contain our indexes and fields), and syncing with Pilosa. This is all done in the `startrace.go` file: ``` package main @@ -450,9 +489,9 @@ Pilosa requires Java 8 or higher and Maven 3 or higher. ##### Create the Environment -To contain the Getting Started project in one place, we will create a new folder as follows: +Create a project folder: ``` -mkdir getting_started && cd getting_started +mkdir getting-started && cd getting-started ``` In this folder, we will download two CSV files to provide data to our fields later on. Download the `stargazer.csv` and `language.csv` files here: @@ -461,61 +500,92 @@ curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/stargaze curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/language.csv ``` -We will now create the java directory that will contain our `pom.xml` file and then import the `pom.xml` file: +We will now create the java directory that will contain our `pom.xml` file and create the `pom.xml` file: ``` mkdir startrace && cd startrace -curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/java/startrace/pom.xml +touch pom.xml ``` -For this specific project, the `pom.xml` file needs to be edited. The file can be edited by typing `nano pom.xml` directly into the terminal or simply using your code editing software. The following needs to be changed: +For this specific project, the `pom.xml` file needs to contain: ``` - - - com.pilosa - pilosa-client - **1.3.1** - - + + + 4.0.0 - - - org.apache.maven.plugins - maven-jar-plugin - 3.0.2 - - - - true - lib/ - **main.java.startrace** - - - - + com.pilosa + getting-started + 1.0.0 + + + + com.pilosa + pilosa-client + 1.3.1 + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.6.1 + + 1.8 + 1.8 + + + + + + org.apache.maven.plugins + maven-jar-plugin + 3.0.2 + + + + true + lib/ + main.java.StarTrace + + + + + + + + org.apache.maven.plugins + maven-shade-plugin + 3.0.0 + + + package + + shade + + + + + + + + + ``` -We will now create the java directory that will contain our `startrace.java` file and create the `startrace.java` file: +We will now create the java directory that will contain our `StarTrace.java` file and create the `StarTrace.java` file: ``` -mkdir src && cd src -mkdir main && cd main -mkdir java && cd java -touch startrace.go +mkdir -p src/main/java && cd src/main/java +touch StarTrace.java ``` This file will be used in the following sections. ##### Create the Schema -Before we can import data or run queries, we need to create our schema. The following imports can be seen from the java-pilosa repo: -``` -import com.pilosa.client.PilosaClient; -import com.pilosa.client.QueryResponse; -import com.pilosa.client.exceptions.PilosaException; -import com.pilosa.client.orm.*; -import com.pilosa.client.csv.FileRecordIterator; -import com.pilosa.client.TimeQuantum; -``` -Create the schema by creating a client which will communicate our schema to Pilosa, creating a schema which will contain our indexes and fields, and syncing with Pilosa. This is all done in the `startrace.java` file: +Before we can import data or run queries, we need to create our schema. The first 6 dependencies are imported from the java-pilosa library. Create the schema by creating a client which will communicate our schema to Pilosa, creating a schema which will contain our indexes and fields, and syncing with Pilosa. This is all done in the `StarTrace.java` file: ``` package main.java; @@ -529,7 +599,7 @@ import com.pilosa.client.TimeQuantum; import java.io.IOException; import java.text.SimpleDateFormat; -public class startrace { +public class StarTrace { public static void main(String []args) throws IOException { // Create the Schema PilosaClient client = PilosaClient.defaultClient(); @@ -562,7 +632,7 @@ Next up is the `language` field, which will contain IDs for programming language ``` The `language` field is a `set` field, but since the default field type is `set`, we don't need to specify it -Your `startrace.java` file should look like: +Your `StarTrace.java` file should look like: ``` package main.java; @@ -576,7 +646,7 @@ import com.pilosa.client.TimeQuantum; import java.io.IOException; import java.text.SimpleDateFormat; -public class startrace { +public class StarTrace { public static void main(String []args) throws IOException { // Create the Schema PilosaClient client = PilosaClient.defaultClient(); @@ -604,7 +674,7 @@ First, we will load our data into the `stargazer` field: FileRecordIterator iterator = FileRecordIterator.fromPath("stargazer.csv", stargazer, timestampFormat); client.importField(stargazer, iterator); ``` -Due to the time aspect of the `stargazer` csv file, we have to specify the time stamp format in the `fromPath` function. We set the variable `timestampFormat` to the format present in the csv file using the function `SimpleDateFormat()` and call the variable in the `fromPath` function, which will take the csv file name, the field name, and the time stamp format and translate the csv file into a format Pilosa can read. +Due to the time aspect of the `stargazer` csv file, we have to specify the time stamp format in the `fromPath` function. We set the variable `timestampFormat` to the format present in the csv file using the function `SimpleDateFormat()` and pass the variable to the `fromPath` function, which will take the csv file name, the field name, and the time stamp format and translate the csv file into a format Pilosa can read. Next, we will load our data into the `language` field: ``` @@ -686,9 +756,9 @@ Pilosa requires Python 2.7 or higher or Python 3.4 or higher. ##### Create the Environment -To contain the Getting Started project in one place, we will create a new folder as follows: +Create a new project folder: ``` -mkdir getting_started && cd getting_started +mkdir getting-started && cd getting-started ``` In this folder, we will download two CSV files to provide data to our fields later on. Download the `stargazer.csv` and `language.csv` files here: ``` @@ -718,14 +788,7 @@ This file will be used in the following sections. ##### Create the Schema -Before we can import data or run queries, we need to create our schema. The following imports can be seen from the python-pilosa repo. This is all done in the `startrace.py` file: -``` -import pilosa -from pilosa import Client, Index, TimeQuantum -from pilosa.imports import csv_column_reader, csv_row_id_column_id -``` - -Create the schema by creating a client which will communicate our schema to Pilosa, creating a schema which will contain our indexes and fields, and syncing with Pilosa: +Before we can import data or run queries, we need to create our schema. The dependencies dealing with `pilosa` are from the python-pilosa library. Create the schema by creating a client which will communicate our schema to Pilosa, creating a schema which will contain our indexes and fields, and syncing with Pilosa. This is all done in the `startrace.py` file: ``` from __future__ import print_function From f40958fd5afe4214bffe7914afd9249e62597ac2 Mon Sep 17 00:00:00 2001 From: Ashley Svetlik Date: Tue, 2 Jul 2019 11:27:27 -0500 Subject: [PATCH 10/17] Improved documentation wording --- docs/getting-started.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index b9881aa99..5422d04c4 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -44,7 +44,7 @@ In order to better understand Pilosa's capabilities, we will create a sample pro Although Pilosa doesn't keep the data in a tabular format, we still use the terms "columns" and "rows" when describing the data model. We put the primary objects in columns, and the properties of those objects in rows. For example, the Star Trace project will contain an index called "repository" which contains columns representing Github repositories, and rows representing properties like programming languages and stargazers. We can better organize the rows by grouping them into sets called Fields. So the "repository" index might have a "languages" field as well as a "stargazers" field. You can learn more about indexes and fields in the [Data Model](../data-model/) section of the documentation. -Pilosa officially supports curl (or any HTTP tool), Go, Java, and Python, however it will accept any client that can send requests to it. In this project, we will walk you through how to use each one to best communicate with the Pilosa server. +Pilosa officially supports three client libraries, for Go, Java and Python. You can also use any HTTP client, such as curl, for quick testing, but official client libraries are the preferred method in production code. Note: If at any time you want to verify the data structure, you can request the schema as follows: @@ -89,8 +89,6 @@ Note: This is the response you should receive once completing this project. It h #### Using Curl -Note: This is not the recommended way to interact with Pilosa, but it is the fastest way to see the efficiency of Pilosa. - ##### Create the Schema Before we can import data or run queries, we need to create our indexes and the fields within them. Let's create the `repository` index first: @@ -266,7 +264,7 @@ Don't try to use arbitrary 64-bit integers as column or row IDs in Pilosa - this #### Using Go -Pilosa supports the two most recent versions of Go. +Pilosa follows the Go policy of supporting the two most recent major versions of Go. ##### Create the Environment @@ -585,7 +583,7 @@ This file will be used in the following sections. ##### Create the Schema -Before we can import data or run queries, we need to create our schema. The first 6 dependencies are imported from the java-pilosa library. Create the schema by creating a client which will communicate our schema to Pilosa, creating a schema which will contain our indexes and fields, and syncing with Pilosa. This is all done in the `StarTrace.java` file: +Before we can import data or run queries, we need to create our schema. You can see the first six dependencies are imported from the java-pilosa library. Create the schema by creating a client which will communicate our schema to Pilosa, creating a schema which will contain our indexes and fields, and syncing with Pilosa. This is all done in the `StarTrace.java` file: ``` package main.java; @@ -788,7 +786,7 @@ This file will be used in the following sections. ##### Create the Schema -Before we can import data or run queries, we need to create our schema. The dependencies dealing with `pilosa` are from the python-pilosa library. Create the schema by creating a client which will communicate our schema to Pilosa, creating a schema which will contain our indexes and fields, and syncing with Pilosa. This is all done in the `startrace.py` file: +Before we can import data or run queries, we need to create our schema. You can see the dependencies dealing with `pilosa` are from the python-pilosa library. Create the schema by creating a client which will communicate our schema to Pilosa, creating a schema which will contain our indexes and fields, and syncing with Pilosa. This is all done in the `startrace.py` file: ``` from __future__ import print_function From 36c75ea416d7264e2b878207bb6c122211ff9b0c Mon Sep 17 00:00:00 2001 From: Ashley Svetlik Date: Tue, 2 Jul 2019 15:21:16 -0500 Subject: [PATCH 11/17] Removed Note before schema check --- docs/getting-started.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index 5422d04c4..6b84f1ac1 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -46,7 +46,7 @@ Although Pilosa doesn't keep the data in a tabular format, we still use the term Pilosa officially supports three client libraries, for Go, Java and Python. You can also use any HTTP client, such as curl, for quick testing, but official client libraries are the preferred method in production code. -Note: If at any time you want to verify the data structure, you can request the schema as follows: +If at any time you want to verify the data structure, you can request the schema as follows: ``` request curl localhost:10101/schema From f991df206c6665aa5f10b42cd885863cb8a7eb5e Mon Sep 17 00:00:00 2001 From: Ashley Svetlik Date: Tue, 2 Jul 2019 15:24:56 -0500 Subject: [PATCH 12/17] Made Schema check into note --- docs/getting-started.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index 6b84f1ac1..7199acd39 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -46,7 +46,8 @@ Although Pilosa doesn't keep the data in a tabular format, we still use the term Pilosa officially supports three client libraries, for Go, Java and Python. You can also use any HTTP client, such as curl, for quick testing, but official client libraries are the preferred method in production code. -If at any time you want to verify the data structure, you can request the schema as follows: +
+

If at any time you want to verify the data structure, you can request the schema as follows: ``` request curl localhost:10101/schema @@ -85,7 +86,9 @@ curl localhost:10101/schema ] } ``` -Note: This is the response you should receive once completing this project. It has also been formatted using `jq`. + +Note: This is the response you should receive once completing this project. It has also been formatted using `jq`. <\p> +

#### Using Curl From e38983782c3cc5ee99e94a9968631c045ef3659d Mon Sep 17 00:00:00 2001 From: Ashley Svetlik Date: Tue, 2 Jul 2019 15:35:07 -0500 Subject: [PATCH 13/17] Fixed Schema check note --- docs/getting-started.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index 7199acd39..a47ff17e2 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -47,7 +47,8 @@ Although Pilosa doesn't keep the data in a tabular format, we still use the term Pilosa officially supports three client libraries, for Go, Java and Python. You can also use any HTTP client, such as curl, for quick testing, but official client libraries are the preferred method in production code.
-

If at any time you want to verify the data structure, you can request the schema as follows: +

If at any time you want to verify the data structure, you can request the schema as follows:<\p> +<\div> ``` request curl localhost:10101/schema @@ -86,8 +87,8 @@ curl localhost:10101/schema ] } ``` - -Note: This is the response you should receive once completing this project. It has also been formatted using `jq`. <\p> +

+

Note: This is the response you should receive once completing this project. It has also been formatted using `jq`. <\p>

#### Using Curl From 614bcff1e0047e488a1bb84638f39d252add2d0c Mon Sep 17 00:00:00 2001 From: Ashley Svetlik Date: Wed, 3 Jul 2019 08:12:58 -0500 Subject: [PATCH 14/17] Deleted redundant paragraph in Sample Project --- docs/getting-started.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index a47ff17e2..2c383ac99 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -15,7 +15,7 @@ nav = [ ## Getting Started Pilosa supports an HTTP interface which uses JSON by default. -Any HTTP tool can be used to interact with the Pilosa server. The examples in this documentation will use curl which is available by default on many UNIX-like systems including Linux and MacOS. However, the best way to interface with the Pilosa server is through one of our three official client libraries. Pilosa currently supports [Go](https://github.com/pilosa/go-pilosa), [Java](https://github.com/pilosa/java-pilosa), and [Python](https://github.com/pilosa/python-pilosa). +Any HTTP tool can be used to interact with the Pilosa server. The examples in this documentation will use [curl](https://curl.haxx.se/) which is available by default on many UNIX-like systems including Linux and MacOS. However, the best way to interface with the Pilosa server is through one of our three official client libraries. Pilosa currently supports [Go](https://github.com/pilosa/go-pilosa), [Java](https://github.com/pilosa/java-pilosa), and [Python](https://github.com/pilosa/python-pilosa).

Note that Pilosa server requires a high limit for open files. Check the documentation of your system to see how to increase it in case you hit that limit. See Open File Limits for more details.

@@ -44,8 +44,6 @@ In order to better understand Pilosa's capabilities, we will create a sample pro Although Pilosa doesn't keep the data in a tabular format, we still use the terms "columns" and "rows" when describing the data model. We put the primary objects in columns, and the properties of those objects in rows. For example, the Star Trace project will contain an index called "repository" which contains columns representing Github repositories, and rows representing properties like programming languages and stargazers. We can better organize the rows by grouping them into sets called Fields. So the "repository" index might have a "languages" field as well as a "stargazers" field. You can learn more about indexes and fields in the [Data Model](../data-model/) section of the documentation. -Pilosa officially supports three client libraries, for Go, Java and Python. You can also use any HTTP client, such as curl, for quick testing, but official client libraries are the preferred method in production code. -

If at any time you want to verify the data structure, you can request the schema as follows:<\p> <\div> From daad23b388f2ec0e412d381c2ee1a36c8d523972 Mon Sep 17 00:00:00 2001 From: Ashley Svetlik Date: Wed, 3 Jul 2019 13:52:18 -0500 Subject: [PATCH 15/17] Made review chnages --- docs/getting-started.md | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index 2c383ac99..52803bac2 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -45,13 +45,12 @@ In order to better understand Pilosa's capabilities, we will create a sample pro Although Pilosa doesn't keep the data in a tabular format, we still use the terms "columns" and "rows" when describing the data model. We put the primary objects in columns, and the properties of those objects in rows. For example, the Star Trace project will contain an index called "repository" which contains columns representing Github repositories, and rows representing properties like programming languages and stargazers. We can better organize the rows by grouping them into sets called Fields. So the "repository" index might have a "languages" field as well as a "stargazers" field. You can learn more about indexes and fields in the [Data Model](../data-model/) section of the documentation.

-

If at any time you want to verify the data structure, you can request the schema as follows:<\p> -<\div> +

If at any time you want to verify the data structure, you can request the schema as follows:

-``` request + curl localhost:10101/schema -``` -``` response + + { "indexes": [ { @@ -84,9 +83,8 @@ curl localhost:10101/schema } ] } -``` -
-

Note: This is the response you should receive once completing this project. It has also been formatted using `jq`. <\p> + +

Note: This is the response you should receive once completing this project. It has also been formatted using [`jq`](https://stedolan.github.io/jq/). <\p>

#### Using Curl @@ -100,7 +98,7 @@ curl localhost:10101/index/repository -X POST ``` response {"success":true} ``` -The index name must be 64 characters or less, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. The same goes for field names. +The index name must be 64 characters or fewer, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. The same goes for field names. Let's create the `stargazer` field which has user IDs of stargazers as its rows: ``` request @@ -325,7 +323,7 @@ Next, let's create the `repository` index: repository := schema.Index("repository") ``` -The index name must be 64 characters or less, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. The same goes for field names. +The index name must be 64 characters or fewer, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. The same goes for field names. Let's create the `stargazer` field which has user IDs of stargazers as its rows: ``` @@ -615,7 +613,7 @@ Next, let's create the `repository` index: ``` Index repository = schema.index("repository"); ``` -The index name must be 64 characters or less, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. The same goes for field names. +The index name must be 64 characters or fewer, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. The same goes for field names. Let's create the `stargazer` field which has user IDs of stargazers as its rows: ``` @@ -624,7 +622,7 @@ Let's create the `stargazer` field which has user IDs of stargazers as its rows: .build(); Field stargazer = repository.field("stargazer", stargazerOptions); ``` -Since our data contains time stamps which represent the time users starred repos, we set the field type to `time` using `fieldTime()`. Time quantum is the resolution of the time we want to use, and we set it to `YEAR_MONTH-DAY` for `stargazer`. +Since our data contains time stamps which represent the time users starred repos, we set the field type to `time` using `fieldTime()`. Time quantum is the resolution of the time we want to use, and we set it to `YEAR_MONTH_DAY` for `stargazer`. Next up is the `language` field, which will contain IDs for programming languages: ``` @@ -775,7 +773,7 @@ We will now create the python environment: python3 -m venv startrace ``` -Next, we activate the python environment we created and install the requirements (and python-pilosa): +Next, we activate the python environment we created and install the single dependency, python-pilosa: ``` source startrace/bin/activate pip install -r requirements.txt @@ -818,7 +816,7 @@ Next, let's create the `repository` index: ``` repository = schema.index("repository") ``` -The index name must be 64 characters or less, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. The same goes for field names. +The index name must be 64 characters or fewer, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. The same goes for field names. Let's create the `stargazer` field which has user IDs of stargazers as its rows: ``` From 2db061ac215eddb4a6dcca889c9bd3888423a0bf Mon Sep 17 00:00:00 2001 From: Ashley Svetlik Date: Wed, 3 Jul 2019 14:02:00 -0500 Subject: [PATCH 16/17] Reformatted Schema Check --- docs/getting-started.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index 52803bac2..61a86890e 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -46,11 +46,12 @@ Although Pilosa doesn't keep the data in a tabular format, we still use the term

If at any time you want to verify the data structure, you can request the schema as follows:

+
- +```request curl localhost:10101/schema - - +``` +```response { "indexes": [ { @@ -83,8 +84,9 @@ curl localhost:10101/schema } ] } - -

Note: This is the response you should receive once completing this project. It has also been formatted using [`jq`](https://stedolan.github.io/jq/). <\p> +``` +

+

Note: This is the response you should receive once completing this project. It has also been formatted using [jq](https://stedolan.github.io/jq/).

#### Using Curl From c6e840ea30f098b8e3469462a76271ce5503c243 Mon Sep 17 00:00:00 2001 From: Ashley Svetlik Date: Wed, 3 Jul 2019 14:12:11 -0500 Subject: [PATCH 17/17] Fixed jq note link --- docs/getting-started.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index 61a86890e..a236d2156 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -86,7 +86,7 @@ curl localhost:10101/schema } ```
-

Note: This is the response you should receive once completing this project. It has also been formatted using [jq](https://stedolan.github.io/jq/).

+

Note: This is the response you should receive once completing this project. It has also been formatted using jq.

#### Using Curl