From 4eed160b2de55f0377135bb3405e933676a0f17a Mon Sep 17 00:00:00 2001 From: Matthew Jaffee Date: Mon, 22 Oct 2018 23:30:13 -0500 Subject: [PATCH 1/2] wip on adding import docs --- docs/api-reference.md | 36 ++++++++++++++++++++++++++++++++++++ docs/getting-started.md | 5 +++++ docs/query-language.md | 4 ++++ 3 files changed, 45 insertions(+) diff --git a/docs/api-reference.md b/docs/api-reference.md index 216a01bca..4a49e3515 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -94,6 +94,40 @@ curl "localhost:10101/index/user/query?columnAttrs=true&shards=0,1" \ By default, all bits and attributes (*for `Row` queries only*) are returned. In order to suppress returning bits, set `excludeBits` query argument to `true`; to suppress returning attributes, set `excludeAttrs` query argument to `true`. +### Import Data + +`POST /index//field/` + +Supports high-rate data ingest to a particular shard of a particular field. The +official client libraries use this endpoint for their import functionality - it +is not usually necessary to use this endpoint directly. See the documentation for +imports for +Go, +Java, +and Python. + +The request payload is protobuf encoded with the following schema. The row or +column Keys fields are used if the field or index is configured for keys +respectively. Otherwise, the RowIDs and ColumnIDs fields are used. They must +have the same number of items, and each index into those two lists represents a +particular bit to be set. Timestamps are optional, but if they exist must also +contain the same number of items as rows and columns. The column IDs must all be +in the shard specified in the request. + +``` +message ImportRequest { + string Index = 1; + string Field = 2; + uint64 Shard = 3; + repeated uint64 RowIDs = 4; + repeated uint64 ColumnIDs = 5; + repeated string RowKeys = 7; + repeated string ColumnKeys = 8; + repeated int64 Timestamps = 6; +} +``` + + ### Create field `POST /index//field/` @@ -158,6 +192,8 @@ curl -XDELETE localhost:10101/index/user/field/language {"success":true} ``` + + ### Get version `GET /version` diff --git a/docs/getting-started.md b/docs/getting-started.md index 563176c62..257e77628 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -89,6 +89,11 @@ The `language` is a `set` field, but since the default field type is `set`, we d #### Import Data From CSV Files +
+

For demonstration purposes, we're using Pilosa's built in utility to import specially formatted CSV files. For more general usage, see how the various client libraries expose the bulk import functionality in Go, Java, and Python.

+
+ + Download the `stargazer.csv` and `language.csv` files here: ``` diff --git a/docs/query-language.md b/docs/query-language.md index 9bc231e6f..0f8feb795 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -67,6 +67,10 @@ Set(, =, [TIMESTAMP]) `Set` assigns a value of 1 to a bit in the binary matrix, thus associating the given row (the `` value) in the given field with the given column. +
+

While using "Set" in PQL is a convenient way to get familiar with Pilosa, it's almost always better to use the import functionality in the Go, Java, and Python clients to ingest lots of data.

+
+ **Result Type:** boolean A return value of `true` indicates that the bit was changed to 1. From 376c2d61ffa7c90df3b90d0251f892dd11008a22 Mon Sep 17 00:00:00 2001 From: Matt Jaffee Date: Tue, 23 Oct 2018 13:50:35 -0500 Subject: [PATCH 2/2] address CR feedback --- docs/api-reference.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/api-reference.md b/docs/api-reference.md index 4a49e3515..f5f06fd57 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -96,7 +96,7 @@ By default, all bits and attributes (*for `Row` queries only*) are returned. In ### Import Data -`POST /index//field/` +`POST /index//field//import` Supports high-rate data ingest to a particular shard of a particular field. The official client libraries use this endpoint for their import functionality - it @@ -106,13 +106,13 @@ imports for Java, and Python. -The request payload is protobuf encoded with the following schema. The row or -column Keys fields are used if the field or index is configured for keys -respectively. Otherwise, the RowIDs and ColumnIDs fields are used. They must -have the same number of items, and each index into those two lists represents a -particular bit to be set. Timestamps are optional, but if they exist must also -contain the same number of items as rows and columns. The column IDs must all be -in the shard specified in the request. +The request payload is protobuf encoded with the following schema. The RowKeys +and/or ColumnKeys fields are used if the pilosa field or index are configured +for keys respectively. Otherwise, the RowIDs and ColumnIDs fields are used. They +must have the same number of items, and each index into those two lists +represents a particular bit to be set. Timestamps are optional, but if they +exist must also contain the same number of items as rows and columns. The +column IDs must all be in the shard specified in the request. ``` message ImportRequest {