From 2c8d9a5ae0c9d1492c771b0305763141dfda8454 Mon Sep 17 00:00:00 2001 From: Ilya Tocar Date: Wed, 28 Feb 2018 16:41:51 -0600 Subject: [PATCH 01/34] roaring: speed-up intersectionCountArrayBitmap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit While looking at benchmarks as a possible go compiler benchmarks. I've tried some optimizations by hand: Move len(b.bitmap) load out of the loop. Remove some type conversions. Use (x >> off) & 1 to get offs bit, instead of x & (1 << of)f >> off. This produces nice speed-up and passes go test roaring. name old time/op new time/op delta Bitmap_IntersectionCount_ArrayRun-6 2.06µs ± 0% 1.57µs ± 0% -24.04% (p=0.000 n=10+9) Bitmap_IntersectionCount_BitmapRun-6 2.24µs ± 0% 2.24µs ± 0% ~ (p=0.913 n=10+9) Bitmap_IntersectionCount_ArrayBitmap-6 2.06µs ± 0% 1.56µs ± 1% -24.05% (p=0.000 n=9+10) --- roaring/roaring.go | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/roaring/roaring.go b/roaring/roaring.go index 8ea82af83..9e233347e 100644 --- a/roaring/roaring.go +++ b/roaring/roaring.go @@ -1907,13 +1907,14 @@ func intersectionCountBitmapRun(a, b *container) (n int) { } func intersectionCountArrayBitmap(a, b *container) (n int) { + ln := len(b.bitmap) for _, val := range a.array { - i := val >> 6 - if i >= uint16(len(b.bitmap)) { + i := int(val >> 6) + if i >= ln { break } off := val % 64 - n += int((b.bitmap[i] & (1 << off)) >> off) + n += int(b.bitmap[i]>>off) & 1 } return n } From 2b1a3c84036b8eef27f604953c6e1bc923f51de2 Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Tue, 6 Mar 2018 17:48:14 -0600 Subject: [PATCH 02/34] Add docs readme --- docs/README.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 docs/README.md diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 000000000..bf4fb3b91 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,5 @@ +Pilosa docs are maintained here, to stay in sync with the codebase. + +Please visit https://www.pilosa.com/docs to view the docs complete with styles, diagrams, and comprehensive search. + +Have you found a discrepancy, typo, or other problem? Please submit an [issue](https://github.com/pilosa/pilosa/issues/new) or a pull request! From e7eb68b1019f9c9564ee99a31816f7eeb118bc62 Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Tue, 6 Mar 2018 17:48:40 -0600 Subject: [PATCH 03/34] Add cosmosa to new external tutorials section --- docs/tutorials.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/tutorials.md b/docs/tutorials.md index 4d787577c..ffaf51d64 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -8,6 +8,10 @@ nav = [ ] +++ +### External Tutorials + +- [Run Pilosa with Microsoft's Azure Cosmos DB](https://github.com/pilosa/cosmosa) + ## Tutorials ### Setting Up a Secure Cluster From 04b17847ceac18f28f4b9d69950a7a90ef3892fd Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Tue, 6 Mar 2018 18:14:22 -0600 Subject: [PATCH 04/34] Clean up glossary --- docs/glossary.md | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/docs/glossary.md b/docs/glossary.md index 50906a0cf..998bda86b 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -20,9 +20,7 @@ nav = [] Attribute: Attributes can be associated to both rows and columns. This metadata is kept separately from the core binary matrix in a BoltDB store. -PQL: Pilosa Query Language - -Index: The Index represents a data namespace. +PQL: [Pilosa Query Language](/docs/query-language). Frame: Frames are used to segment rows into different categories - row ids are namespaced by frame such that the same row id in a different frame refers to a different row. For Ranked frames, rows are kept in sorted order within the frame. @@ -38,7 +36,7 @@ nav = [] Anti-entropy: A periodic process that compares each slice and its replicas across the cluster to repair inconsistencies. -Node: An individual running instance of Pilosa server which belongs to a cluster. +Node: An individual running instance of Pilosa server which belongs to a cluster. Cluster: A cluster consists of one or more nodes which share a cluster configuration. The cluster also defines how data is replicated throughout and how internode communication is coordinated. Pilosa does not have a leader node, all data is evenly distributed, and any node can respond to queries. @@ -46,13 +44,12 @@ nav = [] Tanimoto: Used for similarity queries on Pilosa data. The Tanimoto Coefficient is the ratio of the intersecting set to the union set as the measure of similarity. -Protobuf:: [Protocol Buffers](https://developers.google.com/protocol-buffers/) is a binary serialization format which Pilosa uses for internal messages, and can be used by clients as an alternative to JSON. +Protobuf: [Protocol Buffers](https://developers.google.com/protocol-buffers/) is a binary serialization format which Pilosa uses for internal messages, and can be used by clients as an alternative to JSON. TOML: We use [TOML](https://github.com/toml-lang/toml) for our configuration file format. -Jump Consistent Hash: A fast, minimal memory, consistent hash algorithm that evenly distributes the workload even when the number of buckets changes. -https://arxiv.org/pdf/1406.2294v1.pdf +Jump Consistent Hash: A fast, minimal memory, [consistent hash algorithm](https://arxiv.org/pdf/1406.2294v1.pdf) that evenly distributes the workload even when the number of buckets changes. -Partition: The consistent hash is compiled with a maximum number of partitions or locations on the unit circle that keys are mapped to. Partitions are then evenly mapped to physical nodes. To add nodes to the cluster you simply need to remap the partitions, and associated data across the new cluster topography. +Partition: The consistent hash is compiled with a maximum number of partitions or locations on the unit circle that keys are mapped to. Partitions are then evenly mapped to physical nodes. To add nodes to the cluster you simply need to remap the partitions, and associate data across the new cluster topography. -Replica: A copy of a [fragment] on a different host from the original. The "cluster.replicas" configuration parameter determines how many replicas of a fragment exist in the cluster (including the original, so a value of 1 means no extra copies are made). +Replica: A copy of a [fragment](#fragment) on a different host from the original. The "cluster.replicas" configuration parameter determines how many replicas of a fragment exist in the cluster (including the original, so a value of 1 means no extra copies are made). From b59901b40361093d3bf013f16904ef8e29475954 Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Wed, 7 Mar 2018 16:13:25 -0600 Subject: [PATCH 05/34] Linkify docs url --- docs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/README.md b/docs/README.md index bf4fb3b91..31b60a622 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,5 +1,5 @@ Pilosa docs are maintained here, to stay in sync with the codebase. -Please visit https://www.pilosa.com/docs to view the docs complete with styles, diagrams, and comprehensive search. +Please visit [our website](https://www.pilosa.com/docs) to view the docs complete with styles, diagrams, and comprehensive search. Have you found a discrepancy, typo, or other problem? Please submit an [issue](https://github.com/pilosa/pilosa/issues/new) or a pull request! From 64bf4827aa73f99c634139afeb3e1b2db81127ad Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Wed, 7 Mar 2018 16:15:18 -0600 Subject: [PATCH 06/34] Minor fixes --- docs/administration.md | 10 +++++----- docs/architecture.md | 3 +++ docs/data-model.md | 10 ++++++++-- docs/examples.md | 2 +- docs/query-language.md | 3 ++- docs/tutorials.md | 2 ++ docs/webui.md | 3 ++- 7 files changed, 23 insertions(+), 10 deletions(-) diff --git a/docs/administration.md b/docs/administration.md index 97ee688f5..664301286 100644 --- a/docs/administration.md +++ b/docs/administration.md @@ -112,17 +112,17 @@ Note: This will only work when the replication factor is >= 2 #### Copying data files manually -- To accomplish this goal you will 1st need: - - List of all Indexes on your cluster - - List of all frames in your Indexes - - Max slice per Index, listed in the /status endpoint +- To accomplish this you will first need: + - List of all indexes on your cluster + - List of all frames in your indexes + - Max slice per index, listed in the /status endpoint - With this information you can query the `/fragment/nodes` endpoint and iterate over each slice - Using the list of slices owned by this node you will then need to manually: - setup a directory structure similar to the other nodes with a path for each Index/Frame - copy each owned slice for an existing node to this new node - Modify the cluster config file to replace the previous node address with the new node address. - Restart the cluster -- Wait for the 1st sync (10 minutes) to validate Index connections +- Wait for the first sync (10 minutes) to validate Index connections ### Diagnostics diff --git a/docs/architecture.md b/docs/architecture.md index 5cc5ff6ee..81a7e54f7 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -18,5 +18,8 @@ Bitmaps are persisted to disk using a file format very similar to the [Roaring B * After the container storage section is an operation log, of unspecified length. ![roaring file format diagram](/img/docs/pilosa-roaring-storage-diagram.png) +*Pilosa Roaring storage format diagram* All values are little-endian. The first two bytes of the cookie is 12348, to reflect incompatibility with the spec, which uses 12346 or 12347. Container types are NOT inferred from their cardinality as in the spec. Instead, the container type is read directly from the descriptive header. + +Check out this [blog post](/blog/adding-rle-support/) for some more details about Roaring in Pilosa. diff --git a/docs/data-model.md b/docs/data-model.md index d448a1de8..1fc5f899a 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -26,7 +26,8 @@ Pilosa lays out data first in rows, so queries which get all the set bits in one Please note that Pilosa is most performant when row and column IDs are sequential starting from 0. You can deviate from this to some degree, but if you try to set a bit with column ID 2^63, bad things will start to happen. -![data model diagram](/img/docs/data-model.svg) +![basic data model diagram](/img/docs/data-model.svg) +*Basic data model diagram* ### Index @@ -51,12 +52,14 @@ Row attributes are namespaced at the Frame level. Ranked Frames maintain a sorted cache of column counts by Row ID (yielding the top rows by columns with a bit set in each). This cache facilitates the TopN query. The cache size defaults to 50,000 and can be set at Frame creation. ![ranked frame diagram](/img/docs/frame-ranked.svg) +*Ranked frame diagram* #### LRU The LRU cache maintains the most recently accessed Rows. ![lru frame diagram](/img/docs/frame-lru.svg) +*LRU frame diagram* ### Time Quantum @@ -92,6 +95,7 @@ SetBit(frame="A", rowID=19, columnID=5) ``` ![inverse frame diagram](/img/docs/frame-inverse.svg) +*Inverse frame diagram* #### Time Quantums @@ -103,6 +107,7 @@ SetBit(frame="A", rowID=8, columnID=3, timestamp="2017-05-19T00:00") ``` ![time quantum frame diagram](/img/docs/frame-time-quantum.svg) +*Time quantum frame diagram* #### BSI Range-Encoding @@ -122,4 +127,5 @@ SetFieldValue(col=2, frame="A", field1=1) SetFieldValue(col=3, frame="A", field1=6) ``` -![BSI diagram](/img/docs/frame-bsi.svg) +![BSI frame diagram](/img/docs/frame-bsi.svg) +*BSI frame diagram* diff --git a/docs/examples.md b/docs/examples.md index bf5e76f29..262684615 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -214,7 +214,7 @@ T(A,B)= Intersect(A,B) / (Count(A) + Count(B) - Intersect(A,B)) A and B are sets of fingerprint bits on in the fingerprints of molecule A and molecule B. AB is the set of common bits of fingerprints of both molecule A and B. The Tanimoto coefficient ranges from 0 when the fingerprints have no bits in common, to 1 when the fingerprints are identical. -All source code to calculate tanimoto for molecule fingerprint using Pilosa is available in a Github repository https://github.com/pilosa/chem-usecase +All source code to calculate tanimoto for molecule fingerprint using Pilosa is available in a [Github repository](https://github.com/pilosa/chem-usecase). #### Data model diff --git a/docs/query-language.md b/docs/query-language.md index 3a5b94d60..83098fe0a 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -13,7 +13,7 @@ nav = [ ### Overview -This section will provide a detailed reference and examples for the Pilosa Query Language (PQL). All PQL queries operate on a single [index]({{< ref "glossary.md#index" >}}) and are passed to Pilosa through the `/index/*index_name*/query` endpoint. You may pass multiple PQL queries in a single request by simply concatenating the queries together - a space is not needed. The results format is always: +This section will provide a detailed reference and examples for the Pilosa Query Language (PQL). All PQL queries operate on a single [index]({{< ref "glossary.md#index" >}}) and are passed to Pilosa through the `/index/INDEX_NAME/query` endpoint. You may pass multiple PQL queries in a single request by simply concatenating the queries together - a space is not needed. The results format is always: ``` {"results":[...]} @@ -380,6 +380,7 @@ have the attribute specified by `field` with one of the values specified in **Result Type:** array of key/count objects **Caveats:** + * Performing a TopN() query on a frame with cache type ranked will return the top bitmaps sorted by count in descending order. * Frames with cache type lru will maintain an LRU (Least Recently Used) cache, thus a TopN() query on this type of frame will return bitmaps sorted in order of most recently set bit. * The frame's cache size determines the number of sorted bitmaps to maintain in the cache for purposes of TopN() queries. There is a tradeoff between performance and accuracy; increasing the cache size will improve accuracy of results at the cost of performance. diff --git a/docs/tutorials.md b/docs/tutorials.md index ffaf51d64..2ffb1b41d 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -49,6 +49,7 @@ openssl req -x509 -newkey rsa:2048 -keyout pilosa.local.key -out pilosa.local.cr ``` The command above creates two files in the current directory: + * `pilosa.local.crt` is the SSL certificate. * `pilosa.local.key` is the private key file which must be kept as secret. @@ -130,6 +131,7 @@ key = "pilosa.local.gossip32" ``` Here is some explanation of the configuration items: + * `data-dir` points to the directory where the Pilosa server writes its data. If it doesn't exist, the server will create it. * `bind` is the address to which the server listens for incoming requests. The address is composed of three parts: scheme, host, and port. The default scheme is `http` so we explicitly specify `https` to use the HTTPS protocol for communication between nodes. * `[cluster]` section contains the settings for a cluster. `hosts` field is the most important, which contains the list of addresses of other nodes. See [Cluster Configuration](https://www.pilosa.com/docs/latest/configuration/#cluster-hosts) for other settings. diff --git a/docs/webui.md b/docs/webui.md index cef8e570d..86874807b 100644 --- a/docs/webui.md +++ b/docs/webui.md @@ -20,7 +20,8 @@ Each query's result will be displayed in the Output section along with the query The Console will keep a record of each query and its result with the latest query on top. -![console](/img/docs/webui-console.png) +![webUI console screenshot](/img/docs/webui-console.png) +*WebUI console screenshot* In addition to standard PQL, the console supports a few special commands, prefixed with `:`. From 2811ed7b09bda8d5fdb2394c2fc0a869e5890308 Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Wed, 7 Mar 2018 16:15:45 -0600 Subject: [PATCH 07/34] Improve and linkify glossary --- docs/glossary.md | 48 ++++++++++++++++++++++++------------------------ 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/docs/glossary.md b/docs/glossary.md index 998bda86b..4607973e9 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -6,50 +6,50 @@ nav = [] ## Glossary -Index: Indexes are the top level container in Pilosa - similar to a database in an RDBMS. Queries cannot operate across multiple indexes. +[Index](../data-model/#index): An Index is a top level container in Pilosa, analogous to a database in an RDBMS. Queries cannot operate across multiple indexes. -Column: Columns are the fundamental horizontal data axis within Pilosa. Columns are global to all Frames within a Index. +[Column](../data-model/#column): Columns are the fundamental horizontal data axis within Pilosa. Columns are global to all [frames](#frame) within an [index](#index). -Row: Rows are the fundamental vertical data axis within Pilosa. They are namespaced to each Frame within a Index. +[Row](../data-model/#row): Rows are the fundamental vertical data axis within Pilosa. They are namespaced to each [frame](#frame) within an [index](#index). Represented as a [bitmap][#bitmap]. -Bit: A bit is the intersection of a Row and Column. +[Bit](../data-model/#overview): Bits are the fundamental unit of data in Pilosa. A bit lives in a [frame](#frame), at the intersection of a [row](#row) and [column](#column). -Bitmap: The on-disk and in-memory representation of a Row. +[Bitmap](../data-model/#overview): The on-disk and in-memory representation of a [row](#row). Implemented with [Roaring](#roaring-bitmap). -Roaring Bitmap: [Roaring Bitmap](http://roaringbitmap.org) is the compressed bitmap format which Pilosa uses. +[Roaring Bitmap](http://roaringbitmap.org): the compressed bitmap format which Pilosa uses to [implement bitmaps](../architecture/#roaring-bitmap-storage-format), for both storage and logical query operations. -Attribute: Attributes can be associated to both rows and columns. This metadata is kept separately from the core binary matrix in a BoltDB store. +[Attribute](../data-model/#attribute): Attributes can be associated to both [rows](#row) and [columns](#column). This metadata is kept separately from the core binary matrix in a [BoltDB](https://github.com/boltdb/bolt) store. -PQL: [Pilosa Query Language](/docs/query-language). +[PQL](../query-language/): Pilosa Query Language. -Frame: Frames are used to segment rows into different categories - row ids are namespaced by frame such that the same row id in a different frame refers to a different row. For Ranked frames, rows are kept in sorted order within the frame. +[Frame](../data-model/#frame): Frames are used to group [rows](#row) into different categories. `RowID`s are namespaced by frame such that the same `RowID` in a different frame refers to a different row. For [ranked](#topn) frames, rows are kept in sorted order within the frame. -View: Views separate the different data layouts within a Frame. The two primary views are Standard and Inverse which represent the typical row/column data and its inverse respectively. Time based Frame Views are automatically generated for each time quantum. Views are internally managed by Pilosa, and never exposed directly via the API. This simplifies the functional interface by separating it from the physical data representation. +[View](../data-model/#view): Views separate the different data layouts within a [Frame](#frame). The two primary views are standard and inverse which represent the typical [row](#row)/[column](#column) data and its inverse respectively (an [inverted index](https://en.wikipedia.org/wiki/Inverted_index), or a matrix transpose). Time based frame views are automatically generated for each time quantum. Views are internally managed by Pilosa, and never exposed directly via the API. This simplifies the functional interface by separating it from the physical data representation. -Fragment: A Fragment is the intersection of a frame and slice in an index. +Fragment: A Fragment is the intersection of a [frame](#frame) and a [slice](#slice) in an [index](#index). -Slice: Columns are sharded on a preset width. Each shard is referred to as a Slice in Pilosa. Slices are operated on in parallel and are evenly distributed across the cluster via a consistent hash. +[Slice](../data-model/#slice): [Columns](#column) are sharded on a preset [width](#slicewidth). Each shard is referred to as a slice in Pilosa. Slices are operated on in parallel and are evenly distributed across the cluster via a [consistent hash](#jump-consistent-hash). -SliceWidth: This is the default number of columns in a slice. +SliceWidth: This is the number of [columns](#column) in a [slice](#slice). By default, 220 or about one million. -MaxSlice: The total number of slices allocated to handle current set of columns. This value is important for all nodes to efficiently distribute queries. +MaxSlice: The total number of [slices](#slice) allocated to handle the current set of [columns](#columns). This value is important for all [nodes](#node) to efficiently distribute queries. -Anti-entropy: A periodic process that compares each slice and its replicas across the cluster to repair inconsistencies. +[Anti-entropy](../configuration/#anti-entropy-interval): A periodic process that compares each [slice](#slice) and its [replicas](#replica) across the [cluster](#cluster) to repair inconsistencies. -Node: An individual running instance of Pilosa server which belongs to a cluster. +Node: An individual running instance of Pilosa server which belongs to a [cluster](#cluster). -Cluster: A cluster consists of one or more nodes which share a cluster configuration. The cluster also defines how data is replicated throughout and how internode communication is coordinated. Pilosa does not have a leader node, all data is evenly distributed, and any node can respond to queries. +Cluster: A cluster consists of one or more [nodes](#node) which share a cluster configuration. The cluster also defines how data is [replicated](#replica) throughout and how internode communication is coordinated. Pilosa does not have a leader node, all data is evenly distributed, and any node can respond to queries. -TopN: Given a Frame and/or RowID this query returns the ordered set of RowID's by the number of columns that have a bit set in that row. +TopN: A [PQL](#pql) query that returns a list of `RowID`s, sorted by the count of [bits](#bit) set in the [row](#row), within a specified [frame](#frame). -Tanimoto: Used for similarity queries on Pilosa data. The Tanimoto Coefficient is the ratio of the intersecting set to the union set as the measure of similarity. +[Tanimoto](../examples/#chemical-similarity-search): Used for similarity queries on Pilosa data. The [Tanimoto Coefficient](https://en.wikipedia.org/wiki/Jaccard_index#Tanimoto_similarity_and_distance) between two [bitmaps](#bitmap) A and B is the ratio of the size of their intersection to the size of their union (|A∩B|/|A∪B|). -Protobuf: [Protocol Buffers](https://developers.google.com/protocol-buffers/) is a binary serialization format which Pilosa uses for internal messages, and can be used by clients as an alternative to JSON. +[Protobuf](https://developers.google.com/protocol-buffers/): Protocol Buffers is a binary serialization format which Pilosa uses for internal messages, and can be used by clients as an alternative to JSON. -TOML: We use [TOML](https://github.com/toml-lang/toml) for our configuration file format. +[TOML](https://github.com/toml-lang/toml): the language used for Pilosa's [configuration file](../configuration). -Jump Consistent Hash: A fast, minimal memory, [consistent hash algorithm](https://arxiv.org/pdf/1406.2294v1.pdf) that evenly distributes the workload even when the number of buckets changes. +[Jump Consistent Hash](https://arxiv.org/pdf/1406.2294v1.pdf): A fast, minimal memory, consistent hash algorithm that evenly distributes the workload even when the number of buckets changes. -Partition: The consistent hash is compiled with a maximum number of partitions or locations on the unit circle that keys are mapped to. Partitions are then evenly mapped to physical nodes. To add nodes to the cluster you simply need to remap the partitions, and associate data across the new cluster topography. +Partition: The [consistent hash](#jump-consistent-hash) maps keys to partitions (or locations on the unit circle), based on a preset maximum number of partitions (256 by default). Partitions are then evenly mapped to physical [nodes](#node). To add nodes to the [cluster](#cluster), the partitions must be remapped, and data is then associated across the new cluster topology. -Replica: A copy of a [fragment](#fragment) on a different host from the original. The "cluster.replicas" configuration parameter determines how many replicas of a fragment exist in the cluster (including the original, so a value of 1 means no extra copies are made). +[Replica](../configuration/#cluster-replicas): A copy of a [fragment](#fragment) on a different [node](#node) than the original. The `cluster.replicas` configuration parameter determines how many replicas of a fragment exist in the cluster. This includes the original, so a value of 1 means no extra copies are made. From cf41e7c3f732badfbc64f8c677db0fc707b1486e Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Wed, 7 Mar 2018 16:17:05 -0600 Subject: [PATCH 08/34] Alphabetize glossary --- docs/glossary.md | 50 ++++++++++++++++++++++++------------------------ 1 file changed, 25 insertions(+), 25 deletions(-) diff --git a/docs/glossary.md b/docs/glossary.md index 4607973e9..2d56bcbd3 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -6,50 +6,50 @@ nav = [] ## Glossary -[Index](../data-model/#index): An Index is a top level container in Pilosa, analogous to a database in an RDBMS. Queries cannot operate across multiple indexes. +[Anti-entropy](../configuration/#anti-entropy-interval): A periodic process that compares each [slice](#slice) and its [replicas](#replica) across the [cluster](#cluster) to repair inconsistencies. -[Column](../data-model/#column): Columns are the fundamental horizontal data axis within Pilosa. Columns are global to all [frames](#frame) within an [index](#index). - -[Row](../data-model/#row): Rows are the fundamental vertical data axis within Pilosa. They are namespaced to each [frame](#frame) within an [index](#index). Represented as a [bitmap][#bitmap]. +[Attribute](../data-model/#attribute): Attributes can be associated to both [rows](#row) and [columns](#column). This metadata is kept separately from the core binary matrix in a [BoltDB](https://github.com/boltdb/bolt) store. [Bit](../data-model/#overview): Bits are the fundamental unit of data in Pilosa. A bit lives in a [frame](#frame), at the intersection of a [row](#row) and [column](#column). [Bitmap](../data-model/#overview): The on-disk and in-memory representation of a [row](#row). Implemented with [Roaring](#roaring-bitmap). -[Roaring Bitmap](http://roaringbitmap.org): the compressed bitmap format which Pilosa uses to [implement bitmaps](../architecture/#roaring-bitmap-storage-format), for both storage and logical query operations. +Cluster: A cluster consists of one or more [nodes](#node) which share a cluster configuration. The cluster also defines how data is [replicated](#replica) throughout and how internode communication is coordinated. Pilosa does not have a leader node, all data is evenly distributed, and any node can respond to queries. -[Attribute](../data-model/#attribute): Attributes can be associated to both [rows](#row) and [columns](#column). This metadata is kept separately from the core binary matrix in a [BoltDB](https://github.com/boltdb/bolt) store. +[Column](../data-model/#column): Columns are the fundamental horizontal data axis within Pilosa. Columns are global to all [frames](#frame) within an [index](#index). -[PQL](../query-language/): Pilosa Query Language. +Fragment: A Fragment is the intersection of a [frame](#frame) and a [slice](#slice) in an [index](#index). [Frame](../data-model/#frame): Frames are used to group [rows](#row) into different categories. `RowID`s are namespaced by frame such that the same `RowID` in a different frame refers to a different row. For [ranked](#topn) frames, rows are kept in sorted order within the frame. -[View](../data-model/#view): Views separate the different data layouts within a [Frame](#frame). The two primary views are standard and inverse which represent the typical [row](#row)/[column](#column) data and its inverse respectively (an [inverted index](https://en.wikipedia.org/wiki/Inverted_index), or a matrix transpose). Time based frame views are automatically generated for each time quantum. Views are internally managed by Pilosa, and never exposed directly via the API. This simplifies the functional interface by separating it from the physical data representation. +[Index](../data-model/#index): An Index is a top level container in Pilosa, analogous to a database in an RDBMS. Queries cannot operate across multiple indexes. -Fragment: A Fragment is the intersection of a [frame](#frame) and a [slice](#slice) in an [index](#index). +[Jump Consistent Hash](https://arxiv.org/pdf/1406.2294v1.pdf): A fast, minimal memory, consistent hash algorithm that evenly distributes the workload even when the number of buckets changes. + +MaxSlice: The total number of [slices](#slice) allocated to handle the current set of [columns](#columns). This value is important for all [nodes](#node) to efficiently distribute queries. + +Node: An individual running instance of Pilosa server which belongs to a [cluster](#cluster). + +Partition: The [consistent hash](#jump-consistent-hash) maps keys to partitions (or locations on the unit circle), based on a preset maximum number of partitions (256 by default). Partitions are then evenly mapped to physical [nodes](#node). To add nodes to the [cluster](#cluster), the partitions must be remapped, and data is then associated across the new cluster topology. + +[PQL](../query-language/): Pilosa Query Language. + +[Protobuf](https://developers.google.com/protocol-buffers/): Protocol Buffers is a binary serialization format which Pilosa uses for internal messages, and can be used by clients as an alternative to JSON. + +[Replica](../configuration/#cluster-replicas): A copy of a [fragment](#fragment) on a different [node](#node) than the original. The `cluster.replicas` configuration parameter determines how many replicas of a fragment exist in the cluster. This includes the original, so a value of 1 means no extra copies are made. + +[Roaring Bitmap](http://roaringbitmap.org): the compressed bitmap format which Pilosa uses to [implement bitmaps](../architecture/#roaring-bitmap-storage-format), for both storage and logical query operations. + +[Row](../data-model/#row): Rows are the fundamental vertical data axis within Pilosa. They are namespaced to each [frame](#frame) within an [index](#index). Represented as a [bitmap][#bitmap]. [Slice](../data-model/#slice): [Columns](#column) are sharded on a preset [width](#slicewidth). Each shard is referred to as a slice in Pilosa. Slices are operated on in parallel and are evenly distributed across the cluster via a [consistent hash](#jump-consistent-hash). SliceWidth: This is the number of [columns](#column) in a [slice](#slice). By default, 220 or about one million. -MaxSlice: The total number of [slices](#slice) allocated to handle the current set of [columns](#columns). This value is important for all [nodes](#node) to efficiently distribute queries. - -[Anti-entropy](../configuration/#anti-entropy-interval): A periodic process that compares each [slice](#slice) and its [replicas](#replica) across the [cluster](#cluster) to repair inconsistencies. - -Node: An individual running instance of Pilosa server which belongs to a [cluster](#cluster). - -Cluster: A cluster consists of one or more [nodes](#node) which share a cluster configuration. The cluster also defines how data is [replicated](#replica) throughout and how internode communication is coordinated. Pilosa does not have a leader node, all data is evenly distributed, and any node can respond to queries. - -TopN: A [PQL](#pql) query that returns a list of `RowID`s, sorted by the count of [bits](#bit) set in the [row](#row), within a specified [frame](#frame). - [Tanimoto](../examples/#chemical-similarity-search): Used for similarity queries on Pilosa data. The [Tanimoto Coefficient](https://en.wikipedia.org/wiki/Jaccard_index#Tanimoto_similarity_and_distance) between two [bitmaps](#bitmap) A and B is the ratio of the size of their intersection to the size of their union (|A∩B|/|A∪B|). -[Protobuf](https://developers.google.com/protocol-buffers/): Protocol Buffers is a binary serialization format which Pilosa uses for internal messages, and can be used by clients as an alternative to JSON. - [TOML](https://github.com/toml-lang/toml): the language used for Pilosa's [configuration file](../configuration). -[Jump Consistent Hash](https://arxiv.org/pdf/1406.2294v1.pdf): A fast, minimal memory, consistent hash algorithm that evenly distributes the workload even when the number of buckets changes. +TopN: A [PQL](#pql) query that returns a list of `RowID`s, sorted by the count of [bits](#bit) set in the [row](#row), within a specified [frame](#frame). -Partition: The [consistent hash](#jump-consistent-hash) maps keys to partitions (or locations on the unit circle), based on a preset maximum number of partitions (256 by default). Partitions are then evenly mapped to physical [nodes](#node). To add nodes to the [cluster](#cluster), the partitions must be remapped, and data is then associated across the new cluster topology. - -[Replica](../configuration/#cluster-replicas): A copy of a [fragment](#fragment) on a different [node](#node) than the original. The `cluster.replicas` configuration parameter determines how many replicas of a fragment exist in the cluster. This includes the original, so a value of 1 means no extra copies are made. +[View](../data-model/#view): Views separate the different data layouts within a [Frame](#frame). The two primary views are standard and inverse which represent the typical [row](#row)/[column](#column) data and its inverse respectively (an [inverted index](https://en.wikipedia.org/wiki/Inverted_index), or a matrix transpose). Time based frame views are automatically generated for each time quantum. Views are internally managed by Pilosa, and never exposed directly via the API. This simplifies the functional interface by separating it from the physical data representation. From a562e73fa4df109afdf7eefeea9e643a6a1cba33 Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Wed, 7 Mar 2018 18:11:22 -0600 Subject: [PATCH 09/34] Minor updates --- docs/administration.md | 2 +- docs/data-model.md | 6 ++++-- docs/query-language.md | 4 ++-- docs/tutorials.md | 2 +- 4 files changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/administration.md b/docs/administration.md index 664301286..bf3da9626 100644 --- a/docs/administration.md +++ b/docs/administration.md @@ -64,7 +64,7 @@ pilosa import -i project -f stargazer --field star_count project-stargazer-count ```
-

Note that you must first create a frame with Range Encoding enabled and a field. View Create Frame for more details.

+

Note that you must first create a frame with range-encoding enabled and a field. View Create Frame for more details.

#### Exporting diff --git a/docs/data-model.md b/docs/data-model.md index 1fc5f899a..9ab684203 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -63,7 +63,7 @@ The LRU cache maintains the most recently accessed Rows. ### Time Quantum -Setting a time quantum on a frame creates extra indices which allow Range queries down to the interval specified. For example - if the time quantum is set to `YMD`, Range queries down to the granularity of a day are supported. +Setting a time quantum on a frame creates extra views which allow Range queries down to the time interval specified. For example - if the time quantum is set to `YMD`, Range queries down to the granularity of a day are supported. ### Attribute @@ -112,7 +112,7 @@ SetBit(frame="A", rowID=8, columnID=3, timestamp="2017-05-19T00:00") #### BSI Range-Encoding Bit-Sliced Indexing (BSI) is the storage method Pilosa uses to represent multi-bit integers in a bitmap index. Integers are stored as n-bit, range-encoded -bit-sliced indexes of base-2, along with an additional bitmap indicating "not null". This means that a 16-bit integer will require 17 bitmaps: one for each 0-bit of the 16 bit-slice components (the 1-bit does not need to be stored because with range-encoding the highest bit position is always 1) and one for the non-null bitmap. Pilosa can evaluate, aggregate, and range queries on these BSI integers. +bit-sliced indexes of base-2, along with an additional bitmap indicating "not null". This means that a 16-bit integer will require 17 bitmaps: one for each 0-bit of the 16 bit-slice components (the 1-bit does not need to be stored because with range-encoding the highest bit position is always 1) and one for the non-null bitmap. Pilosa can evaluate `Sum` and `Range` queries on these BSI integers. Internally Pilosa stores each BSI `field` as a `view` within a `frame`. The 'rowIDs' of the `view` are composed of the base-2 representation of the integer. Pilosa manages the base-2 offset and translation that efficiently packs the integer value within the minimum set of rows. @@ -129,3 +129,5 @@ SetFieldValue(col=3, frame="A", field1=6) ![BSI frame diagram](/img/docs/frame-bsi.svg) *BSI frame diagram* + +Check out this [blog post](/blog/range-encoded-bitmaps/) for some more details about BSI in Pilosa. diff --git a/docs/query-language.md b/docs/query-language.md index 83098fe0a..56d7c2087 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -473,7 +473,7 @@ Returns bits that are true for the comparison operator. **Examples:** In our source data, commitactivity was counted over the last year. -The following greater-than Range query returns all repositories having more than 100 commits. +The following greater-than `Range` query returns all repositories having more than 100 commits. ``` Range(frame="stats", commitactivity > 100) @@ -513,7 +513,7 @@ Sum([BITMAP_CALL], , ) **Description:** -Returns the count and computed sum of all bitmap encoded integer values across the `field` in this `frame`. The optional Bitmap call filters the bits used in this computation. +Returns the count and computed sum of all BSI integer values across the `field` in this `frame`. The optional `Bitmap` call filters the bits used in this computation. **Result Type:** object with the computed sum and count of the bitmap field. diff --git a/docs/tutorials.md b/docs/tutorials.md index 2ffb1b41d..9e6364cf0 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -241,7 +241,7 @@ Check out our [Administration Guide](https://www.pilosa.com/docs/latest/administ #### Introduction -Pilosa can store integer values associated to the columns in an index, and those values are used to support range and aggregate queries. In this tutorial we will show how to set up integer fields, populate those fields with data, and query the fields. The example index we're going to create will represent fictional patients at a medical facility and various bits of information about those patients. +Pilosa can store integer values associated to the columns in an index, and those values are used to support `Range` and `Sum` queries. In this tutorial we will show how to set up integer fields, populate those fields with data, and query the fields. The example index we're going to create will represent fictional patients at a medical facility and various bits of information about those patients. First, create an index called `patients`: ``` request From f371e1c2f2a21d59e76e07fa8c276a0f367449fe Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Wed, 7 Mar 2018 18:11:39 -0600 Subject: [PATCH 10/34] Add glossary terms --- docs/glossary.md | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/docs/glossary.md b/docs/glossary.md index 2d56bcbd3..5b2a24ffd 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -12,12 +12,16 @@ nav = [] [Bit](../data-model/#overview): Bits are the fundamental unit of data in Pilosa. A bit lives in a [frame](#frame), at the intersection of a [row](#row) and [column](#column). -[Bitmap](../data-model/#overview): The on-disk and in-memory representation of a [row](#row). Implemented with [Roaring](#roaring-bitmap). +[Bitmap](../data-model/#overview): The on-disk and in-memory representation of a [row](#row). Implemented with [Roaring](#roaring-bitmap). `Bitmap` is also the basic [PQL](#pql) query for reading a Bitmap. + +[BSI](../data-model/#bsi-range-encoding) Bit-sliced indexing is the method Pilosa uses to represent multi-bit integers. Integer values are stored in [fields](#field), and can be used for [Range](#range) and [Sum](#sum) queries. Cluster: A cluster consists of one or more [nodes](#node) which share a cluster configuration. The cluster also defines how data is [replicated](#replica) throughout and how internode communication is coordinated. Pilosa does not have a leader node, all data is evenly distributed, and any node can respond to queries. [Column](../data-model/#column): Columns are the fundamental horizontal data axis within Pilosa. Columns are global to all [frames](#frame) within an [index](#index). +[Field](../data-model/#bsi-range-encoding): A group of rows used to store integer values with [BSI](#bsi), for use in [Range](#range) and [Sum](#sum) queries. + Fragment: A Fragment is the intersection of a [frame](#frame) and a [slice](#slice) in an [index](#index). [Frame](../data-model/#frame): Frames are used to group [rows](#row) into different categories. `RowID`s are namespaced by frame such that the same `RowID` in a different frame refers to a different row. For [ranked](#topn) frames, rows are kept in sorted order within the frame. @@ -30,26 +34,34 @@ nav = [] Node: An individual running instance of Pilosa server which belongs to a [cluster](#cluster). -Partition: The [consistent hash](#jump-consistent-hash) maps keys to partitions (or locations on the unit circle), based on a preset maximum number of partitions (256 by default). Partitions are then evenly mapped to physical [nodes](#node). To add nodes to the [cluster](#cluster), the partitions must be remapped, and data is then associated across the new cluster topology. +Partition: The [consistent hash](#jump-consistent-hash) maps keys to partitions (or locations on the unit circle), based on a preset maximum number of partitions. Partitions are then evenly mapped to physical [nodes](#node). To add nodes to the [cluster](#cluster), the partitions must be remapped, and data is then associated across the new cluster topology. `DefaultPartitionN` is 256. It can be modified, but only at compile time, and before ingesting any data. [PQL](../query-language/): Pilosa Query Language. [Protobuf](https://developers.google.com/protocol-buffers/): Protocol Buffers is a binary serialization format which Pilosa uses for internal messages, and can be used by clients as an alternative to JSON. +[Range](../query-lanuage/#range):: A [PQL](#pql) query that returns bits based on comparison to timestamps, set according to the [time quantum](#time-quantum). + +[Range (BSI)](../query-lanuage/#range-bsi):: A [PQL](#pql) query that returns bits based on comparison to integers stored in [BSI](#bsi) [fields](#field). + [Replica](../configuration/#cluster-replicas): A copy of a [fragment](#fragment) on a different [node](#node) than the original. The `cluster.replicas` configuration parameter determines how many replicas of a fragment exist in the cluster. This includes the original, so a value of 1 means no extra copies are made. [Roaring Bitmap](http://roaringbitmap.org): the compressed bitmap format which Pilosa uses to [implement bitmaps](../architecture/#roaring-bitmap-storage-format), for both storage and logical query operations. -[Row](../data-model/#row): Rows are the fundamental vertical data axis within Pilosa. They are namespaced to each [frame](#frame) within an [index](#index). Represented as a [bitmap][#bitmap]. +[Row](../data-model/#row): Rows are the fundamental vertical data axis within Pilosa. They are namespaced to each [frame](#frame) within an [index](#index). Represented as a [Bitmap](#bitmap). [Slice](../data-model/#slice): [Columns](#column) are sharded on a preset [width](#slicewidth). Each shard is referred to as a slice in Pilosa. Slices are operated on in parallel and are evenly distributed across the cluster via a [consistent hash](#jump-consistent-hash). -SliceWidth: This is the number of [columns](#column) in a [slice](#slice). By default, 220 or about one million. +SliceWidth: This is the number of [columns](#column) in a [slice](#slice). `SliceWidth` defaults to 220 or about one million. It can be modified, but only at compile time, and before ingesting any data. -[Tanimoto](../examples/#chemical-similarity-search): Used for similarity queries on Pilosa data. The [Tanimoto Coefficient](https://en.wikipedia.org/wiki/Jaccard_index#Tanimoto_similarity_and_distance) between two [bitmaps](#bitmap) A and B is the ratio of the size of their intersection to the size of their union (|A∩B|/|A∪B|). +[Sum](../query-language/#sum): A [PQL](#pql) query that returns the sum of integers stored in [BSI](#bsi) [fields](#field). + +[Tanimoto](../examples/#chemical-similarity-search): Used for similarity queries on Pilosa data. The [Tanimoto Coefficient](https://en.wikipedia.org/wiki/Jaccard_index#Tanimoto_similarity_and_distance) between two [Bitmaps](#bitmap) A and B is the ratio of the size of their intersection to the size of their union (|A∩B|/|A∪B|). + +[Time quantum](../data-model/#time-quantum): Defines the granularity to be used for time [Range](#range) queries. [TOML](https://github.com/toml-lang/toml): the language used for Pilosa's [configuration file](../configuration). -TopN: A [PQL](#pql) query that returns a list of `RowID`s, sorted by the count of [bits](#bit) set in the [row](#row), within a specified [frame](#frame). +[TopN](../query-language#topn): A [PQL](#pql) query that returns a list of `RowID`s, sorted by the count of [bits](#bit) set in the [row](#row), within a specified [frame](#frame). -[View](../data-model/#view): Views separate the different data layouts within a [Frame](#frame). The two primary views are standard and inverse which represent the typical [row](#row)/[column](#column) data and its inverse respectively (an [inverted index](https://en.wikipedia.org/wiki/Inverted_index), or a matrix transpose). Time based frame views are automatically generated for each time quantum. Views are internally managed by Pilosa, and never exposed directly via the API. This simplifies the functional interface by separating it from the physical data representation. +[View](../data-model/#view): Views separate the different data layouts within a [Frame](#frame). The two primary views are standard and inverse which represent the typical [row](#row)/[column](#column) data and its inverse respectively (an [inverted index](https://en.wikipedia.org/wiki/Inverted_index), or a matrix transpose). Time based frame views are automatically generated for each [time quantum](#time-quantum). Views are internally managed by Pilosa, and never exposed directly via the API. This simplifies the functional interface by separating it from the physical data representation. From cb0ac00eb3a5d74ce4e735997589082e81af954e Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Thu, 8 Mar 2018 14:31:03 -0600 Subject: [PATCH 11/34] Switch ref links to relative urls --- docs/api-reference.md | 12 ++++++------ docs/configuration.md | 2 +- docs/examples.md | 2 +- docs/getting-started.md | 2 +- docs/introduction.md | 2 +- docs/query-language.md | 8 ++++---- 6 files changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/api-reference.md b/docs/api-reference.md index 52ea2c369..5a28313b1 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -102,9 +102,9 @@ Creates a frame in the given index with the given name. The request payload is in JSON, and may contain the `options` field. The `options` field is a JSON object which may contain the following fields: -* `timeQuantum` (string): [Time Quantum]({{< ref "data-model.md#time-quantum" >}}) for this frame. -* `inverseEnabled` (boolean): Enables [the inverted view]({{< ref "data-model.md#inverse" >}}) for this frame if `true`. -* `cacheType` (string): [ranked]({{< ref "data-model.md#ranked" >}}) or [LRU]({{< ref "data-model.md#lru" >}}) caching on this frame. Default is `lru`. +* `timeQuantum` (string): [Time Quantum](../data-model#time-quantum) for this frame. +* `inverseEnabled` (boolean): Enables [the inverted view](../data-model#inverse) for this frame if `true`. +* `cacheType` (string): [ranked](../data-model#ranked) or [LRU](../data-model#lru) caching on this frame. Default is `lru`. * `cacheSize` (int): Number of rows to keep in the cache. Default 50,000. * `rangeEnabled` (boolean): Enables range-encoded fields in this frame. * `fields` (array): List of range-encoded fields. @@ -205,9 +205,9 @@ Creates an input definition in the given index with the given name. The request payload is JSON, and it must contain the fields `frames` and `fields`. `frames` is an array of frames used within this input definition. Each frame must contain a `name` and may contain the following options: -* `timeQuantum` (string): [Time Quantum]({{< ref "data-model.md#time-quantum" >}}) for this frame. -* `inverseEnabled` (boolean): Enables [the inverted view]({{< ref "data-model.md#inverse" >}}) for this frame if `true`. -* `cacheType` (string): [ranked]({{< ref "data-model.md#ranked" >}}) or [LRU]({{< ref "data-model.md#lru" >}}) caching on this frame. Default is `lru`. +* `timeQuantum` (string): [Time Quantum](../data-model#time-quantum) for this frame. +* `inverseEnabled` (boolean): Enables [the inverted view](../data-model#inverse) for this frame if `true`. +* `cacheType` (string): [ranked](../data-model#ranked) or [LRU](../data-model#lru) caching on this frame. Default is `lru`. * `cacheSize` (int): Number of rows to keep in the cache. Default 50,000. The `fields` array contains a series of JSON objects describing how to process each field received in the input data. Each `field` object must contain a `name` which maps to the source JSON field name. One field must be defined at the `primaryKey`. The `primarykey` source field name must equal the column label for the `Index`, and its value must be an unsigned integer which maps directly to a columnID in Pilosa. diff --git a/docs/configuration.md b/docs/configuration.md index 33a78018e..692a0a17a 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -112,7 +112,7 @@ Any flag that has a value that is a comma separated list on the command line bec #### Gossip Seed -* Description: When using the gossip [Cluster Type]({{< ref "#cluster-type" >}}), this specifies which internal host should be used to initialize membership in the cluster. Typcially this can be the address of any available host in the cluster. For example, when starting a three-node cluster made up of `node0`, `node1`, and `node2`, the `gossip-seed` for all three nodes can be configured to be the address of `node0`. +* Description: When using the gossip [Cluster Type](#cluster-type), this specifies which internal host should be used to initialize membership in the cluster. Typcially this can be the address of any available host in the cluster. For example, when starting a three-node cluster made up of `node0`, `node1`, and `node2`, the `gossip-seed` for all three nodes can be configured to be the address of `node0`. * Flag: `--gossip.seed="localhost:11101"` * Env: `PILOSA_GOSSIP_SEED="localhost:11101"` * Config: diff --git a/docs/examples.md b/docs/examples.md index 262684615..0cef7b987 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -263,7 +263,7 @@ python import_from_sdf.py -p -file id_fingerprint.csv ``` -First, follow the instruction in the [getting started]({{< ref "getting-started.md" >}}) guide to run a Pilosa server. Then create the indexes and frames according to the schemas outlined in the Data Model section above. +First, follow the instruction in the [getting started](../getting-started.md) guide to run a Pilosa server. Then create the indexes and frames according to the schemas outlined in the Data Model section above. The option cacheSize should be set as amount of chembl_id to calculate effectively for the whole data set, so we need to calculate amount of chembl_id. We have total 1678393 chembl_id (it will displayed after import_from_sdf.py script running), then the cacheSize should be >= 1678393 ``` curl localhost:10101/index/mole \ diff --git a/docs/getting-started.md b/docs/getting-started.md index f68805f53..7cfcaac9b 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -20,7 +20,7 @@ Any HTTP tool can be used to interact with the Pilosa server. The examples in th ### Starting Pilosa -Follow the steps in the [Install]({{< ref "installation.md" >}}) document to install Pilosa. +Follow the steps in the [Install](../installation.md) document to install Pilosa. Execute the following in a terminal to run Pilosa with the default configuration (Pilosa will be available at `localhost:10101`): ``` pilosa server diff --git a/docs/introduction.md b/docs/introduction.md index d499c8c0a..a49bd1539 100644 --- a/docs/introduction.md +++ b/docs/introduction.md @@ -16,4 +16,4 @@ It is designed primarly for speed and horizontal scalability. If you have data w "What attributes are the most common?", "Which objects have these specific attributes?", "What groups of attributes often appear together?" Pilosa is designed to answer these types of queries in real time, suitable for use with high rate data streams, or to power a user interface. -Once you have Pilosa [installed]({{< ref "installation.md" >}}), the [getting started]({{< ref "getting-started.md" >}}) guide will show you the basics of interacting with Pilosa and give you some pointers for deeper exploration. +Once you have Pilosa [installed](../installation.md), the [getting started](../getting-started.md) guide will show you the basics of interacting with Pilosa and give you some pointers for deeper exploration. diff --git a/docs/query-language.md b/docs/query-language.md index 56d7c2087..105893e66 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -13,7 +13,7 @@ nav = [ ### Overview -This section will provide a detailed reference and examples for the Pilosa Query Language (PQL). All PQL queries operate on a single [index]({{< ref "glossary.md#index" >}}) and are passed to Pilosa through the `/index/INDEX_NAME/query` endpoint. You may pass multiple PQL queries in a single request by simply concatenating the queries together - a space is not needed. The results format is always: +This section will provide a detailed reference and examples for the Pilosa Query Language (PQL). All PQL queries operate on a single [index](../glossary#index) and are passed to Pilosa through the `/index/INDEX_NAME/query` endpoint. You may pass multiple PQL queries in a single request by simply concatenating the queries together - a space is not needed. The results format is always: ``` {"results":[...]} @@ -45,7 +45,7 @@ curl localhost:10101/index/repository/query \ #### Arguments and Types -* `frame` The frame specifies on which Pilosa [frame]({{< ref "glossary.md#frame" >}}) the query will operate. Valid frame names are lower case strings; they start with an alphanumeric character, and contain only alphanumeric characters and `_-`. They must be 64 characters or less in length. +* `frame` The frame specifies on which Pilosa [frame](../glossary#frame) the query will operate. Valid frame names are lower case strings; they start with an alphanumeric character, and contain only alphanumeric characters and `_-`. They must be 64 characters or less in length. * `ROW_LABEL` The default row label is `rowID`, changing the default is deprecated. * `COL_LABEL` The default column label is `columnID`, changing the default is deprecated. * `TIMESTAMP` This is a timestamp in quotes with the following format `"YYYY-MM-DDTHH:MM"` (e.g. "2006-01-02T15:04") @@ -118,7 +118,7 @@ SetRowAttrs queries always return `null` upon success. SetRowAttrs(frame="stargazer", rowID=10, username="mrpi", active=true) ``` -Set username value and active status for user 10. These are arbitrary key/value pairs which have no meaning to Pilosa. You can see the attributes you've set on a row with a [Bitmap]({{< ref "query-language.md#bitmap" >}}) query like so `Bitmap(frame="stargazer", stargazer_id=10)`. +Set username value and active status for user 10. These are arbitrary key/value pairs which have no meaning to Pilosa. You can see the attributes you've set on a row with a [Bitmap](../query-language#bitmap) query like so `Bitmap(frame="stargazer", stargazer_id=10)`. ``` SetRowAttrs(frame="stargazer", rowID=10, username=null) @@ -150,7 +150,7 @@ SetColumnAttrs queries always return `null` upon success. Setting a value of `nu SetColumnAttrs(columnID=10, stars=123, url="http://projects.pilosa.com/10", active=true) ``` -Set url value and active status for project 10. These are arbitrary key/value pairs which have no meaning to Pilosa. You can see the attributes you've set on a column with a [Bitmap]({{< ref "query-language.md#bitmap" >}}) query like so `Bitmap(frame="stargazer", columnID=10)`. +Set url value and active status for project 10. These are arbitrary key/value pairs which have no meaning to Pilosa. You can see the attributes you've set on a column with a [Bitmap](../query-language#bitmap) query like so `Bitmap(frame="stargazer", columnID=10)`. ``` SetColumnAttrs(columnID=10, url=null) From 85799739af929f270ec12b8c39b7649a29967e89 Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Thu, 8 Mar 2018 15:55:21 -0600 Subject: [PATCH 12/34] Address review comments --- docs/README.md | 2 +- docs/examples.md | 2 +- docs/getting-started.md | 2 +- docs/glossary.md | 4 ++-- docs/introduction.md | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/README.md b/docs/README.md index 31b60a622..4bad84cd7 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,5 +1,5 @@ Pilosa docs are maintained here, to stay in sync with the codebase. -Please visit [our website](https://www.pilosa.com/docs) to view the docs complete with styles, diagrams, and comprehensive search. +Please visit [our website](https://www.pilosa.com/docs/) to view the docs complete with styles, diagrams, and comprehensive search. Have you found a discrepancy, typo, or other problem? Please submit an [issue](https://github.com/pilosa/pilosa/issues/new) or a pull request! diff --git a/docs/examples.md b/docs/examples.md index 0cef7b987..12dd69617 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -263,7 +263,7 @@ python import_from_sdf.py -p -file id_fingerprint.csv ``` -First, follow the instruction in the [getting started](../getting-started.md) guide to run a Pilosa server. Then create the indexes and frames according to the schemas outlined in the Data Model section above. +First, follow the instruction in the [getting started](../getting-started) guide to run a Pilosa server. Then create the indexes and frames according to the schemas outlined in the Data Model section above. The option cacheSize should be set as amount of chembl_id to calculate effectively for the whole data set, so we need to calculate amount of chembl_id. We have total 1678393 chembl_id (it will displayed after import_from_sdf.py script running), then the cacheSize should be >= 1678393 ``` curl localhost:10101/index/mole \ diff --git a/docs/getting-started.md b/docs/getting-started.md index 7cfcaac9b..6a9eb54be 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -20,7 +20,7 @@ Any HTTP tool can be used to interact with the Pilosa server. The examples in th ### Starting Pilosa -Follow the steps in the [Install](../installation.md) document to install Pilosa. +Follow the steps in the [Install](../installation) document to install Pilosa. Execute the following in a terminal to run Pilosa with the default configuration (Pilosa will be available at `localhost:10101`): ``` pilosa server diff --git a/docs/glossary.md b/docs/glossary.md index 5b2a24ffd..f6fa29478 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -40,9 +40,9 @@ nav = [] [Protobuf](https://developers.google.com/protocol-buffers/): Protocol Buffers is a binary serialization format which Pilosa uses for internal messages, and can be used by clients as an alternative to JSON. -[Range](../query-lanuage/#range):: A [PQL](#pql) query that returns bits based on comparison to timestamps, set according to the [time quantum](#time-quantum). +[Range](../query-language/#range):: A [PQL](#pql) query that returns bits based on comparison to timestamps, set according to the [time quantum](#time-quantum). -[Range (BSI)](../query-lanuage/#range-bsi):: A [PQL](#pql) query that returns bits based on comparison to integers stored in [BSI](#bsi) [fields](#field). +[Range (BSI)](../query-language/#range-bsi):: A [PQL](#pql) query that returns bits based on comparison to integers stored in [BSI](#bsi) [fields](#field). [Replica](../configuration/#cluster-replicas): A copy of a [fragment](#fragment) on a different [node](#node) than the original. The `cluster.replicas` configuration parameter determines how many replicas of a fragment exist in the cluster. This includes the original, so a value of 1 means no extra copies are made. diff --git a/docs/introduction.md b/docs/introduction.md index a49bd1539..7e7049c6c 100644 --- a/docs/introduction.md +++ b/docs/introduction.md @@ -16,4 +16,4 @@ It is designed primarly for speed and horizontal scalability. If you have data w "What attributes are the most common?", "Which objects have these specific attributes?", "What groups of attributes often appear together?" Pilosa is designed to answer these types of queries in real time, suitable for use with high rate data streams, or to power a user interface. -Once you have Pilosa [installed](../installation.md), the [getting started](../getting-started.md) guide will show you the basics of interacting with Pilosa and give you some pointers for deeper exploration. +Once you have Pilosa [installed](../installation), the [getting started](../getting-started) guide will show you the basics of interacting with Pilosa and give you some pointers for deeper exploration. From 13e3b0444335779da311fa7bc1f948faa1ea9555 Mon Sep 17 00:00:00 2001 From: Ilias Dimos Date: Fri, 9 Mar 2018 14:12:15 +0200 Subject: [PATCH 13/34] Fix misspells in comments --- ctl/import_test.go | 2 +- executor.go | 2 +- fragment_test.go | 2 +- gossip/gossip.go | 2 +- handler.go | 2 +- roaring/roaring.go | 4 ++-- roaring/roaring_test.go | 2 +- statsd/statsd.go | 2 +- 8 files changed, 9 insertions(+), 9 deletions(-) diff --git a/ctl/import_test.go b/ctl/import_test.go index 5979bdbee..254c21a05 100644 --- a/ctl/import_test.go +++ b/ctl/import_test.go @@ -85,7 +85,7 @@ func TestImportCommand_Run(t *testing.T) { } } -// Ensure that the ImportValue path runs (note: we have specifed a value +// Ensure that the ImportValue path runs (note: we have specified a value // for cm.Field. Because the handler doesn't return errors (it sends them // to the logger), we don't get an error returned at `cm.Run()` even though // we haven't setup frame `f` to be RangeEnabled. diff --git a/executor.go b/executor.go index d12aeff1f..9e831c9f0 100644 --- a/executor.go +++ b/executor.go @@ -801,7 +801,7 @@ func (e *Executor) executeFieldRangeSlice(ctx context.Context, index string, c * return NewBitmap(), nil } - // LT[E] and GT[E] should return all not-null if selected range fully encompases valid field range. + // LT[E] and GT[E] should return all not-null if selected range fully encompasses valid field range. if (cond.Op == pql.LT && value > field.Max) || (cond.Op == pql.LTE && value >= field.Max) || (cond.Op == pql.GT && value < field.Min) || (cond.Op == pql.GTE && value <= field.Min) { return frag.FieldNotNull(field.BitDepth()) diff --git a/fragment_test.go b/fragment_test.go index 8a54d3b07..c08b20a21 100644 --- a/fragment_test.go +++ b/fragment_test.go @@ -163,7 +163,7 @@ func TestFragment_SetFieldValue(t *testing.T) { t.Fatal("expected change") } - // Non-existant value. + // Non-existent value. if value, exists, err := f.FieldValue(100, 11); err != nil { t.Fatal(err) } else if value != 0 { diff --git a/gossip/gossip.go b/gossip/gossip.go index 3afdabe31..5a6233a94 100644 --- a/gossip/gossip.go +++ b/gossip/gossip.go @@ -135,7 +135,7 @@ type gossipConfig struct { // newTransport returns a NetTransport based on the memberlist configuration. // It will dynamically bind to a port if conf.BindPort is 0. -// This is useful for test cases where specifiying a port is not reasonable. +// This is useful for test cases where specifying a port is not reasonable. func newTransport(conf *memberlist.Config) (*memberlist.NetTransport, error) { if conf.LogOutput != nil && conf.Logger != nil { return nil, fmt.Errorf("Cannot specify both LogOutput and Logger. Please choose a single log configuration setting.") diff --git a/handler.go b/handler.go index ae04625a6..ca2e6a4ef 100644 --- a/handler.go +++ b/handler.go @@ -995,7 +995,7 @@ func (h *Handler) handleDeleteView(w http.ResponseWriter, r *http.Request) { // Delete the view. if err := f.DeleteView(viewName); err != nil { - // Ingore this error becuase views do not exist on all nodes due to slice distribution. + // Ingore this error because views do not exist on all nodes due to slice distribution. if err != ErrInvalidView { http.Error(w, err.Error(), http.StatusBadRequest) return diff --git a/roaring/roaring.go b/roaring/roaring.go index 3cc537a08..8fb906ef4 100644 --- a/roaring/roaring.go +++ b/roaring/roaring.go @@ -2645,7 +2645,7 @@ func differenceRunRun(a, b *container) *container { for apos < alen && bpos < blen { switch { case alast < bstart: - // current A-run entirely preceeds current B-run: keep full A-run, advance to next A-run + // current A-run entirely precedes current B-run: keep full A-run, advance to next A-run output.runs = append(output.runs, interval16{start: astart, last: alast}) apos++ if apos < alen { @@ -2653,7 +2653,7 @@ func differenceRunRun(a, b *container) *container { alast = a.runs[apos].last } case blast < astart: - // current B-run entirely preceeds current A-run: advance to next B-run + // current B-run entirely precedes current A-run: advance to next B-run bpos++ if bpos < blen { bstart = b.runs[bpos].start diff --git a/roaring/roaring_test.go b/roaring/roaring_test.go index 6378ee67f..7c5539b34 100644 --- a/roaring/roaring_test.go +++ b/roaring/roaring_test.go @@ -291,7 +291,7 @@ func TestBitmap_Max(t *testing.T) { } } -// Ensure CountRange is correct even if rangekey is prior to inital container. +// Ensure CountRange is correct even if rangekey is prior to initial container. func TestBitmap_BitmapCountRangeEdgeCase(t *testing.T) { s := uint64(2009 * 1048576) e := uint64(2010 * 1048576) diff --git a/statsd/statsd.go b/statsd/statsd.go index d46dd637f..c7c072ae3 100644 --- a/statsd/statsd.go +++ b/statsd/statsd.go @@ -24,7 +24,7 @@ import ( "github.com/pilosa/pilosa" ) -// StatsD protocal wrapper using the DataDog library that added Tags to the StatsD protocal +// StatsD protocol wrapper using the DataDog library that added Tags to the StatsD protocol // statsD defailt host is "127.0.0.1:8125" const ( From 1b4445a00d2c25d1192a06ce01e55c3f47074f0d Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Fri, 9 Mar 2018 15:27:00 -0600 Subject: [PATCH 14/34] Fix formatting --- docs/configuration.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index 692a0a17a..bc9bd75b0 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -218,7 +218,8 @@ Any flag that has a value that is a comma separated list on the command line bec [profile] cpu-time = "30s" ``` -##### Metric Service + +#### Metric Service * Description: Which stats service to use. Choose from [statsd, expvar]. * Flag: `--metric.service=statsd` * Env: `PILOSA_METRIC_SERVICE=statsd' @@ -229,7 +230,7 @@ Any flag that has a value that is a comma separated list on the command line bec service = “statsd” ``` -##### Metric Host +#### Metric Host * Description: Address of the StatsD service host. * Flag: `--metric.host=localhost:8125` * Env: `PILOSA_METRIC_HOST=localhost:8125' @@ -240,7 +241,7 @@ Any flag that has a value that is a comma separated list on the command line bec host = "localhost:8125" ``` -##### Metric Poll Interval +#### Metric Poll Interval * Description: Polling interval for runtime metrics. * Flag: `metric.poll-interval=”0m15s”` @@ -252,7 +253,7 @@ Any flag that has a value that is a comma separated list on the command line bec poll-interval = "0m15s" ``` -##### Metric Diagnostics +#### Metric Diagnostics * Description: Enable diagnostic reporting. To disable diagnostics set to false. * Flag: `metric.diagnostics` @@ -265,7 +266,7 @@ Any flag that has a value that is a comma separated list on the command line bec ``` -##### TLS Certificate +#### TLS Certificate * Description: Path to the TLS certificate to use for serving HTTPS. Usually has one of`.crt` or `.pem` extensions. * Flag: `tls.certificate=/srv/pilosa/certs/server.crt` @@ -277,7 +278,7 @@ Any flag that has a value that is a comma separated list on the command line bec certificate = "/srv/pilosa/certs/server.crt" ``` -##### TLS Certificate Key +#### TLS Certificate Key * Description: Path to the TLS certificate key to use for serving HTTPS. Usually has the `.key` extension. * Flag: `tls.key=/srv/pilosa/certs/server.key` @@ -289,7 +290,7 @@ Any flag that has a value that is a comma separated list on the command line bec key = "/srv/pilosa/certs/server.key" ``` -##### TLS Skip Verify +#### TLS Skip Verify * Description: Disables verification for checking TLS certificates. This configuration item is mainly useful for using self-signed certificates for a Pilosa cluster. Do not use in production since it makes man-in-the-middle attacks trivial. * Flag: `tls.skip-verify` From 8cfe7583cec6426b060cf2b645daf5238c3b19cc Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Fri, 9 Mar 2018 15:27:21 -0600 Subject: [PATCH 15/34] Fix broken anchor links --- docs/administration.md | 6 +++--- docs/glossary.md | 10 +++++----- docs/tutorials.md | 4 ++-- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/administration.md b/docs/administration.md index bf3da9626..d3548553f 100644 --- a/docs/administration.md +++ b/docs/administration.md @@ -150,9 +150,9 @@ You can opt-out of the Pilosa diagnostics reporting by setting either the comman Pilosa can be configured to emit metrics pertaining to its internal processes in one of two formats: Expvar or StatsD. Metric recording is disabled by default. The metrics configuration options are: - - [Host](../configuration#metrics-host): specify host that receives metric events - - [Poll Interval](../configuration#metrics-poll-interval): specify polling interval for runtime metrics - - [Service](../configuration#metrics-service): declare type StatsD or Expvar + - [Host](../configuration#metric-host): specify host that receives metric events + - [Poll Interval](../configuration#metric-poll-interval): specify polling interval for runtime metrics + - [Service](../configuration#metric-service): declare type StatsD or Expvar #### Tags StatsD Tags adhere to the DataDog format (key:value), and we tag the following: diff --git a/docs/glossary.md b/docs/glossary.md index f6fa29478..6ea0549ef 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -14,13 +14,13 @@ nav = [] [Bitmap](../data-model/#overview): The on-disk and in-memory representation of a [row](#row). Implemented with [Roaring](#roaring-bitmap). `Bitmap` is also the basic [PQL](#pql) query for reading a Bitmap. -[BSI](../data-model/#bsi-range-encoding) Bit-sliced indexing is the method Pilosa uses to represent multi-bit integers. Integer values are stored in [fields](#field), and can be used for [Range](#range) and [Sum](#sum) queries. +[BSI](../data-model/#bsi-range-encoding) Bit-sliced indexing is the method Pilosa uses to represent multi-bit integers. Integer values are stored in [fields](#field), and can be used for [Range](#range-bsi) and [Sum](#sum) queries. Cluster: A cluster consists of one or more [nodes](#node) which share a cluster configuration. The cluster also defines how data is [replicated](#replica) throughout and how internode communication is coordinated. Pilosa does not have a leader node, all data is evenly distributed, and any node can respond to queries. [Column](../data-model/#column): Columns are the fundamental horizontal data axis within Pilosa. Columns are global to all [frames](#frame) within an [index](#index). -[Field](../data-model/#bsi-range-encoding): A group of rows used to store integer values with [BSI](#bsi), for use in [Range](#range) and [Sum](#sum) queries. +[Field](../data-model/#bsi-range-encoding): A group of rows used to store integer values with [BSI](#bsi), for use in [Range](#range-bsi) and [Sum](#sum) queries. Fragment: A Fragment is the intersection of a [frame](#frame) and a [slice](#slice) in an [index](#index). @@ -30,7 +30,7 @@ nav = [] [Jump Consistent Hash](https://arxiv.org/pdf/1406.2294v1.pdf): A fast, minimal memory, consistent hash algorithm that evenly distributes the workload even when the number of buckets changes. -MaxSlice: The total number of [slices](#slice) allocated to handle the current set of [columns](#columns). This value is important for all [nodes](#node) to efficiently distribute queries. +MaxSlice: The total number of [slices](#slice) allocated to handle the current set of [columns](#column). This value is important for all [nodes](#node) to efficiently distribute queries. Node: An individual running instance of Pilosa server which belongs to a [cluster](#cluster). @@ -40,9 +40,9 @@ nav = [] [Protobuf](https://developers.google.com/protocol-buffers/): Protocol Buffers is a binary serialization format which Pilosa uses for internal messages, and can be used by clients as an alternative to JSON. -[Range](../query-language/#range):: A [PQL](#pql) query that returns bits based on comparison to timestamps, set according to the [time quantum](#time-quantum). +[Range](../query-language/#range-queries):: A [PQL](#pql) query that returns bits based on comparison to timestamps, set according to the [time quantum](#time-quantum). -[Range (BSI)](../query-language/#range-bsi):: A [PQL](#pql) query that returns bits based on comparison to integers stored in [BSI](#bsi) [fields](#field). +[Range (BSI)](../query-language/#range-bsi):: A [PQL](#pql) query that returns bits based on comparison to integers stored in [BSI](#bsi) [fields](#field). [Replica](../configuration/#cluster-replicas): A copy of a [fragment](#fragment) on a different [node](#node) than the original. The `cluster.replicas` configuration parameter determines how many replicas of a fragment exist in the cluster. This includes the original, so a value of 1 means no extra copies are made. diff --git a/docs/tutorials.md b/docs/tutorials.md index 9e6364cf0..f9b4a1494 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -24,7 +24,7 @@ This tutorial assumes that you are using a UNIX-like system, such as Linux or Ma #### Installing Pilosa and Creating the Directory Structure -If you haven't already done so, install Pilosa server on your computer. For Linux and WSL (Windows Subsystem for Linux) use the [Installing on Linux](https://www.pilosa.com/docs/latest/installation/#installing-on-linux) instructions. For MacOS use the [Installing on MacOS](https://www.pilosa.com/docs/latest/installation/#installing-on-macos). We do not support precompiled releases for other platforms, but you can always compile it yourself from source. See [Build from Source](https://www.pilosa.com/docs/latest/installation/#build-from-source). +If you haven't already done so, install Pilosa server on your computer. For Linux and WSL (Windows Subsystem for Linux) use the [Installing on Linux](../installation/#installing-on-linux) instructions. For MacOS use the [Installing on MacOS](../installation/#installing-on-macos). We do not support precompiled releases for other platforms, but you can always compile it yourself from source. See [Build from Source](../installation/#build-from-source). After installing Pilosa, you may have to add it to your `$PATH`. Check that you can run Pilosa from the command line: ``` @@ -134,7 +134,7 @@ Here is some explanation of the configuration items: * `data-dir` points to the directory where the Pilosa server writes its data. If it doesn't exist, the server will create it. * `bind` is the address to which the server listens for incoming requests. The address is composed of three parts: scheme, host, and port. The default scheme is `http` so we explicitly specify `https` to use the HTTPS protocol for communication between nodes. -* `[cluster]` section contains the settings for a cluster. `hosts` field is the most important, which contains the list of addresses of other nodes. See [Cluster Configuration](https://www.pilosa.com/docs/latest/configuration/#cluster-hosts) for other settings. +* `[cluster]` section contains the settings for a cluster. `hosts` field is the most important, which contains the list of addresses of other nodes. See [Cluster Configuration](../configuration/#cluster-hosts) for other settings. * `[tls]` section contains the TLS settings, including the path to the SSL certificate and the corresponding key. Set `skip-verify` to `true` in order to disable host name verification and other security measures. Do not set `skip-verify` to `true` on production servers. * `[gossip]` section contains settings for the Gossip protocol. `seed` is the host and port for the main gossip node which coordinates other nodes. The `port` setting is the gossip listen address for the node. It should be different for each node, if the cluster is running on the same computer, otherwise you can set it to the same value. Finally, the `key` points to the gossip encryption key we created before. From 00bbdc1107c52f33f62eccfcf5e3fd4364d0a8c5 Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Tue, 13 Mar 2018 14:50:09 -0500 Subject: [PATCH 16/34] Move SetFieldValue to write section --- docs/query-language.md | 48 +++++++++++++++++++++--------------------- 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/docs/query-language.md b/docs/query-language.md index 105893e66..f14be7ca6 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -187,6 +187,30 @@ ClearBit(frame="stargazer", columnID=10, rowID=1) Remove relationship between the stargazer in row 1 and the repository in column 10 from the stargazer frame. +#### SetFieldValue + +**Spec:** + +``` +SetFieldValue(, , ) +``` + +**Description:** + +`SetFieldValue` assigns an integer value with the specified field name to the `columnID` in the given `frame`. + +**Result Type:** null + +SetFieldValue returns `null` upon success. + +**Examples:** + +Set the number of pull requests of repository 10. +``` +SetFieldValue(col=10, frame="stats", pullrequests=2) +``` + + ### Read Operations #### Bitmap @@ -527,27 +551,3 @@ Sum(frame="stats", field="diskusage") Return `{"sum":10,"count":3}` * Result is the size of all repositories in kilobytes, plus the number of repositories. - - -#### SetFieldValue - -**Spec:** - -``` -SetFieldValue(, , ) -``` - -**Description:** - -`SetFieldValue` assigns an integer value with the specified field name to the `columnID` in the given `frame`. - -**Result Type:** null - -SetFieldValue returns `null` upon success. - -**Examples:** - -Set the number of pull requests of repository 10. -``` -SetFieldValue(col=10, frame="stats", pullrequests=2) -``` From fae429f971bc79e212c5a7cf6e9f0cdbbd1170a6 Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Thu, 15 Mar 2018 10:57:19 -0500 Subject: [PATCH 17/34] Remove outdated column labels --- docs/data-model.md | 12 ++++++------ docs/query-language.md | 2 +- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/data-model.md b/docs/data-model.md index 9ab684203..6a57a96a1 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -119,12 +119,12 @@ Internally Pilosa stores each BSI `field` as a `view` within a `frame`. The 'row For example, the following `SetFieldValue()` queries will result in the data described in the illustration below: ``` -SetFieldValue(col=1, frame="A", field0=1) -SetFieldValue(col=2, frame="A", field0=2) -SetFieldValue(col=3, frame="A", field0=3) -SetFieldValue(col=4, frame="A", field0=7) -SetFieldValue(col=2, frame="A", field1=1) -SetFieldValue(col=3, frame="A", field1=6) +SetFieldValue(columnID=1, frame="A", field0=1) +SetFieldValue(columnID=2, frame="A", field0=2) +SetFieldValue(columnID=3, frame="A", field0=3) +SetFieldValue(columnID=4, frame="A", field0=7) +SetFieldValue(columnID=2, frame="A", field1=1) +SetFieldValue(columnID=3, frame="A", field1=6) ``` ![BSI frame diagram](/img/docs/frame-bsi.svg) diff --git a/docs/query-language.md b/docs/query-language.md index f14be7ca6..d612328c6 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -207,7 +207,7 @@ SetFieldValue returns `null` upon success. Set the number of pull requests of repository 10. ``` -SetFieldValue(col=10, frame="stats", pullrequests=2) +SetFieldValue(columnID=10, frame="stats", pullrequests=2) ``` From 1452bf24d640b2b95f1596e359506ea888b381c7 Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Thu, 15 Mar 2018 11:07:07 -0500 Subject: [PATCH 18/34] Clarify Sum description --- docs/query-language.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/query-language.md b/docs/query-language.md index d612328c6..83cb946e3 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -537,7 +537,7 @@ Sum([BITMAP_CALL], , ) **Description:** -Returns the count and computed sum of all BSI integer values across the `field` in this `frame`. The optional `Bitmap` call filters the bits used in this computation. +Returns the count and computed sum of all BSI integer values in the `field` in this `frame`. If the optional `Bitmap` call is supplied, columns with set bits are summed, otherwise the sum is across all columns. **Result Type:** object with the computed sum and count of the bitmap field. From 37015e32a5eddb80a812f9ac9a7f51c315ad084a Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Mon, 19 Mar 2018 14:25:36 -0500 Subject: [PATCH 19/34] Ensure internal links end with slash --- docs/administration.md | 6 +++--- docs/api-reference.md | 14 +++++++------- docs/examples.md | 4 ++-- docs/getting-started.md | 4 ++-- docs/glossary.md | 2 +- docs/introduction.md | 2 +- docs/query-language.md | 8 ++++---- 7 files changed, 20 insertions(+), 20 deletions(-) diff --git a/docs/administration.md b/docs/administration.md index d3548553f..44d3fcbb6 100644 --- a/docs/administration.md +++ b/docs/administration.md @@ -150,9 +150,9 @@ You can opt-out of the Pilosa diagnostics reporting by setting either the comman Pilosa can be configured to emit metrics pertaining to its internal processes in one of two formats: Expvar or StatsD. Metric recording is disabled by default. The metrics configuration options are: - - [Host](../configuration#metric-host): specify host that receives metric events - - [Poll Interval](../configuration#metric-poll-interval): specify polling interval for runtime metrics - - [Service](../configuration#metric-service): declare type StatsD or Expvar + - [Host](../configuration/#metric-host): specify host that receives metric events + - [Poll Interval](../configuration/#metric-poll-interval): specify polling interval for runtime metrics + - [Service](../configuration/#metric-service): declare type StatsD or Expvar #### Tags StatsD Tags adhere to the DataDog format (key:value), and we tag the following: diff --git a/docs/api-reference.md b/docs/api-reference.md index 5a28313b1..1dbc59865 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -78,7 +78,7 @@ In order to send protobuf binaries in the request and response, set `Content-Typ The response doesn't include column attributes by default. To return them, set the `columnAttrs` query argument to `true`. -The query is executed for all [slices](../data-model#slice) by default. To use specified slices only, set the `slices` query argument to a comma-separated list of slice indices. +The query is executed for all [slices](../data-model/#slice) by default. To use specified slices only, set the `slices` query argument to a comma-separated list of slice indices. ``` request curl "localhost:10101/index/user/query?columnAttrs=true&slices=0,1" \ @@ -102,9 +102,9 @@ Creates a frame in the given index with the given name. The request payload is in JSON, and may contain the `options` field. The `options` field is a JSON object which may contain the following fields: -* `timeQuantum` (string): [Time Quantum](../data-model#time-quantum) for this frame. -* `inverseEnabled` (boolean): Enables [the inverted view](../data-model#inverse) for this frame if `true`. -* `cacheType` (string): [ranked](../data-model#ranked) or [LRU](../data-model#lru) caching on this frame. Default is `lru`. +* `timeQuantum` (string): [Time Quantum](../data-model/#time-quantum) for this frame. +* `inverseEnabled` (boolean): Enables [the inverted view](../data-model/#inverse) for this frame if `true`. +* `cacheType` (string): [ranked](../data-model/#ranked) or [LRU](../data-model/#lru) caching on this frame. Default is `lru`. * `cacheSize` (int): Number of rows to keep in the cache. Default 50,000. * `rangeEnabled` (boolean): Enables range-encoded fields in this frame. * `fields` (array): List of range-encoded fields. @@ -205,9 +205,9 @@ Creates an input definition in the given index with the given name. The request payload is JSON, and it must contain the fields `frames` and `fields`. `frames` is an array of frames used within this input definition. Each frame must contain a `name` and may contain the following options: -* `timeQuantum` (string): [Time Quantum](../data-model#time-quantum) for this frame. -* `inverseEnabled` (boolean): Enables [the inverted view](../data-model#inverse) for this frame if `true`. -* `cacheType` (string): [ranked](../data-model#ranked) or [LRU](../data-model#lru) caching on this frame. Default is `lru`. +* `timeQuantum` (string): [Time Quantum](../data-model/#time-quantum) for this frame. +* `inverseEnabled` (boolean): Enables [the inverted view](../data-model/#inverse) for this frame if `true`. +* `cacheType` (string): [ranked](../data-model/#ranked) or [LRU](../data-model/#lru) caching on this frame. Default is `lru`. * `cacheSize` (int): Number of rows to keep in the cache. Default 50,000. The `fields` array contains a series of JSON objects describing how to process each field received in the input data. Each `field` object must contain a `name` which maps to the source JSON field name. One field must be defined at the `primaryKey`. The `primarykey` source field name must equal the column label for the `Index`, and its value must be an unsigned integer which maps directly to a columnID in Pilosa. diff --git a/docs/examples.md b/docs/examples.md index 12dd69617..589539081 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -17,7 +17,7 @@ New York City released an extremely detailed data set of over 1 billion taxi rid Transportation in general is a compelling use case for Pilosa as it often involves multiple disparate data sources, as well as high rate, real time, and extremely large amounts of data (particularly if one wants to draw reasonable conclusions). -We've written a tool to help import the NYC taxi data into Pilosa - this tool is part of the [PDK](../pdk) (Pilosa Development Kit), and takes advantage of a number of reusable modules that may help you import other data as well. Follow along and we'll explain the whole process step by step. +We've written a tool to help import the NYC taxi data into Pilosa - this tool is part of the [PDK](../pdk/) (Pilosa Development Kit), and takes advantage of a number of reusable modules that may help you import other data as well. Follow along and we'll explain the whole process step by step. After initial setup, the PDK import tool does everything we need to define a Pilosa schema, map data to bitmaps accordingly, and import it into Pilosa. @@ -263,7 +263,7 @@ python import_from_sdf.py -p -file id_fingerprint.csv ``` -First, follow the instruction in the [getting started](../getting-started) guide to run a Pilosa server. Then create the indexes and frames according to the schemas outlined in the Data Model section above. +First, follow the instruction in the [getting started](../getting-started/) guide to run a Pilosa server. Then create the indexes and frames according to the schemas outlined in the Data Model section above. The option cacheSize should be set as amount of chembl_id to calculate effectively for the whole data set, so we need to calculate amount of chembl_id. We have total 1678393 chembl_id (it will displayed after import_from_sdf.py script running), then the cacheSize should be >= 1678393 ``` curl localhost:10101/index/mole \ diff --git a/docs/getting-started.md b/docs/getting-started.md index 6a9eb54be..0b773ea2c 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -20,7 +20,7 @@ Any HTTP tool can be used to interact with the Pilosa server. The examples in th ### Starting Pilosa -Follow the steps in the [Install](../installation) document to install Pilosa. +Follow the steps in the [Install](../installation/) document to install Pilosa. Execute the following in a terminal to run Pilosa with the default configuration (Pilosa will be available at `localhost:10101`): ``` pilosa server @@ -42,7 +42,7 @@ curl localhost:10101/status In order to better understand Pilosa's capabilities, we will create a sample project called "Star Trace" containing information about the top 1,000 most recently updated Github repositories which have "go" in their name. The Star Trace index will include data points such as programming language, tags, and stargazers—people who have starred a project. -Although Pilosa doesn't keep the data in a tabular format, we still use the terms "columns" and "rows" when describing the data model. We put the primary objects in columns, and the properties of those objects in rows. For example, the Star Trace project will contain an index called "repository" which contains columns representing Github repositories, and rows representing properties like programming languages and tags. We can better organize the rows by grouping them into sets called Frames. So the "repository" index might have a "languages" frame as well as a "tags" frame. You can learn more about indexes and frames in the [Data Model](../data-model) section of the documentation. +Although Pilosa doesn't keep the data in a tabular format, we still use the terms "columns" and "rows" when describing the data model. We put the primary objects in columns, and the properties of those objects in rows. For example, the Star Trace project will contain an index called "repository" which contains columns representing Github repositories, and rows representing properties like programming languages and tags. We can better organize the rows by grouping them into sets called Frames. So the "repository" index might have a "languages" frame as well as a "tags" frame. You can learn more about indexes and frames in the [Data Model](../data-model/) section of the documentation. #### Create the Schema diff --git a/docs/glossary.md b/docs/glossary.md index 6ea0549ef..91664eb5c 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -62,6 +62,6 @@ nav = [] [TOML](https://github.com/toml-lang/toml): the language used for Pilosa's [configuration file](../configuration). -[TopN](../query-language#topn): A [PQL](#pql) query that returns a list of `RowID`s, sorted by the count of [bits](#bit) set in the [row](#row), within a specified [frame](#frame). +[TopN](../query-language/#topn): A [PQL](#pql) query that returns a list of `RowID`s, sorted by the count of [bits](#bit) set in the [row](#row), within a specified [frame](#frame). [View](../data-model/#view): Views separate the different data layouts within a [Frame](#frame). The two primary views are standard and inverse which represent the typical [row](#row)/[column](#column) data and its inverse respectively (an [inverted index](https://en.wikipedia.org/wiki/Inverted_index), or a matrix transpose). Time based frame views are automatically generated for each [time quantum](#time-quantum). Views are internally managed by Pilosa, and never exposed directly via the API. This simplifies the functional interface by separating it from the physical data representation. diff --git a/docs/introduction.md b/docs/introduction.md index 7e7049c6c..d2ebb4662 100644 --- a/docs/introduction.md +++ b/docs/introduction.md @@ -16,4 +16,4 @@ It is designed primarly for speed and horizontal scalability. If you have data w "What attributes are the most common?", "Which objects have these specific attributes?", "What groups of attributes often appear together?" Pilosa is designed to answer these types of queries in real time, suitable for use with high rate data streams, or to power a user interface. -Once you have Pilosa [installed](../installation), the [getting started](../getting-started) guide will show you the basics of interacting with Pilosa and give you some pointers for deeper exploration. +Once you have Pilosa [installed](../installation/), the [getting started](../getting-started/) guide will show you the basics of interacting with Pilosa and give you some pointers for deeper exploration. diff --git a/docs/query-language.md b/docs/query-language.md index 83cb946e3..bea99804c 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -13,7 +13,7 @@ nav = [ ### Overview -This section will provide a detailed reference and examples for the Pilosa Query Language (PQL). All PQL queries operate on a single [index](../glossary#index) and are passed to Pilosa through the `/index/INDEX_NAME/query` endpoint. You may pass multiple PQL queries in a single request by simply concatenating the queries together - a space is not needed. The results format is always: +This section will provide a detailed reference and examples for the Pilosa Query Language (PQL). All PQL queries operate on a single [index](../glossary/#index) and are passed to Pilosa through the `/index/INDEX_NAME/query` endpoint. You may pass multiple PQL queries in a single request by simply concatenating the queries together - a space is not needed. The results format is always: ``` {"results":[...]} @@ -45,7 +45,7 @@ curl localhost:10101/index/repository/query \ #### Arguments and Types -* `frame` The frame specifies on which Pilosa [frame](../glossary#frame) the query will operate. Valid frame names are lower case strings; they start with an alphanumeric character, and contain only alphanumeric characters and `_-`. They must be 64 characters or less in length. +* `frame` The frame specifies on which Pilosa [frame](../glossary/#frame) the query will operate. Valid frame names are lower case strings; they start with an alphanumeric character, and contain only alphanumeric characters and `_-`. They must be 64 characters or less in length. * `ROW_LABEL` The default row label is `rowID`, changing the default is deprecated. * `COL_LABEL` The default column label is `columnID`, changing the default is deprecated. * `TIMESTAMP` This is a timestamp in quotes with the following format `"YYYY-MM-DDTHH:MM"` (e.g. "2006-01-02T15:04") @@ -118,7 +118,7 @@ SetRowAttrs queries always return `null` upon success. SetRowAttrs(frame="stargazer", rowID=10, username="mrpi", active=true) ``` -Set username value and active status for user 10. These are arbitrary key/value pairs which have no meaning to Pilosa. You can see the attributes you've set on a row with a [Bitmap](../query-language#bitmap) query like so `Bitmap(frame="stargazer", stargazer_id=10)`. +Set username value and active status for user 10. These are arbitrary key/value pairs which have no meaning to Pilosa. You can see the attributes you've set on a row with a [Bitmap](../query-language/#bitmap) query like so `Bitmap(frame="stargazer", stargazer_id=10)`. ``` SetRowAttrs(frame="stargazer", rowID=10, username=null) @@ -150,7 +150,7 @@ SetColumnAttrs queries always return `null` upon success. Setting a value of `nu SetColumnAttrs(columnID=10, stars=123, url="http://projects.pilosa.com/10", active=true) ``` -Set url value and active status for project 10. These are arbitrary key/value pairs which have no meaning to Pilosa. You can see the attributes you've set on a column with a [Bitmap](../query-language#bitmap) query like so `Bitmap(frame="stargazer", columnID=10)`. +Set url value and active status for project 10. These are arbitrary key/value pairs which have no meaning to Pilosa. You can see the attributes you've set on a column with a [Bitmap](../query-language/#bitmap) query like so `Bitmap(frame="stargazer", columnID=10)`. ``` SetColumnAttrs(columnID=10, url=null) From bd76bfc25bfd5c1ad4915d4fdf721d8670b4db39 Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Mon, 19 Mar 2018 14:49:41 -0500 Subject: [PATCH 20/34] Fix a few more internal links --- docs/client-libraries.md | 6 +++--- docs/examples.md | 2 +- docs/query-language.md | 2 +- docs/webui.md | 4 ++-- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/client-libraries.md b/docs/client-libraries.md index a328318ee..1c360741c 100644 --- a/docs/client-libraries.md +++ b/docs/client-libraries.md @@ -16,7 +16,7 @@ This section contains example code for client libraries in several languages. Pl You can find the Go client library for Pilosa at our [Go Pilosa Repository](https://github.com/pilosa/go-pilosa). Check out its [README](https://github.com/pilosa/go-pilosa/blob/master/README.md) for more information and installation instructions. -We are going to use the index you have created in the [Getting Started](../getting-started) section. Before carrying on, make sure that example index is created, sample stargazer data is imported and Pilosa server is running on the default address: `http://localhost:10101`. +We are going to use the index you have created in the [Getting Started](../getting-started/) section. Before carrying on, make sure that example index is created, sample stargazer data is imported and Pilosa server is running on the default address: `http://localhost:10101`. Error handling has been omitted in the example below for brevity. @@ -98,7 +98,7 @@ func main() { You can find the Python client library for Pilosa at our [Python Pilosa Repository](https://github.com/pilosa/python-pilosa). Check out its [README](https://github.com/pilosa/python-pilosa/blob/master/README.md) for more information and installation instructions. -We are going to use the index you have created in the [Getting Started](../getting-started) section. Before carrying on, make sure that example index is created, sample stargazer data is imported and Pilosa server is running on the default address: `http://localhost:10101`. +We are going to use the index you have created in the [Getting Started](../getting-started/) section. Before carrying on, make sure that example index is created, sample stargazer data is imported and Pilosa server is running on the default address: `http://localhost:10101`. Error handling has been omitted in the example below for brevity. @@ -171,7 +171,7 @@ client.query(stargazer.setbit(99999, 77777)) You can find the Java client library for Pilosa at our [Java Pilosa Repository](https://github.com/pilosa/java-pilosa). Check out its [README](https://github.com/pilosa/java-pilosa/blob/master/README.md) for more information and installation instructions. -We are going to use the index you have created in the [Getting Started](../getting-started) section. Before carrying on, make sure that example index is created, sample stargazer data is imported and Pilosa server is running on the default address: `http://localhost:10101`. +We are going to use the index you have created in the [Getting Started](../getting-started/) section. Before carrying on, make sure that example index is created, sample stargazer data is imported and Pilosa server is running on the default address: `http://localhost:10101`. Error handling has been omitted in the example below for brevity. diff --git a/docs/examples.md b/docs/examples.md index 589539081..cf9885a3f 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -163,7 +163,7 @@ durm := pdk.CustomMapper{ #### Import process -After designing this schema and mapping, we capture it in a JSON definition file that can be read by the PDK import tool. Running `pdk taxi` runs the import based on the information in this file. See [PDK](../pdk) for more details on this process. +After designing this schema and mapping, we capture it in a JSON definition file that can be read by the PDK import tool. Running `pdk taxi` runs the import based on the information in this file. See [PDK](../pdk/) for more details on this process. #### Queries diff --git a/docs/query-language.md b/docs/query-language.md index bea99804c..0ccca3a80 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -31,7 +31,7 @@ The default row label is `rowID`, and the default column label is `columnID`. Ch ##### Examples -Before running any of the example queries below, follow the instructions in the [Getting Started](../getting-started) section to set up an index, frames, and populate them with some data. +Before running any of the example queries below, follow the instructions in the [Getting Started](../getting-started/) section to set up an index, frames, and populate them with some data. The examples just show the PQL quer(ies) needed - to run the query `SetBit(frame="stargazer", columnID=10, rowID=1)` against a server using curl, you would: ``` request diff --git a/docs/webui.md b/docs/webui.md index 86874807b..97fa2ce8f 100644 --- a/docs/webui.md +++ b/docs/webui.md @@ -14,7 +14,7 @@ This can be used for constructing queries and viewing the cluster status. ### Console -The [Console view](http://localhost:10101/#console) allows you to enter [PQL](../query-language) queries and run them against your locally running server. First you must select an Index with the Select index dropdown. +The [Console view](http://localhost:10101/#console) allows you to enter [PQL](../query-language/) queries and run them against your locally running server. First you must select an Index with the Select index dropdown. Each query's result will be displayed in the Output section along with the query time. @@ -31,7 +31,7 @@ In addition to standard PQL, the console supports a few special commands, prefix - `:create frame ` - `:delete frame ` -Frame creation also supports options like `timeQuantum` or `inverseEnabled`. When creating a new frame, add options by using the keys documented in [API reference](../api-reference). +Frame creation also supports options like `timeQuantum` or `inverseEnabled`. When creating a new frame, add options by using the keys documented in [API reference](../api-reference/). - `:create index timeQuantum=YM` - `:create frame inverseEnabled=true cacheSize=10000` From fa66de0461cd809ffec2f72d815cd56750321521 Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Mon, 19 Mar 2018 16:13:54 -0500 Subject: [PATCH 21/34] Fix one more link --- docs/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/glossary.md b/docs/glossary.md index 91664eb5c..612c65dd4 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -60,7 +60,7 @@ nav = [] [Time quantum](../data-model/#time-quantum): Defines the granularity to be used for time [Range](#range) queries. -[TOML](https://github.com/toml-lang/toml): the language used for Pilosa's [configuration file](../configuration). +[TOML](https://github.com/toml-lang/toml): the language used for Pilosa's [configuration file](../configuration/). [TopN](../query-language/#topn): A [PQL](#pql) query that returns a list of `RowID`s, sorted by the count of [bits](#bit) set in the [row](#row), within a specified [frame](#frame). From 02a822bcd389a3b28e3206523f4a5a686ff2fa47 Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Tue, 20 Mar 2018 12:44:39 -0500 Subject: [PATCH 22/34] Merge README-dev.md into CONTRIBUTING.md --- CONTRIBUTING.md | 73 +++++++++++++++++++++++++++++++++++++++++-------- README-dev.md | 69 ---------------------------------------------- 2 files changed, 61 insertions(+), 81 deletions(-) delete mode 100644 README-dev.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d15c42ed7..10b19eae1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,30 +2,79 @@ ## Reporting a bug -If you have discovered a bug and don't see it in the [github issue tracker][5], [open a new issue][1] +If you have discovered a bug and don't see it in the [github issue tracker][5], [open a new issue][1]. ## Submitting a feature request -Feature requests are managed in Github issues. New features typically go through a [Proposal Process][4] +Feature requests are managed in Github issues, organized with [Zenhub](https://www.zenhub.com/), which is publicly available as a browser extension. New features typically go through a [Proposal Process][4] which starts by [opening a new issue][1] that describes the new feature proposal. -## Submitting code changes +## Making code contributions Before you start working on new features, you should [open a new issue][1] to let others know what -you're doing before you start working, otherwise you run the risk of duplicating effort. This also +you're doing, otherwise you run the risk of duplicating effort. This also gives others an opportunity to provide input for your feature. If you want to help but you aren't sure where to start, check out our [github label for low-effort issues][6]. -- Fork the [Pilosa repository][2] and then clone your fork: - ```shell - git clone git@github.com:/pilosa.git +### Development Environment + +- Ensure you have a recent version of [Go](https://golang.org/dl/) installed. Pilosa generally supports the current and previous minor versions; check our [travis file](../.travis.yml) for the most up-to-date information. + +- Make sure `$GOPATH` environment variable points to your Go working directory and `$PATH` incudes `$GOPATH/bin`. + +- Fork the [Pilosa repository][2] to your own account. + +- Create a directory (note that we use `github.com/pilosa`, NOT `github.com/USER`) and clone your own Pilosa repo: + + ```sh + mkdir -p ${GOPATH}/src/github.com/pilosa && cd $_ + git clone git@github.com:${USER}/pilosa.git + ``` + +- `cd` to your pilosa directory: + + ```sh + cd ${GOPATH}/src/github.com/pilosa/pilosa + ``` + +- Install `dep` to manage dependencies: + + ```sh + go get -u github.com/golang/dep/cmd/dep + ``` + +- Install Pilosa command line tools: + + ```sh + make install + # or: + # dep ensure && go install github.com/pilosa/pilosa/cmd/... + ``` + + Running `pilosa` should now run a Pilosa instance. + +- In order to sync your fork with upstream Pilosa repo, add an *upstream* to your repo: + + ```sh + cd ${GOPATH}/src/github.com/pilosa/pilosa + git remote add upstream git@github.com:pilosa/pilosa.git + ``` + +### Submitting code changes + +- Before starting to work on a task, sync your branch with the upstream: + + ```sh + git fetch upstream + git checkout master + git merge upstream/master ``` - Create a local feature branch: - ```shell + ```sh git checkout -b something-amazing ``` @@ -33,13 +82,13 @@ If you want to help but you aren't sure where to start, check out our [github la - Make sure that you've written tests for your new feature, and then run the tests: - ```shell + ```sh make test ``` - Verify that your pull request is applied to the latest version of code on github: - ```shell + ```sh git remote add upstream git@github.com:pilosa/pilosa.git git fetch upstream git rebase -i upstream/master @@ -47,7 +96,7 @@ If you want to help but you aren't sure where to start, check out our [github la - Push to your fork: - ```shell + ```sh git push -u something-amazing ``` @@ -59,4 +108,4 @@ If you want to help but you aren't sure where to start, check out our [github la [3]: https://github.com/pilosa/pilosa/compare/ [4]: https://github.com/pilosa/general/blob/master/proposal.md [5]: https://github.com/pilosa/pilosa/issues -[6]: https://github.com/pilosa/pilosa/issues?q=is%3Aopen+is%3Aissue+label%3Anewcomer \ No newline at end of file +[6]: https://github.com/pilosa/pilosa/issues?q=is%3Aopen+is%3Aissue+label%3Anewcomer diff --git a/README-dev.md b/README-dev.md deleted file mode 100644 index 809befbc3..000000000 --- a/README-dev.md +++ /dev/null @@ -1,69 +0,0 @@ - -Development Environment -======================= - -Install Go versions 1.6.2+ or 1.7 for your platform. - -Fork `github.com/pilosa/pilosa` to your own account. The forked repo will be private. - -Make sure `$GOPATH` environment variable points to your Go working directory and `$PATH` incudes `$GOPATH/bin`. - -Create a directory (note that we use `github.com/pilosa`, NOT `github.com/USER`) and clone your own Pilosa repo: - -```sh -mkdir -p ${GOPATH}/src/github.com/pilosa && cd $_ -git clone git@github.com:${USER}/pilosa.git -``` - -`cd` to your pilosa directory: - -```sh -cd ${GOPATH}/src/github.com/pilosa/pilosa -``` - -Install `dep` to manage dependencies: - -```sh -go get -u github.com/golang/dep/cmd/dep -``` - -Install Pilosa command line tools: - -```sh -make install -# or: -# dep ensure && go install github.com/pilosa/pilosa/cmd/... -``` - -Running `pilosa` should now run a Pilosa instance. - -In order to sync your fork with upstream Pilosa repo, add an *upstream* to your repo: - -```sh -cd ${GOPATH}/src/github.com/pilosa/pilosa -git remote add upstream git@github.com:pilosa/pilosa.git -``` - -Before starting to work on a task, sync your branch with the upstream: - -```sh -git fetch upstream -git checkout master -git merge upstream/master -``` - -Create a branch for the task: - -```sh -git checkout -b a-branch-for-the-task -``` - -Update the code in the branch, and commit it. - -Push it to your own repo: - -```sh -git push --set-upstream origin a-branch-for-the-task -``` - -All left to do is creating a pull request on github.com. From 457b1f394c2de0bddb1852c3d5580a6f3ab37de2 Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Tue, 20 Mar 2018 12:46:11 -0500 Subject: [PATCH 23/34] Add format details to docs readme --- docs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/README.md b/docs/README.md index 4bad84cd7..6a9f50b45 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,4 +1,4 @@ -Pilosa docs are maintained here, to stay in sync with the codebase. +Pilosa docs are maintained here, to stay in sync with the codebase. The format is [Blackfriday](https://github.com/russross/blackfriday) markdown, with some Hugo [front matter](https://gohugo.io/content-management/front-matter/). Please visit [our website](https://www.pilosa.com/docs/) to view the docs complete with styles, diagrams, and comprehensive search. From ef4c7098288fd93d4840d9ff0c723d4ad254c232 Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Tue, 20 Mar 2018 12:47:22 -0500 Subject: [PATCH 24/34] Deprecate input definitionn in docs --- docs/api-reference.md | 16 ++++++++++ docs/getting-started.md | 9 ++++-- docs/input-definition.md | 63 ++++------------------------------------ 3 files changed, 28 insertions(+), 60 deletions(-) diff --git a/docs/api-reference.md b/docs/api-reference.md index 1dbc59865..2c5309da0 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -199,6 +199,10 @@ curl localhost:10101/index/repository/frame/stats/field/pullrequests \ ### Create input definition +
+Input definition is deprecated as of v0.9. +
+ `POST /index//input-definition/` Creates an input definition in the given index with the given name. @@ -263,6 +267,10 @@ curl localhost:10101/index/user/input-definition/stargazer-input \ ### Get input definition +
+Input definition is deprecated as of v0.9. +
+ `GET /index//input-definition/` Returns the given input definition as JSON. @@ -276,6 +284,10 @@ curl -XGET localhost:10101/index/user/input-definition/stargazer-input ### Remove input definition +
+Input definition is deprecated as of v0.9. +
+ `DELETE /index//input-definition/` Removes the given input definition. @@ -289,6 +301,10 @@ curl -XDELETE localhost:10101/index/user/input-definition/stargazer-input ### Process input data +
+Input definition is deprecated as of v0.9. +
+ `POST /index//input/` Processes the JSON payload using the given input definition. diff --git a/docs/getting-started.md b/docs/getting-started.md index 0b773ea2c..bd227d154 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -85,7 +85,7 @@ curl localhost:10101/index/repository/frame/language \ #### Import Data From CSV Files -If you import data using csv files and without input defintion, download the `stargazer.csv` and `language.csv` files in that repo. +Download the `stargazer.csv` and `language.csv` files here: ``` curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/stargazer.csv @@ -110,7 +110,12 @@ docker exec -it pilosa /pilosa import -i repository -f language /language.csv Note that, both the user IDs and the repository IDs were remapped to sequential integers in the data files, they don't correspond to actual Github IDs anymore. You can check out `languages.txt` to see the mapping for languages. ### Input Definition -Alternatively Pilosa can import JSON data using an [Input Definition](../input-definition/) describing the schema and ETL rules to process the data. + +
+Input definition is deprecated as of v0.9. +
+ +Alternatively Pilosa can import JSON data using an [Input Definition](../input-definition/) describing the schema and ETL rules to process the data. #### Make Some Queries diff --git a/docs/input-definition.md b/docs/input-definition.md index 7d50c6f9d..9d0e5fe6b 100644 --- a/docs/input-definition.md +++ b/docs/input-definition.md @@ -1,64 +1,11 @@ +++ title = "Input Definition" weight = 8 -nav = [ - "Create the Schema", - "Import Data", -] +++ ## Input Definition -This document builds on the data import concepts introduced in [Getting Started](../getting-started/). -Here we will demonstrate creating the index's schema and data definition. Then using this definition to import JSON data. - -### Create the Schema - -Input definitions allow users to define a schema based on their data and to provide data to Pilosa in a more standard format like JSON. Once an input definition is created, we can send data to Pilosa as JSON, and as long as the data adheres to the definition, Pilosa will internally perform all of the appropriate mutations. - -Before creating a schema, let's create the repository index first: - -``` -curl localhost:10101/index/repository -X POST -``` -The sample input definition schema for the "Star Trace" project is at [Pilosa Getting Started repository](https://github.com/pilosa/getting-started) in the `input_definition.json` file. Download it using: -``` -curl -OL https://github.com/pilosa/getting-started/raw/master/input_definition.json -``` - -Run the following to create the input definition: -``` -curl localhost:10101/index/repository/input-definition/stargazer -d @input_definition.json -``` - -Instead of creating a `stargazer` frame and a `language` frame individually like in [Getting Started](../getting-started/), we can create multiple frames in one input definition. -We can also set `repo_id` for multiple frames at the same time by providing field actions. There are three options for valueDestination: - - - value-to-row: The value for this field is used as the `rowID`. - - single-row-boolean: The value must be a boolean, and this specifies `SetBit()` or `ClearBit()`, a `rowID` must be specified for this destination type. - - mapping: The value for this field is used to lookup a `rowID` in a map. A valueMap is required for this destination type. - - set-timestamp: The value for this field is used to lookup timestamp and set timestamp for the whole frame - -### Import Data - -The sample data for the input definition we created above is in the `json_input.json` file at [Pilosa Getting Started repository](https://github.com/pilosa/getting-started). Download it using: -``` -curl -OL https://github.com/pilosa/getting-started/raw/master/json_input.json -``` - -Then run the following to import it: -``` -curl localhost:10101/index/repository/input/stargazer -d @json_input.json -``` - -As defined in the input definition, field name `language_id` maps language to a corresponding id defined in `valueMap` and sets the appropriate bit in the `language` frame. The value corresponding to field name `stargazer_id` is added to the `stargazer` frame as rowID. -The data input above is equivalent to the following `SetBit()` operations: - -``` -curl localhost:10101/index/repository/query \ - -X POST \ - -d 'SetBit(frame="stargazer", columnID=91720568, rowID=513114) - SetBit(frame="stargazer", columnID=91720568, rowID=513114, timestamp="2017-05-18T20:40") - SetBit(frame="language", columnID=91720568, rowID=5) - SetBit(frame="language", columnID=95122322, rowID=17) - ' -``` +
+Input definition is deprecated as of Pilosa v0.9.
+
+The previous version of this page is still available here. +
From 0800c0348a7163d1a75be1670673a017ce7fe4b7 Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Tue, 20 Mar 2018 12:48:46 -0500 Subject: [PATCH 25/34] Add readthedocs link --- docs/client-libraries.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/client-libraries.md b/docs/client-libraries.md index 1c360741c..d5074a059 100644 --- a/docs/client-libraries.md +++ b/docs/client-libraries.md @@ -96,7 +96,7 @@ func main() { ### Python -You can find the Python client library for Pilosa at our [Python Pilosa Repository](https://github.com/pilosa/python-pilosa). Check out its [README](https://github.com/pilosa/python-pilosa/blob/master/README.md) for more information and installation instructions. +You can find the Python client library for Pilosa at our [Python Pilosa Repository](https://github.com/pilosa/python-pilosa). Check out its [README](https://github.com/pilosa/python-pilosa/blob/master/README.md) or [readthedocs](https://pilosa.readthedocs.io/en/latest/) for more information and installation instructions. We are going to use the index you have created in the [Getting Started](../getting-started/) section. Before carrying on, make sure that example index is created, sample stargazer data is imported and Pilosa server is running on the default address: `http://localhost:10101`. From 54cb6ce47ed3ef01b8c39578f0ffab514ab49ff3 Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Tue, 20 Mar 2018 14:21:16 -0500 Subject: [PATCH 26/34] Add description for external tutorials --- docs/tutorials.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/tutorials.md b/docs/tutorials.md index f9b4a1494..ccbcc36bc 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -10,6 +10,8 @@ nav = [ ### External Tutorials +Some of our tutorials work better as standalone repos, since you can `git clone` the instructions, code, and data all at once. Officially supported tutorials are listed here. + - [Run Pilosa with Microsoft's Azure Cosmos DB](https://github.com/pilosa/cosmosa) ## Tutorials From 9e5d011b4a70bc3d9de1a12f7931b4b16b9ece69 Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Tue, 20 Mar 2018 14:44:51 -0500 Subject: [PATCH 27/34] Remove 'go get' in favor of 'git clone' or curlbash --- CONTRIBUTING.md | 4 ++-- docs/installation.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 10b19eae1..8a6ff2b4f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -39,10 +39,10 @@ If you want to help but you aren't sure where to start, check out our [github la cd ${GOPATH}/src/github.com/pilosa/pilosa ``` -- Install `dep` to manage dependencies: +- [Install](https://github.com/golang/dep/#installation) `dep` to manage dependencies: ```sh - go get -u github.com/golang/dep/cmd/dep + curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh ``` - Install Pilosa command line tools: diff --git a/docs/installation.md b/docs/installation.md index 5a0ea3086..e26ddf94a 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -141,7 +141,7 @@ There are four ways to install Pilosa on MacOS: Use [Homebrew](https://brew.sh/) 2. Clone the repo: ``` - go get -d github.com/pilosa/pilosa + git clone https://github.com/pilosa/pilosa.git $GOPATH/src/github.com/pilosa/pilosa ``` 3. Build the Pilosa repo (the `make generate-statik` line isn't necessary but builds a nice web console into Pilosa): @@ -294,7 +294,7 @@ There are three ways to install Pilosa on Linux: download the binary (recommende 2. Clone the repo: ``` - go get -d github.com/pilosa/pilosa + git clone https://github.com/pilosa/pilosa.git $GOPATH/src/github.com/pilosa/pilosa ``` 3. Build the Pilosa repo: From 23e65cc225988b12ca334898d06d3ae11f2a4ebc Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Tue, 20 Mar 2018 14:45:27 -0500 Subject: [PATCH 28/34] Clean up a few links --- CONTRIBUTING.md | 12 ++++++++---- docs/installation.md | 4 ++-- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8a6ff2b4f..402895f1d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -20,9 +20,9 @@ If you want to help but you aren't sure where to start, check out our [github la ### Development Environment -- Ensure you have a recent version of [Go](https://golang.org/dl/) installed. Pilosa generally supports the current and previous minor versions; check our [travis file](../.travis.yml) for the most up-to-date information. +- Ensure you have a recent version of [Go](https://golang.org/doc/install) installed. Pilosa generally supports the current and previous minor versions; check our [travis file](../.travis.yml) for the most up-to-date information. -- Make sure `$GOPATH` environment variable points to your Go working directory and `$PATH` incudes `$GOPATH/bin`. +- Make sure `$GOPATH` environment variable points to your Go working directory and `$PATH` incudes `$GOPATH/bin`, as described [here](https://golang.org/doc/code.html#GOPATH). - Fork the [Pilosa repository][2] to your own account. @@ -49,8 +49,12 @@ If you want to help but you aren't sure where to start, check out our [github la ```sh make install - # or: - # dep ensure && go install github.com/pilosa/pilosa/cmd/... + ``` + + or + + ``` + dep ensure && go install github.com/pilosa/pilosa/cmd/... ``` Running `pilosa` should now run a Pilosa instance. diff --git a/docs/installation.md b/docs/installation.md index e26ddf94a..71904b147 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -136,7 +136,7 @@ There are four ways to install Pilosa on MacOS: Use [Homebrew](https://brew.sh/) 1. Install the prerequisites: - * [Go](https://golang.org/doc/install). Be sure to set the `$GOPATH` and `$PATH` environment variables as described here (https://golang.org/doc/code.html#GOPATH). + * [Go](https://golang.org/doc/install). Be sure to set the `$GOPATH` and `$PATH` environment variables as described [here](https://golang.org/doc/code.html#GOPATH). * [Git](https://git-scm.com/) 2. Clone the repo: @@ -289,7 +289,7 @@ There are three ways to install Pilosa on Linux: download the binary (recommende 1. Install the prerequisites: - * [Go](https://golang.org/doc/install). Be sure to set the `$GOPATH` and `$PATH` environment variables as described here (https://golang.org/doc/code.html#GOPATH). + * [Go](https://golang.org/doc/install). Be sure to set the `$GOPATH` and `$PATH` environment variables as described [here](https://golang.org/doc/code.html#GOPATH). * [Git](https://git-scm.com/) 2. Clone the repo: From 96c7f285e54b9f5c7770ee59f8410529edc1369d Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Tue, 20 Mar 2018 14:52:15 -0500 Subject: [PATCH 29/34] Ensure directory exists before cloning --- docs/installation.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/installation.md b/docs/installation.md index 71904b147..67558d904 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -141,7 +141,8 @@ There are four ways to install Pilosa on MacOS: Use [Homebrew](https://brew.sh/) 2. Clone the repo: ``` - git clone https://github.com/pilosa/pilosa.git $GOPATH/src/github.com/pilosa/pilosa + mkdir -p ${GOPATH}/src/github.com/pilosa && cd $_ + git clone https://github.com/pilosa/pilosa.git ``` 3. Build the Pilosa repo (the `make generate-statik` line isn't necessary but builds a nice web console into Pilosa): @@ -294,7 +295,8 @@ There are three ways to install Pilosa on Linux: download the binary (recommende 2. Clone the repo: ``` - git clone https://github.com/pilosa/pilosa.git $GOPATH/src/github.com/pilosa/pilosa + mkdir -p ${GOPATH}/src/github.com/pilosa && cd $_ + git clone https://github.com/pilosa/pilosa.git ``` 3. Build the Pilosa repo: From edb63e70b19029c5442cf3a1a46f89d0b72a409d Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Tue, 20 Mar 2018 15:17:48 -0500 Subject: [PATCH 30/34] Synchronize mac and linux install sections --- docs/installation.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/installation.md b/docs/installation.md index 67558d904..9ca90ccf2 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -145,7 +145,7 @@ There are four ways to install Pilosa on MacOS: Use [Homebrew](https://brew.sh/) git clone https://github.com/pilosa/pilosa.git ``` -3. Build the Pilosa repo (the `make generate-statik` line isn't necessary but builds a nice web console into Pilosa): +3. Build the Pilosa repo (the `make generate-statik` line isn't necessary but builds a nice [webUI](../webui/) into Pilosa): ``` cd $GOPATH/src/github.com/pilosa/pilosa make generate-statik @@ -204,7 +204,7 @@ There are four ways to install Pilosa on MacOS: Use [Homebrew](https://brew.sh/) docker version ``` -If you don't see the server listed, start the Docker application. + If you don't see the server listed, start the Docker application. 3. Pull the official Pilosa image from Docker Hub: ``` @@ -299,9 +299,10 @@ There are three ways to install Pilosa on Linux: download the binary (recommende git clone https://github.com/pilosa/pilosa.git ``` -3. Build the Pilosa repo: +3. Build the Pilosa repo (the `make generate-statik` line isn't necessary but builds a nice [webUI](../webui/) into Pilosa): ``` cd $GOPATH/src/github.com/pilosa/pilosa + make generate-statik make install ``` From 38812a38aa166766c0c39991948c1b87abd19e72 Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Tue, 20 Mar 2018 15:21:48 -0500 Subject: [PATCH 31/34] Improve links --- docs/api-reference.md | 3 ++- docs/webui.md | 5 +++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/api-reference.md b/docs/api-reference.md index 2c5309da0..9092c92bc 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -107,9 +107,10 @@ The request payload is in JSON, and may contain the `options` field. The `option * `cacheType` (string): [ranked](../data-model/#ranked) or [LRU](../data-model/#lru) caching on this frame. Default is `lru`. * `cacheSize` (int): Number of rows to keep in the cache. Default 50,000. * `rangeEnabled` (boolean): Enables range-encoded fields in this frame. -* `fields` (array): List of range-encoded fields. +* `fields` (array): List of range-encoded [fields](../data-model/#bsi-range-encoding). Each individual `field` contains the following: + * `name` (string): Field name. * `type` (string): Field type, currently only "int" is supported. * `min` (int): Minimum value allowed for this field. diff --git a/docs/webui.md b/docs/webui.md index 97fa2ce8f..525f1aed9 100644 --- a/docs/webui.md +++ b/docs/webui.md @@ -9,7 +9,8 @@ nav = [ ## WebUI -The Pilosa server comes packaged with in-browser WebUI. When you run a local Pilosa server on the default host, you can access it at [localhost:10101](http://localhost:10101) +The Pilosa server comes packaged with in-browser WebUI. When you run a local Pilosa server on the default host, you can access it at [localhost:10101](http://localhost:10101). + This can be used for constructing queries and viewing the cluster status. ### Console @@ -31,7 +32,7 @@ In addition to standard PQL, the console supports a few special commands, prefix - `:create frame ` - `:delete frame ` -Frame creation also supports options like `timeQuantum` or `inverseEnabled`. When creating a new frame, add options by using the keys documented in [API reference](../api-reference/). +Index and frame creation also supports options like timeQuantum or inverseEnabled. Check the API reference sections for [index](../api-reference/#change-index-time-quantum) and [frame](../api-reference/#create-frame) options to see other available keys. - `:create index timeQuantum=YM` - `:create frame inverseEnabled=true cacheSize=10000` From daca793c7f7b7782878f5849cda4e7d879ac1bfd Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Tue, 20 Mar 2018 16:00:51 -0500 Subject: [PATCH 32/34] Remove reference to deprecated index timeQuantum --- docs/webui.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/webui.md b/docs/webui.md index 525f1aed9..738fb20fd 100644 --- a/docs/webui.md +++ b/docs/webui.md @@ -32,9 +32,8 @@ In addition to standard PQL, the console supports a few special commands, prefix - `:create frame ` - `:delete frame ` -Index and frame creation also supports options like timeQuantum or inverseEnabled. Check the API reference sections for [index](../api-reference/#change-index-time-quantum) and [frame](../api-reference/#create-frame) options to see other available keys. +Frame creation also supports options like `timeQuantum` or `inverseEnabled`. When creating a new frame, add options by using the keys documented in [API reference](../api-reference/#create-frame). -- `:create index timeQuantum=YM` - `:create frame inverseEnabled=true cacheSize=10000` From cfd083042f9d8c69f86f5646b27b92925055d24c Mon Sep 17 00:00:00 2001 From: Cody Soyland Date: Wed, 21 Mar 2018 11:44:59 -0500 Subject: [PATCH 33/34] Add CHANGELOG changes from v0.8 to master --- CHANGELOG.md | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9676b4f89..ac62abc67 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,53 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/) and this project adheres to [Semantic Versioning](http://semver.org/). +## [0.8.8] - 2018-02-19 + +This version contains 1 contribution from 2 contributors. There are 4 files changed, 1,153 insertions, and 618 deletions. + +### Fixed + +- Bug fixes and improved test coverage in roaring ([#1118](https://github.com/pilosa/pilosa/pull/1118)) + +## [0.8.7] - 2018-02-12 + +This version contains 1 contribution from 1 contributors. There are 2 files changed, 84 insertions, and 4 deletions. + +### Fixed + +- Fix a shift logic bug in bitmapZeroRange ([#1111](https://github.com/pilosa/pilosa/pull/1111)) + +## [0.8.6] - 2018-02-09 + +This version contains 2 contributions from 2 contributors. There are 3 files changed, 171 insertions, and 6 deletions. + +### Fixed + +- Fix overflow bug in differenceRunArray [#1106](https://github.com/pilosa/pilosa/pull/1106) +- Fix bug where count and bitmap queries could return different numbers [#1083](https://github.com/pilosa/pilosa/pull/1083) + +## [0.8.5] - 2018-01-18 + +This version contains 1 contribution from 1 contributor. There is 1 file changed, 1 insertion, and 0 deletions. + +### Fixed + +- Bind Docker container on all interfaces ([#1061](https://github.com/pilosa/pilosa/pull/1061)) + +## [0.8.4] - 2018-01-10 + +This version contains 4 contributions from 3 contributors. There are 17 files changed, 974 insertions, and 221 deletions. + +### Fixed + +- Group the write operations in syncBlock by MaxWritesPerRequest ([#1038](https://github.com/pilosa/pilosa/pull/1038)) +- Change gossip config from memberlist.DefaultLocalConfig to memberlist.DefaultWANConfig ([#1033](https://github.com/pilosa/pilosa/pull/1033)) + +### Performance + +- Change AttrBlock handler calls to support protobuf instead of json ([#1046](https://github.com/pilosa/pilosa/pull/1046)) +- Use RLock instead of Lock in a few places ([#1042](https://github.com/pilosa/pilosa/pull/1042)) + ## [0.8.3] - 2017-12-12 This version contains 1 contribution from 1 contributor. There are 2 files changed, 59 insertions, and 42 deletions. From 48081c12659f6513ecc55efae0012491db24760a Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Thu, 22 Mar 2018 11:30:51 -0500 Subject: [PATCH 34/34] Move external tutorial list to a note under the main heading --- docs/tutorials.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/docs/tutorials.md b/docs/tutorials.md index ccbcc36bc..2401cdfe4 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -8,14 +8,18 @@ nav = [ ] +++ -### External Tutorials - -Some of our tutorials work better as standalone repos, since you can `git clone` the instructions, code, and data all at once. Officially supported tutorials are listed here. - -- [Run Pilosa with Microsoft's Azure Cosmos DB](https://github.com/pilosa/cosmosa) - ## Tutorials +
+ +Some of our tutorials work better as standalone repos, since you can git clone the instructions, code, and data all at once. Officially supported tutorials are listed here.
+
+ + +
+ ### Setting Up a Secure Cluster #### Introduction