diff --git a/docs/administration.md b/docs/administration.md index 97ee688f5..664301286 100644 --- a/docs/administration.md +++ b/docs/administration.md @@ -112,17 +112,17 @@ Note: This will only work when the replication factor is >= 2 #### Copying data files manually -- To accomplish this goal you will 1st need: - - List of all Indexes on your cluster - - List of all frames in your Indexes - - Max slice per Index, listed in the /status endpoint +- To accomplish this you will first need: + - List of all indexes on your cluster + - List of all frames in your indexes + - Max slice per index, listed in the /status endpoint - With this information you can query the `/fragment/nodes` endpoint and iterate over each slice - Using the list of slices owned by this node you will then need to manually: - setup a directory structure similar to the other nodes with a path for each Index/Frame - copy each owned slice for an existing node to this new node - Modify the cluster config file to replace the previous node address with the new node address. - Restart the cluster -- Wait for the 1st sync (10 minutes) to validate Index connections +- Wait for the first sync (10 minutes) to validate Index connections ### Diagnostics diff --git a/docs/architecture.md b/docs/architecture.md index 5cc5ff6ee..81a7e54f7 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -18,5 +18,8 @@ Bitmaps are persisted to disk using a file format very similar to the [Roaring B * After the container storage section is an operation log, of unspecified length. ![roaring file format diagram](/img/docs/pilosa-roaring-storage-diagram.png) +*Pilosa Roaring storage format diagram* All values are little-endian. The first two bytes of the cookie is 12348, to reflect incompatibility with the spec, which uses 12346 or 12347. Container types are NOT inferred from their cardinality as in the spec. Instead, the container type is read directly from the descriptive header. + +Check out this [blog post](/blog/adding-rle-support/) for some more details about Roaring in Pilosa. diff --git a/docs/data-model.md b/docs/data-model.md index d448a1de8..1fc5f899a 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -26,7 +26,8 @@ Pilosa lays out data first in rows, so queries which get all the set bits in one Please note that Pilosa is most performant when row and column IDs are sequential starting from 0. You can deviate from this to some degree, but if you try to set a bit with column ID 2^63, bad things will start to happen. -![data model diagram](/img/docs/data-model.svg) +![basic data model diagram](/img/docs/data-model.svg) +*Basic data model diagram* ### Index @@ -51,12 +52,14 @@ Row attributes are namespaced at the Frame level. Ranked Frames maintain a sorted cache of column counts by Row ID (yielding the top rows by columns with a bit set in each). This cache facilitates the TopN query. The cache size defaults to 50,000 and can be set at Frame creation. ![ranked frame diagram](/img/docs/frame-ranked.svg) +*Ranked frame diagram* #### LRU The LRU cache maintains the most recently accessed Rows. ![lru frame diagram](/img/docs/frame-lru.svg) +*LRU frame diagram* ### Time Quantum @@ -92,6 +95,7 @@ SetBit(frame="A", rowID=19, columnID=5) ``` ![inverse frame diagram](/img/docs/frame-inverse.svg) +*Inverse frame diagram* #### Time Quantums @@ -103,6 +107,7 @@ SetBit(frame="A", rowID=8, columnID=3, timestamp="2017-05-19T00:00") ``` ![time quantum frame diagram](/img/docs/frame-time-quantum.svg) +*Time quantum frame diagram* #### BSI Range-Encoding @@ -122,4 +127,5 @@ SetFieldValue(col=2, frame="A", field1=1) SetFieldValue(col=3, frame="A", field1=6) ``` -![BSI diagram](/img/docs/frame-bsi.svg) +![BSI frame diagram](/img/docs/frame-bsi.svg) +*BSI frame diagram* diff --git a/docs/examples.md b/docs/examples.md index bf5e76f29..262684615 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -214,7 +214,7 @@ T(A,B)= Intersect(A,B) / (Count(A) + Count(B) - Intersect(A,B)) A and B are sets of fingerprint bits on in the fingerprints of molecule A and molecule B. AB is the set of common bits of fingerprints of both molecule A and B. The Tanimoto coefficient ranges from 0 when the fingerprints have no bits in common, to 1 when the fingerprints are identical. -All source code to calculate tanimoto for molecule fingerprint using Pilosa is available in a Github repository https://github.com/pilosa/chem-usecase +All source code to calculate tanimoto for molecule fingerprint using Pilosa is available in a [Github repository](https://github.com/pilosa/chem-usecase). #### Data model diff --git a/docs/query-language.md b/docs/query-language.md index 3a5b94d60..83098fe0a 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -13,7 +13,7 @@ nav = [ ### Overview -This section will provide a detailed reference and examples for the Pilosa Query Language (PQL). All PQL queries operate on a single [index]({{< ref "glossary.md#index" >}}) and are passed to Pilosa through the `/index/*index_name*/query` endpoint. You may pass multiple PQL queries in a single request by simply concatenating the queries together - a space is not needed. The results format is always: +This section will provide a detailed reference and examples for the Pilosa Query Language (PQL). All PQL queries operate on a single [index]({{< ref "glossary.md#index" >}}) and are passed to Pilosa through the `/index/INDEX_NAME/query` endpoint. You may pass multiple PQL queries in a single request by simply concatenating the queries together - a space is not needed. The results format is always: ``` {"results":[...]} @@ -380,6 +380,7 @@ have the attribute specified by `field` with one of the values specified in **Result Type:** array of key/count objects **Caveats:** + * Performing a TopN() query on a frame with cache type ranked will return the top bitmaps sorted by count in descending order. * Frames with cache type lru will maintain an LRU (Least Recently Used) cache, thus a TopN() query on this type of frame will return bitmaps sorted in order of most recently set bit. * The frame's cache size determines the number of sorted bitmaps to maintain in the cache for purposes of TopN() queries. There is a tradeoff between performance and accuracy; increasing the cache size will improve accuracy of results at the cost of performance. diff --git a/docs/tutorials.md b/docs/tutorials.md index ffaf51d64..2ffb1b41d 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -49,6 +49,7 @@ openssl req -x509 -newkey rsa:2048 -keyout pilosa.local.key -out pilosa.local.cr ``` The command above creates two files in the current directory: + * `pilosa.local.crt` is the SSL certificate. * `pilosa.local.key` is the private key file which must be kept as secret. @@ -130,6 +131,7 @@ key = "pilosa.local.gossip32" ``` Here is some explanation of the configuration items: + * `data-dir` points to the directory where the Pilosa server writes its data. If it doesn't exist, the server will create it. * `bind` is the address to which the server listens for incoming requests. The address is composed of three parts: scheme, host, and port. The default scheme is `http` so we explicitly specify `https` to use the HTTPS protocol for communication between nodes. * `[cluster]` section contains the settings for a cluster. `hosts` field is the most important, which contains the list of addresses of other nodes. See [Cluster Configuration](https://www.pilosa.com/docs/latest/configuration/#cluster-hosts) for other settings. diff --git a/docs/webui.md b/docs/webui.md index cef8e570d..86874807b 100644 --- a/docs/webui.md +++ b/docs/webui.md @@ -20,7 +20,8 @@ Each query's result will be displayed in the Output section along with the query The Console will keep a record of each query and its result with the latest query on top. -![console](/img/docs/webui-console.png) +![webUI console screenshot](/img/docs/webui-console.png) +*WebUI console screenshot* In addition to standard PQL, the console supports a few special commands, prefixed with `:`.