diff --git a/docs/administration.md b/docs/administration.md index 6764823e9..71ab54672 100644 --- a/docs/administration.md +++ b/docs/administration.md @@ -70,9 +70,9 @@ pilosa import -i project -f stargazer-counts project-stargazer-counts.csv #### Exporting -Exporting data to csv can be performed on a live instance of Pilosa. You need to specify the index and the field. The API also expects the slice number, but the `pilosa export` sub command will export all slices within a field. The data will be in csv format `Row,Column` and sorted by column. +Exporting data to csv can be performed on a live instance of Pilosa. You need to specify the index and the field. The API also expects the shard number, but the `pilosa export` sub command will export all shards within a field. The data will be in csv format `Row,Column` and sorted by column. ```request -curl "http://localhost:10101/export?index=repository&field=stargazer&slice=0" \ +curl "http://localhost:10101/export?index=repository&field=stargazer&shard=0" \ --header "Accept: text/csv" ``` ```response @@ -122,7 +122,7 @@ Pilosa v0.9 introduces a few compatibility changes that need to be addressed. Pilosa v0.9 adds two new files to the data directory, an `.id` file and a `.topology` file. Due to the way Pilosa internally shards indices, upgrading a Pilosa cluster will result in data loss if an existing cluster is brought up without these files. New clusters will generate them automatically, but you may migrate an existing cluster by using a tool we called [`topology-generator`](https://github.com/pilosa/upgrade-utils/tree/master/v0.9/topology-generator): -1. Observe the `cluster.hosts` configuration value in Pilosa v0.8. The ordering of the nodes in the config file is significant, as it determines shard (AKA slice) ownership. Pilosa v0.9 uses UUIDs for each node, and the ordering is alphabetical. +1. Observe the `cluster.hosts` configuration value in Pilosa v0.8. The ordering of the nodes in the config file is significant, as it determines shard ownership. Pilosa v0.9 uses UUIDs for each node, and the ordering is alphabetical. 2. Install the `topology-generator`: `go get github.com/pilosa/upgrade-utils/v0.9/topology-generator`. 3. Run the `topology-generator`. There are two arguments: the number of nodes and the output directory. For this example, we'll assume a 3-node cluster and place the files in the current working directory: `topology-generator 3 .`. 4. This tool will generate a file, `topology`, and multiple id files, called `nodeX.id`, X being the node index position. @@ -211,7 +211,7 @@ curl localhost:10101/cluster/resize/set-coordinator \ ### Backup/restore -Pilosa continuously writes out the in-memory bitmap data to disk. This data is organized by Index->Field->Views->Fragment->numbered slice files. These data files can be routinely backed up to restore nodes in a cluster. +Pilosa continuously writes out the in-memory bitmap data to disk. This data is organized by Index->Field->Views->Fragment->numbered shard files. These data files can be routinely backed up to restore nodes in a cluster. Depending on the size of your data you have two options. For a small dataset you can rely on the periodic anti-entropy sync process to replicate existing data back to this node. @@ -231,11 +231,11 @@ Note: This will only work when the replication factor is >= 2 - To accomplish this you will first need: - List of all indexes on your cluster - List of all fields in your indexes - - Max slice per index, listed in the `/slices/max` endpoint -- With this information you can query the `/internal/fragment/nodes` endpoint and iterate over each slice -- Using the list of slices owned by this node you will then need to manually: + - Max shard per index, listed in the `/internal/shards/max` endpoint +- With this information you can query the `/internal/fragment/nodes` endpoint and iterate over each shard +- Using the list of shards owned by this node you will then need to manually: - setup a directory structure similar to the other nodes with a path for each Index/Field - - copy each owned slice for an existing node to this new node + - copy each owned shard for an existing node to this new node - Modify the cluster config file to replace the previous node address with the new node address. - Restart the cluster - Wait for the first sync (10 minutes) to validate Index connections @@ -253,7 +253,7 @@ Each Pilosa cluster is configured by default to share anonymous usage details wi - **TimeQuantumEnabled:** Time Quantum Fields in use. - **NumIndexes:** Number of indexes in the Cluster. - **NumFields:** Number of fields in the Cluster. -- **NumSlices:** Number of slices in the Cluster. +- **NumShards:** Number of shards in the Cluster. - **NumViews:** Number of views in the Cluster. - **OpenFiles:** Open file handle count. - **GoRoutines:** Go routine count. @@ -276,14 +276,14 @@ StatsD Tags adhere to the DataDog format (key:value), and we tag the following: - Index - Field - View -- Slice +- Shard #### Events We currently track the following events - **Index:** The creation of a new index. - **Field:** The creation of a new field. -- **MaxSlice:** The creation of a new Slice. +- **MaxShard:** The creation of a new Shard. - **SetBit:** Count of set bits. - **ClearBit:** Count of cleared bits. - **ImportBit:** During a bulk data import this represents the count of bits created. diff --git a/docs/api-reference.md b/docs/api-reference.md index 41e0288ce..c3a0144df 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -78,10 +78,10 @@ In order to send protobuf binaries in the request and response, set `Content-Typ The response doesn't include column attributes by default. To return them, set the `columnAttrs` query argument to `true`. -The query is executed for all [slices](../data-model/#slice) by default. To use specified slices only, set the `slices` query argument to a comma-separated list of slice indices. +The query is executed for all [shards](../data-model/#shard) by default. To use specified shards only, set the `shards` query argument to a comma-separated list of slice indices. ``` request -curl "localhost:10101/index/user/query?columnAttrs=true&slices=0,1" \ +curl "localhost:10101/index/user/query?columnAttrs=true&shards=0,1" \ -X POST \ -d 'Row(language=5)' ``` diff --git a/docs/client-libraries.md b/docs/client-libraries.md index a492cf9b5..a277141ec 100644 --- a/docs/client-libraries.md +++ b/docs/client-libraries.md @@ -90,7 +90,7 @@ func main() { fmt.Println("User 14 or 19 starred, written in language 1:", response.Result().Row().Columns) // Set user 99999 as a stargazer for repository 77777? - client.Query(stargazer.SetBit(99999, 77777)) + client.Query(stargazer.Set(99999, 77777)) } ``` @@ -174,7 +174,7 @@ mutually_starred = client.query(query).result.row.columns print("User 14 or 19 starred, written in language 1:", mutually_starred) # Set user 99999 as a stargazer for repository 77777 -client.query(stargazer.setbit(99999, 77777)) +client.query(stargazer.set(99999, 77777)) ``` Running the above program should produce output like this: @@ -275,7 +275,7 @@ public class StarTrace { System.out.println("User 14 or 19 starred, written in language 1: " + repositoryIDs); // Set user 99999 as a stargazer for repository 77777: - client.query(stargazer.setBit(99999, 77777)); + client.query(stargazer.set(99999, 77777)); } } ``` diff --git a/docs/configuration.md b/docs/configuration.md index 200c91449..22a67ddcf 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -106,7 +106,7 @@ The config file is in the [toml format](https://github.com/toml-lang/toml) and h #### Max Writes Per Request -* Description: Maximum number of mutating commands allowed per request. This includes SetBit, ClearBit, SetRowAttrs, SetColumnAttrs, and SetFieldValue. +* Description: Maximum number of mutating commands allowed per request. This includes Set, Clear, SetRowAttrs, and SetColumnAttrs. * Flag: `--max-writes-per-request=5000` * Env: `PILOSA_MAX_WRITES_PER_REQUEST=5000` * Config: diff --git a/docs/data-model.md b/docs/data-model.md index bdcec4861..e453d9245 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -9,7 +9,7 @@ nav = [ "Field", "Time Quantum", "Attribute", - "Slice", + "Shard", "View", ] +++ @@ -64,13 +64,13 @@ Entities: Simple queries: - Relational | Pilosa ----------------------------------------------|------------------------------------ - `select ID from People where Name = 'Bob'` | `Row(Name="Bob")` - `select ID from People where Age > 30` | `Range(Age > 30)` - `select ID from People where Member = true` | `Row(Member=0)` # TODO this is unfortunate + Relational | Pilosa +-----------------------------------------------|------------------------------------ + `select ID from People where Name = 'Bob'` | `Row(Name="Bob")` + `select ID from People where Age > 30` | `Range(Age > 30)` + `select ID from People where Member = true` | `Row(Member=0)` -In the relational model, joins are often necessary. Because Pilosa supports extremely high cardinality in both rows and columns, many types of joins are accomplished with basic Pilosa queries across multiple fields. For example, this SQL join: +Note that `Row(Member=0)` selects all entities with a bit set in row 0 of the Member field. We could just as well use row 1 to store this, in which case we would use `Row(Member=1)`, which looks a bit more intuitive. In the relational model, joins are often necessary. Because Pilosa supports extremely high cardinality in both rows and columns, many types of joins are accomplished with basic Pilosa queries across multiple fields. For example, this SQL join: ```sql select AVG(p.Age) from People p @@ -141,7 +141,7 @@ Set(3, A=8, 2017-05-19T00:00) Bit-Sliced Indexing (BSI) is the storage method Pilosa uses to represent multi-bit integers in a bitmap index. Integers are stored as n-bit, range-encoded bit-sliced indexes of base-2, along with an additional row indicating "not null". This means that a 16-bit integer will require 17 rows: one for each 0-bit of the 16 bit-slice components (the 1-bit does not need to be stored because with range-encoding the highest bit position is always 1) and one for the non-null row. Pilosa can evaluate `Range`, `Min`, `Max`, and `Sum` queries on these BSI integers. The result of a `Sum` query includes a count, which can be used to compute an average with no other overhead. -Internally Pilosa stores each BSI (TODO!!!!!) `field` as a `view` within a `frame`. The rows of the `view` contain the base-2 representations of the integer values. Pilosa manages the base-2 offset and translation that efficiently packs the integer value within the minimum set of rows. +Internally Pilosa stores each BSI `field` as a `view`. The rows of the `view` contain the base-2 representations of the integer values. Pilosa manages the base-2 offset and translation that efficiently packs the integer value within the minimum set of rows. For example, the following `Set()` queries executed against BSI fields will result in the data described in the diagram below: diff --git a/docs/examples.md b/docs/examples.md index 1660e8592..692cd7e39 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -3,7 +3,6 @@ title = "Examples" weight = 4 nav = [ "Transportation", - "Chemical similarity search", ] +++ diff --git a/docs/glossary.md b/docs/glossary.md index eb5dd780e..81323674d 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -56,7 +56,7 @@ nav = [] [Row](../data-model/#row): Rows are the fundamental vertical data axis within Pilosa. They are namespaced to each [field](#field) within an [index](#index). Represented as a [Bitmap](#bitmap). -[Slice](../data-model/#slice): Prior to Pilosa 1.0, shards were known as slices. +[Slice](../data-model/#shard): Prior to Pilosa 1.0, shards were known as slices. [Shard](../data-model/#shard): [Columns](#column) are [sharded](https://en.wikipedia.org/wiki/Shard_(database_architecture)) on a preset [width](#shardwidth). Shards are operated on in parallel and are evenly distributed across the cluster via a [consistent hash](#jump-consistent-hash). diff --git a/docs/pdk.md b/docs/pdk.md index 2ca4b6ee8..dd6ddaf61 100644 --- a/docs/pdk.md +++ b/docs/pdk.md @@ -18,7 +18,7 @@ Running `pdk -h` will give the most up to date list of all the tools and example `pdk kafka` reads either JSON or Avro encoded records from Kafka (using the Confluent Schema Registry in the case of Avro), and indexes them in Pilosa. Each record from Kafka is assigned a Pilosa column, and each value in a record is -assigned a row or field. Frame and field names are built from the "path" through +assigned a row or field. Pilosa field names are built from the "path" through the record to arrive at that field. For example: ```json @@ -38,30 +38,30 @@ the record to arrive at that field. For example: This JSON object would result in the following Pilosa schema: -| Name | Field | Type | Min | Max | Size | -|----------------|-----------|--------|-----|------------|--------| -| name | | ranked | | | 100000 | -| favorite_foods | | ranked | | | 100000 | -| default | | ranked | | | 100000 | -| | age | int | 0 | 2147483647 | | -| location | | ranked | | | 1000 | -| | latitude | int | 0 | 2147483647 | | -| | longitude | int | 0 | 2147483647 | | -| location-city | | ranked | | | 100000 | -| location-state | | ranked | | | 100000 | +| Field | Type | Min | Max | Size | +|----------------|--------|-----|------------|--------| +| name | ranked | | | 100000 | +| favorite_foods | ranked | | | 100000 | +| default | ranked | | | 100000 | +| age | int | 0 | 2147483647 | | +| location | ranked | | | 1000 | +| latitude | int | 0 | 2147483647 | | +| longitude | int | 0 | 2147483647 | | +| location-city | ranked | | | 100000 | +| location-state | ranked | | | 100000 | -All frames are created as ranked frames by default, with the cache size listed above. Fields are created with -a minimum size of zero and a fixed maximum of 2147483647. Fields at the top level -are created in the default frame. Frames are a dash-separated concatenation of -all key values in the path - you can see this with frames like location-city. +All set fields are created as ranked fields by default, with the cache size +listed above. Integer fields are created with a minimum size of zero and a +fixed maximum of 2147483647. Field names are a dash-separated concatenation of +all key values in the path - you can see this with fields like location-city. Most of the options to `pdk kafka` are self-explanatory (kafka hosts, pilosa hosts, kafka topics, kafka group, etc.), but there are a few options that give some control over the way data is indexed, and ingestion performance. -* `--batch-size`: The batch size controls how many set bits or values are batched up to be imported *per frame*. So for fields that have one value per record, you have to wait for `batch-size` records to come through before you'll see the data indexed in Pilosa. Fields like `favorite_foods` which can have multiple values could be indexed sooner. -* `--framer.collapse`: This is a list of strings which will be removed from the frame names created by dash-concatentating all names in the JSON path to a value. E.G. if "location" were listed in `framer.collapse`, then there would be frames named "city" and "state" rather than "location-city" and "location-state". +* `--batch-size`: The batch size controls how many set bits or values are batched up to be imported *per field*. So for fields that have one value per record, you have to wait for `batch-size` records to come through before you'll see the data indexed in Pilosa. Fields like `favorite_foods` which can have multiple values could be indexed sooner. +* `--framer.collapse`: This is a list of strings which will be removed from the field names created by dash-concatentating all names in the JSON path to a value. E.G. if "location" were listed in `framer.collapse`, then there would be fields named "city" and "state" rather than "location-city" and "location-state". * `--framer.ignore`: This allows you to skip indexing on any path containing these strings. If you have a field like email address or some other unique ID, you might not want to index it. * `--subject-path`: If nothing is passed for this option, then each record will be assigned a unique sequential column ID. If `subject-path` is specified, then the value at this path in the record will be mapped to a column ID. If the same value appears in another record, the same column ID will be used. * `--proxy`: The PDK ingests data, but also keeps a mapping for string values to row IDs, and from subjects to column ids. Because of this, querying Pilosa directly may not be useful, since it only returns integer row and column ids. The PDK will start a proxy server which intercepts requests to Pilosa using strings for row and column ids, and translates them to the integers that Pilosa understands. It will also translate responses so that (e.g.) a TopN query will return `{"results":[[{"Key":"chipotle dip","Count":1},{"Key":"corn chips","Count":1}]]}`. By default, the mapping is stored in an embedded leveldb. diff --git a/docs/query-language.md b/docs/query-language.md index 62c912ba4..2ea85a06e 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -58,7 +58,7 @@ curl localhost:10101/index/repository/query \ **Spec:** ``` -Set(, field=, [TIMESTAMP]) +Set(, =, [TIMESTAMP]) ``` **Description:** @@ -202,12 +202,12 @@ SetColumnAttrs(10, url=null) {"results":[null]} ``` -#### ClearBit +#### Clear **Spec:** ``` -Clear(, field=) +Clear(, =) ``` **Description:** @@ -241,7 +241,7 @@ This represents removing the relationship between the user with id=1 and the rep **Spec:** ``` -Row(field=) +Row(=) ``` **Description:** @@ -425,7 +425,7 @@ TopN([ROW_CALL], , [n=UINT], **Description:** -Return the id and count of the top `n` bitmaps (by count of bits) in the field. +Return the id and count of the top `n` rows (by count of bits) in the field. The `attrName` and `attrValues` arguments work together to only return rows which have the attribute specified by `attrName` with one of the values specified in `attrValues`. @@ -434,11 +434,11 @@ have the attribute specified by `attrName` with one of the values specified in **Caveats:** -* Performing a TopN() query on a field with cache type ranked will return the top bitmaps sorted by count in descending order. -* Fields with cache type lru will maintain an LRU (Least Recently Used replacement policy) cache, thus a TopN query on this type of field will return bitmaps sorted in order of most recently set bit. -* The field's cache size determines the number of sorted bitmaps to maintain in the cache for purposes of TopN queries. There is a tradeoff between performance and accuracy; increasing the cache size will improve accuracy of results at the cost of performance. -* Once full, the cache will truncate the set of bitmaps according to the field option CacheSize. Bitmaps that straddle the limit and have the same count will be truncated in no particular order. -* The TopN query's attribute filter is applied to the existing sorted cache of bitmaps. Bitmaps that fall outside of the sorted cache range, even if they would normally pass the filter, are ignored. +* Performing a TopN() query on a field with cache type ranked will return the top rows sorted by count in descending order. +* Fields with cache type lru will maintain an LRU (Least Recently Used replacement policy) cache, thus a TopN query on this type of field will return rows sorted in order of most recently set bit. +* The field's cache size determines the number of sorted rows to maintain in the cache for purposes of TopN queries. There is a tradeoff between performance and accuracy; increasing the cache size will improve accuracy of results at the cost of performance. +* Once full, the cache will truncate the set of rows according to the field option CacheSize. Rows that straddle the limit and have the same count will be truncated in no particular order. +* The TopN query's attribute filter is applied to the existing sorted cache of rows. Rows that fall outside of the sorted cache range, even if they would normally pass the filter, are ignored. See [field creation](../api-reference/#create-field) for more information about the cache. @@ -466,7 +466,7 @@ TopN(stargazer, n=2) * Results are the top two rows (users) sorted by number of bits set (repositories they've starred) in descending order. -Filter based on an existing Bitmap: +Filter based on an existing row: ```request TopN(Row(language=1), stargazer, n=2) ``` @@ -491,7 +491,7 @@ TopN(stargazer, n=2, attrName=active, attrValues=[true]) **Spec:** ``` -Range(field=, , ) +Range(=, , ) ``` **Description:** @@ -632,7 +632,7 @@ Sum([ROW_CALL], field=) Returns the count and computed sum of all BSI integer values in the `field`. If the optional `Row` call is supplied, columns with set bits are summed, otherwise the sum is across all columns. -**Result Type:** object with the computed sum and count of the bitmap field. +**Result Type:** object with the computed sum and count of the values in the integer field. **Examples:**