diff --git a/docs/client-libraries.md b/docs/client-libraries.md index 5044bae9d..c3a78984d 100644 --- a/docs/client-libraries.md +++ b/docs/client-libraries.md @@ -10,7 +10,7 @@ nav = [ ## Client Libraries -This section contains example code for client libraries in several languages. Please remember that when modeling your data in Pilosa, it is best to keep row and column ids sequential. It is not wise to use the output of a hash, or randomly distributed ids with Pilosa. +This section contains example code for client libraries in several languages. Please remember that when modeling your data in Pilosa, it is best to keep row and column ids sequential. It is best to avoid using the the output of a hash, or randomly distributed ids with Pilosa. ### Go diff --git a/docs/data-model.md b/docs/data-model.md index c14ed0dba..9a1f9cfc5 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -22,16 +22,16 @@ The central component of Pilosa's data model is a boolean matrix. Each cell in t Rows and columns can represent anything (they could even represent the same set of things). Pilosa can associate arbitrary key/value pairs (referred to as attributes) to rows and columns, but queries and storage are optimized around the core matrix. -Pilosa lays out data first in rows, so queries which get all the set bits in one or many rows, or compute a combining operation on multiple rows such as Intersect or Union are the fastest. Pilosa also has the ability to categorize rows into different "frames" and quickly retrieve the top rows in a frame sorted by the number of bits set in each row. +Pilosa lays out data first in rows, so queries which get all the set bits in one or many rows, or compute a combining operation on multiple rows such as Intersect or Union are the fastest. Pilosa categorizes rows into different *frames* and quickly retrieve the top rows in a frame sorted by the number of bits set in each row. -Please note that Pilosa is most performant when row and column IDs are sequential starting from 0. You can deviate from this to some degree, but if you try to set a bit with column ID 2^63, bad things will start to happen. +Please note that Pilosa is most performant when row and column IDs are sequential starting from 0. You can deviate from this to some degree, but setting a bit with column ID 263 on a single-node cluster, for example, will not work well due to memory limitations. ![basic data model diagram](/img/docs/data-model.svg) *Basic data model diagram* ### Index -The purpose of the Index is to represent a data namespace. You cannot perform cross-index queries. Column-level attributes are global to the Index. +The purpose of the Index is to represent a data namespace. You cannot perform cross-index queries. ### Column @@ -77,7 +77,7 @@ inner join Cars c on pc.CarID=c.ID where c.Make = 'Ford' ``` -can be accomplished with a Pilosa query like this (note that [Sum](/docs/query-language#sum) returns both the sum and count, from which the average is easily computed): +can be accomplished with a Pilosa query like this (note that [Sum](../query-language/#sum) returns a json object containing both the sum and count, from which the average is easily computed): ```pql Sum(Bitmap(frame="Car-Make", row=[Ford]), frame=Default, field=Age) @@ -105,17 +105,19 @@ Setting a time quantum on a frame creates extra views which allow Range queries ### Attribute -Attributes are arbitrary key/value pairs that can be associated to both rows or columns. This metadata is stored in a separate BoltDB data structure. +Attributes are arbitrary key/value pairs that can be associated with either rows or columns. This metadata is stored in a separate BoltDB data structure. + +Column-level attributes are common across an index. That is, each column attribute applies to all bits in the corresponding column, across all frames in an index. Row attributes apply to all bits in the corresponding row. ### Slice -Indexes are sharded into groups of columns called Slices - each Slice contains a fixed number of columns which is the SliceWidth. SliceWidth is a non-configurable constant set to 220. +Indexes are sharded into groups of columns called Slices. Each Slice contains a fixed number of columns, which is the SliceWidth. SliceWidth is a constant that can only be modified at compile time, and before ingesting data. The default value is 220. -Columns are sharded on a preset width, and each shard is referred to as a Slice. Slices are operated on in parallel, and they are evenly distributed across a cluster via a consistent hash algorithm. +Query operations run in parallel, and they are evenly distributed across a cluster via a consistent hash algorithm. ### View -Views represent the various data layouts within a Frame. The primary View is called Standard, and it contains the typical Row and Column data. Time-based Views are automatically generated for each time quantum. Views are internally managed by Pilosa, and never exposed directly via the API. This simplifies the functional interface from the physical data representation. +Views represent the various data layouts within a Frame. The primary View is called Standard, and it contains the typical Row and Column data. Time-based Views are automatically generated for each time quantum. Views are internally managed by Pilosa, and never exposed directly via the API. #### Standard @@ -123,7 +125,7 @@ The standard View contains the same Row/Column format as the input data. #### Time Quantums -If a Frame has a time quantum, then Views are generated for each of the defined time segments. For example, for a frame with a time quantum of `YMD`, the following `SetBit()` queries will result in the data described in the illustration below: +If a Frame has a time quantum, then Views are generated for each of the defined time segments. For example, for a frame with a time quantum of `YMD`, the following `SetBit()` queries will result in the data described in the diagram below: ``` SetBit(frame="A", row=8, col=3, timestamp="2017-05-18T00:00") @@ -135,12 +137,11 @@ SetBit(frame="A", row=8, col=3, timestamp="2017-05-19T00:00") #### BSI Range-Encoding -Bit-Sliced Indexing (BSI) is the storage method Pilosa uses to represent multi-bit integers in a bitmap index. Integers are stored as n-bit, range-encoded -bit-sliced indexes of base-2, along with an additional bitmap indicating "not null". This means that a 16-bit integer will require 17 bitmaps: one for each 0-bit of the 16 bit-slice components (the 1-bit does not need to be stored because with range-encoding the highest bit position is always 1) and one for the non-null bitmap. Pilosa can evaluate `Range`, `Min`, `Max`, and `Sum` queries on these BSI integers. +Bit-Sliced Indexing (BSI) is the storage method Pilosa uses to represent multi-bit integers in a bitmap index. Integers are stored as n-bit, range-encoded bit-sliced indexes of base-2, along with an additional bitmap indicating "not null". This means that a 16-bit integer will require 17 bitmaps: one for each 0-bit of the 16 bit-slice components (the 1-bit does not need to be stored because with range-encoding the highest bit position is always 1) and one for the non-null bitmap. Pilosa can evaluate `Range`, `Min`, `Max`, and `Sum` queries on these BSI integers. -Internally Pilosa stores each BSI `field` as a `view` within a `frame`. The rows of the `view` are composed of the base-2 representation of the integer. Pilosa manages the base-2 offset and translation that efficiently packs the integer value within the minimum set of rows. +Internally Pilosa stores each BSI `field` as a `view` within a `frame`. The rows of the `view` contain the base-2 representations of the integer values. Pilosa manages the base-2 offset and translation that efficiently packs the integer value within the minimum set of rows. -For example, the following `SetFieldValue()` queries will result in the data described in the illustration below: +For example, the following `SetFieldValue()` queries will result in the data described in the diagram below: ``` SetFieldValue(col=1, frame="A", field0=1) diff --git a/docs/examples.md b/docs/examples.md index d7830a2a8..42476ab57 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -123,7 +123,7 @@ These same objects are represented in the JSON definition file: } ``` -Here, we define a list of Mappers, each including a name, which we use to refer to the mapper later, in the list of BitMappers. We can also do this with Parsers, but a few simple Parsers that need no configuration are available by default. We also have a list of Fields, which is simply a map of field names to column indices. We use these names in the BitMapper definitions to keep things human-readable. +Here, we define a list of Mappers, each including a name, which we use to refer to the mapper later, in the list of BitMappers. We can also do this with Parsers, but a few simple Parsers that need no configuration are available by default. We also have a list of Fields, which is simply a map of field names (in the source data) to column indices (in Pilosa). We use these names in the BitMapper definitions to keep things human-readable. **total_amount_dollars:** Here we use the rounding mapping again, so each row represents rides with a total cost that rounds to the row's ID. The BitMapper definition is very similar to the previous one. @@ -197,12 +197,16 @@ for pcount, topn in zip(pcounts, resp.json()['results']): average_amounts.append(float(wsum)/count) ``` +
+Note that the BSI-powered Sum query now provides a an alternative approach to this kind of query. +
+ For more examples and details, see this [ipython notebook](https://github.com/pilosa/notebooks/blob/master/taxi-use-case.ipynb). ### Chemical similarity search
-This example uses the inverse frames feature, which is deprecated as of v0.9.0. This will soon be updated to reflect the current Pilosa API. +This example uses the inverse frames feature, which is deprecated as of v0.9.0. The example will soon be updated to reflect the current Pilosa API.
#### Overview diff --git a/docs/getting-started.md b/docs/getting-started.md index 570002b6d..358e6ee9f 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -39,7 +39,7 @@ curl localhost:10101/status ### Sample Project -In order to better understand Pilosa's capabilities, we will create a sample project called "Star Trace" containing information about the top 1,000 most recently updated Github repositories which have "go" in their name. The Star Trace index will include data points such as programming language, tags, and stargazers—people who have starred a project. +In order to better understand Pilosa's capabilities, we will create a sample project called "Star Trace" containing information about 1,000 popular Github repositories which have "go" in their name. The Star Trace index will include data points such as programming language, tags, and stargazers—people who have starred a project. Although Pilosa doesn't keep the data in a tabular format, we still use the terms "columns" and "rows" when describing the data model. We put the primary objects in columns, and the properties of those objects in rows. For example, the Star Trace project will contain an index called "repository" which contains columns representing Github repositories, and rows representing properties like programming languages and tags. We can better organize the rows by grouping them into sets called Frames. So the "repository" index might have a "languages" frame as well as a "tags" frame. You can learn more about indexes and frames in the [Data Model](../data-model/) section of the documentation. @@ -225,7 +225,7 @@ curl localhost:10101/index/repository/query \ ``` Please note that while user ID 99999 may not be sequential with the other column IDs, it is still a relatively low number. -Don't try to use arbitrary 64-bit integers as column or row IDs in Pilosa - this will lead to poor performance, out of memory errors, and more. +Don't try to use arbitrary 64-bit integers as column or row IDs in Pilosa - this will lead to problems such as poor performance and out of memory errors. diff --git a/docs/query-language.md b/docs/query-language.md index 7d9f22886..a99ac96a5 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -19,7 +19,7 @@ This section will provide a detailed reference and examples for the Pilosa Query {"results":[...]} ``` -There will be one item in the `results` array for each PQL query in the request. The type of each item in the array will depend on the type of query - each query in the reference below lists it's result type. +There will be one item in the `results` array for each PQL query in the request. The type of each item in the array will depend on the type of query - each query in the reference below lists its result type. #### Conventions @@ -79,7 +79,7 @@ A return value of `false` indicates that the bit was already set to 1 and nothin SetBit(frame="stargazer", col=10, row=1) ``` -This query illustrates setting a bit in the stargazer frame. User with id=1 has starred repository with id=10. +This sets a bit in the stargazer frame, representing that the user with id=1 has starred the repository with id=10. SetBit also supports providing a timestamp. To write the date that a user starred a repository. ```