diff --git a/docs/administration.md b/docs/administration.md index accbc46de..6764823e9 100644 --- a/docs/administration.md +++ b/docs/administration.md @@ -24,19 +24,19 @@ Pilosa holds all row/column bitmap data in main memory. While this data is compr #### CPUs -Pilosa is a concurrent application written in Go and can take full advantage of multicore machines. The main unit of parallelism is the [slice](../data-model/#slice), so a single query will only use a number of cores up to the number of slices stored on that host. Multiple queries can still take advantage of multiple cores as well though, so tuning in this area is dependent on the expected workload. +Pilosa is a concurrent application written in Go and can take full advantage of multicore machines. The main unit of parallelism is the [shard](../data-model/#shard), so a single query will only use a number of cores up to the number of shards stored on that host. Multiple queries can still take advantage of multiple cores as well, so tuning in this area is dependent upon the expected workload. #### Disk -Even though the main dataset is in memory Pilosa does back up to disk frequently. We recommend SSDs—especially if you have a write heavy application. +Even though the main dataset is in memory Pilosa backs up to disk frequently. We recommend SSDs—especially if you have a write-heavy application. #### Network -Pilosa is designed to be a distributed application, with data replication shared across the cluster. As such every write and read needs to communicate with several nodes. Therefore fast internode communication is essential. If using a service like AWS we recommend that all node exist in the same region and availability zone. The inherent latency of spreading a Pilosa cluster across physical regions it not usually worth the redundancy protection. Since Pilosa is designed to be an indexing service there already should be a system of record, or ability to rebuild a cluster quickly from backups. +Pilosa is designed to be a distributed application, with data replication replicated across the cluster. As such, every write and read needs to communicate with several nodes. Therefore fast internode communication is essential. If using a service like AWS we recommend that all nodes exist in the same region and availability zone. The inherent latency of spreading a Pilosa cluster across physical regions is not usually worth the redundancy protection. Since Pilosa is designed to be an indexing service there should already be a system of record, or ability to rebuild a cluster quickly from backups. #### Overview -While Pilosa does have some high system requirements it is not a best practice to set up a cluster with the fewest, largest machines available. You want an evenly distributed load across several nodes in a cluster to easily recover from a single node failure, and have the resource capacity to handle a missing node until it's repaired or replaced. Nor is it advisable to have many small machines. The internode network traffic will become a bottleneck. You can always add nodes later, but that does require some down time. +While Pilosa does have some high system requirements it is not a best practice to set up a cluster with the fewest, largest machines available. You want an evenly distributed load across several nodes in a cluster to easily recover from a single node failure, and have the resource capacity to handle a missing node until it's repaired or replaced. Nor is it advisable to have many small machines, as the internode network traffic will become a bottleneck. You can always add nodes later, but that does require some down time. ### Open File Limits @@ -56,23 +56,23 @@ When importing large datasets remember it is much faster to pre sort the data by pilosa import --sort -i project -f stargazer project-stargazer.csv ``` -##### Importing Field Values +##### Importing Integer Values -If you are using [BSI Range-Encoding](../data-model/#bsi-range-encoding) field values, you can import field values for a single frame and single field using `--field`. The CSV file should be in the format `Column,Value`. +If you are using [integer](../data-model/#bsi-range-encoding) field values, the CSV file should be in the format `Column,Value`. ``` -pilosa import -i project -f stargazer --field star_count project-stargazer-counts.csv +pilosa import -i project -f stargazer-counts project-stargazer-counts.csv ```
-

Note that you must first create a frame and a field. View Create Frame for more details.

+

Note that you must first create a field. View Create Field for more details. The `-e` flag can create the necessary schema when using a field of type "set".

#### Exporting -Exporting data to csv can be performed on a live instance of Pilosa. You need to specify the index and the frame. The API also expects the slice number, but the `pilosa export` sub command will export all slices within a Frame. The data will be in csv format `Row,Column` and sorted by column. +Exporting data to csv can be performed on a live instance of Pilosa. You need to specify the index and the field. The API also expects the slice number, but the `pilosa export` sub command will export all slices within a field. The data will be in csv format `Row,Column` and sorted by column. ```request -curl "http://localhost:10101/export?index=repository&frame=stargazer&slice=0" \ +curl "http://localhost:10101/export?index=repository&field=stargazer&slice=0" \ --header "Accept: text/csv" ``` ```response @@ -132,8 +132,8 @@ Pilosa v0.9 adds two new files to the data directory, an `.id` file and a `.topo **Application changes**: 1. Row and column labels were deprecated in Pilosa v0.8, and removed in Pilosa v0.9. Make sure that your application does not attempt to use a custom row or column label, as they are no longer supported. -2. If your application relies on the implicit creation of [time quantums](../glossary/#time-quantum) by inheriting the time-quantum setting of the index, you must begin explicitly enabling the time quantum per-frame, as index-level time-quantums have been removed. -3. Inverse frames have been deprecated, removed from docs, and will be unsupported in the next release. +2. If your application relies on the implicit creation of [time quantums](../glossary/#time-quantum) by inheriting the time-quantum setting of the index, you must begin explicitly enabling the time quantum per-field, as index-level time-quantums have been removed. +3. Inverse fields have been deprecated, removed from docs, and will be unsupported in the next release. ### Resizing the Cluster @@ -211,7 +211,7 @@ curl localhost:10101/cluster/resize/set-coordinator \ ### Backup/restore -Pilosa continuously writes out the in-memory bitmap data to disk. This data is organized by Index->Frame->Views->Fragment->numbered slice files. These data files can be routinely backed up to restore nodes in a cluster. +Pilosa continuously writes out the in-memory bitmap data to disk. This data is organized by Index->Field->Views->Fragment->numbered slice files. These data files can be routinely backed up to restore nodes in a cluster. Depending on the size of your data you have two options. For a small dataset you can rely on the periodic anti-entropy sync process to replicate existing data back to this node. @@ -230,11 +230,11 @@ Note: This will only work when the replication factor is >= 2 - To accomplish this you will first need: - List of all indexes on your cluster - - List of all frames in your indexes + - List of all fields in your indexes - Max slice per index, listed in the `/slices/max` endpoint - With this information you can query the `/internal/fragment/nodes` endpoint and iterate over each slice - Using the list of slices owned by this node you will then need to manually: - - setup a directory structure similar to the other nodes with a path for each Index/Frame + - setup a directory structure similar to the other nodes with a path for each Index/Field - copy each owned slice for an existing node to this new node - Modify the cluster config file to replace the previous node address with the new node address. - Restart the cluster @@ -249,10 +249,10 @@ Each Pilosa cluster is configured by default to share anonymous usage details wi - **Cluster:** List of nodes in the cluster. - **NumNodes:** Number of nodes in the cluster. - **NumCPU:** Number of cores per node -- **BSIEnabled:** Bit Slice Index Frames in use. -- **TimeQuantumEnabled:** Time Quantum Frames in use. +- **BSIEnabled:** Bit Sliced Index Fields in use. +- **TimeQuantumEnabled:** Time Quantum Fields in use. - **NumIndexes:** Number of indexes in the Cluster. -- **NumFrames:** Number of frames in the Cluster. +- **NumFields:** Number of fields in the Cluster. - **NumSlices:** Number of slices in the Cluster. - **NumViews:** Number of views in the Cluster. - **OpenFiles:** Open file handle count. @@ -274,7 +274,7 @@ StatsD Tags adhere to the DataDog format (key:value), and we tag the following: - NodeID - Index -- Frame +- Field - View - Slice @@ -282,7 +282,7 @@ StatsD Tags adhere to the DataDog format (key:value), and we tag the following: We currently track the following events - **Index:** The creation of a new index. -- **Frame:** The creation of a new frame. +- **Field:** The creation of a new field. - **MaxSlice:** The creation of a new Slice. - **SetBit:** Count of set bits. - **ClearBit:** Count of cleared bits. diff --git a/docs/api-reference.md b/docs/api-reference.md index f94cc5009..41e0288ce 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -17,7 +17,7 @@ Returns the schema of all indexes in JSON. curl -XGET localhost:10101/index ``` ``` response -{"indexes":[{"name":"user","frames":[{"name":"collab"}]}]} +{"indexes":[{"name":"user","fields":[{"name":"collab"}]}]} ``` ### List index schema @@ -30,7 +30,7 @@ Returns the schema of the specified index in JSON. curl -XGET localhost:10101/index/user ``` ``` response -{"index":{"name":"user"}, "frames":[{"name":"collab"}]}]} +{"name":"user", "fields":[{"name":"collab"}]} ``` ### Create index @@ -43,7 +43,7 @@ Creates an index with the given name. curl -XPOST localhost:10101/index/user ``` ``` response -{} +{"success":true} ``` ### Remove index @@ -56,7 +56,7 @@ Removes the given index. curl -XDELETE localhost:10101/index/user ``` ``` response -{} +{"success":true} ``` ### Query index @@ -68,10 +68,10 @@ Sends a [query](../query-language/) to the Pilosa server with the given index. T ``` request curl localhost:10101/index/user/query \ -X POST \ - -d 'Bitmap(frame="language", row=5)' + -d 'Row(language=5)' ``` ``` response -{"results":[{"attrs":{},"bits":[100]}]} +{"results":[{"attrs":{},"columns":[100]}]} ``` In order to send protobuf binaries in the request and response, set `Content-Type` and `Accept` headers to: `application/x-protobuf`. @@ -83,87 +83,66 @@ The query is executed for all [slices](../data-model/#slice) by default. To use ``` request curl "localhost:10101/index/user/query?columnAttrs=true&slices=0,1" \ -X POST \ - -d 'Bitmap(frame="language", row=5)' + -d 'Row(language=5)' ``` ``` response { - "results":[{"attrs":{},"bits":[100]}], + "results":[{"attrs":{},"columns":[100]}], "columnAttrs":[{"id":100,"attrs":{"name":"Klingon"}}] } ``` -By default, all bits and attributes (*for `Bitmap` queries only*) are returned. In order to suppress returning bits, set `excludeBits` query argument to `true`; to suppress returning attributes, set `excludeAttrs` query argument to `true`. +By default, all bits and attributes (*for `Row` queries only*) are returned. In order to suppress returning bits, set `excludeBits` query argument to `true`; to suppress returning attributes, set `excludeAttrs` query argument to `true`. -### Create frame +### Create field -`POST /index//frame/` +`POST /index//field/` -Creates a frame in the given index with the given name. +Creates a field in the given index with the given name. The request payload is in JSON, and may contain the `options` field. The `options` field is a JSON object which may contain the following fields: -* `timeQuantum` (string): [Time Quantum](../data-model/#time-quantum) for this frame. -* `cacheType` (string): [ranked](../data-model/#ranked) or [LRU](../data-model/#lru) caching on this frame. Default is `lru`. +* `timeQuantum` (string): [Time Quantum](../data-model/#time-quantum) for this field. +* `cacheType` (string): [ranked](../data-model/#ranked) or [LRU](../data-model/#lru) caching on this field. Default is `lru`. * `cacheSize` (int): Number of rows to keep in the cache. Default 50,000. * `fields` (array): List of range-encoded [fields](../data-model/#bsi-range-encoding). Each individual `field` contains the following: * `name` (string): Field name. -* `type` (string): Field type, currently only "int" is supported. +* `type` (string): Field type, "set", "int" or "time". * `min` (int): Minimum value allowed for this field. * `max` (int): Maximum value allowed for this field. Integer fields are stored as n-bit range-encoded values. Pilosa supports 63-bit, signed integers with values between `min` and `max`. ``` request -curl localhost:10101/index/user/frame/language -X POST +curl localhost:10101/index/user/field/language -X POST ``` ``` response -{} +{"success":true} ``` ``` request -curl localhost:10101/index/repository/frame/stats \ +curl localhost:10101/index/repository/field/stats \ -X POST \ -d '{"fields": [{"name": "pullrequests", "type": "int", "min": 0, "max": 1000000}]}' ``` ``` response -{} +{"success":true} ``` -### Remove frame +### Remove field -`DELETE /index//frame/` +`DELETE /index//field/` -Removes the given frame. +Removes the given field. ``` request -curl -XDELETE localhost:10101/index/user/frame/language +curl -XDELETE localhost:10101/index/user/field/language ``` ``` response -{} -``` - -### Create Field - -`POST /index//frame//field/` - -Creates a new field to store integer values in the given frame. - -The request payload is JSON, and it must contain the fields `type`, `min`, `max`. - -* `type` (string): Field type, currently only "int" is supported. -* `min` (int): Minimum value allowed for this field. -* `max` (int): Maximum value allowed for this field. - -``` request -curl localhost:10101/index/repository/frame/stats/field/pullrequests \ - -X POST \ - -d '{"type": "int", "min": 0, "max": 1000000}' -``` -``` response -{} +{"success":true} ``` ### Get version @@ -191,7 +170,7 @@ in a multi-node cluster, the cache is only recalculated on the node that receives the request. ``` request -curl -XGET localhost:10101/recalculate-caches +curl -XPOST localhost:10101/recalculate-caches ``` Response: `204 No Content` diff --git a/docs/data-model.md b/docs/data-model.md index ad6a9ea9d..bdcec4861 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -6,7 +6,7 @@ nav = [ "Index", "Column", "Row", - "Frame", + "Field", "Time Quantum", "Attribute", "Slice", @@ -22,7 +22,7 @@ The central component of Pilosa's data model is a boolean matrix. Each cell in t Rows and columns can represent anything (they could even represent the same set of things - a [bigraph](https://en.wikipedia.org/wiki/Bigraph)). Pilosa can associate arbitrary key/value pairs (referred to as attributes) to rows and columns, but queries and storage are optimized around the core matrix. -Pilosa lays out data first in rows, so queries which get all the set bits in one or many rows, or compute a combining operation on multiple rows such as Intersect or Union are the fastest. Pilosa categorizes rows into different *fields* and quickly retrieves the top rows in a field sorted by the number of bits set in each row. +Pilosa lays out data first in rows, so queries which get all the set bits in one or many rows, or compute a combining operation on multiple rows such as Intersect or Union are the fastest. Pilosa categorizes rows into different *fields* and quickly retrieves the top rows in a field sorted by the number of columns set in each row. Please note that Pilosa is most performant when row and column IDs are sequential starting from 0. You can deviate from this to some degree, but setting a bit with column ID 263 on a single-node cluster, for example, will not work well due to memory limitations. @@ -43,12 +43,14 @@ Row ids are sequential increasing integers namespaced to each Field within an In ### Field -Fields are used to segment rows within an index, for example to define different functional groups. A Pilosa field might correspond to a single field in a relational table, where each row in a standard Pilosa field represents a single possible value of the relational field. Similarly, a field with BSI values could represent all possible integer values of a relational field. +Fields are used to segment rows within an index, for example to define different functional groups. A Pilosa field might correspond to a single field in a relational table, where each row in a standard Pilosa field represents a single possible value of the relational field. Similarly, an integer field could represent all possible integer values of a relational field. #### Relational Analogy The Pilosa index is a flexible structure; it can represent any sort of high-cardinality binary matrix. We have explored a number of modeling patterns in Pilosa use cases; one accessible example is a direct analogy to the relational model, summarized here. +TODO diagram showing a few rows of a relational table and corresponding pilosa index + Entities: Relational | Pilosa @@ -64,9 +66,9 @@ Simple queries: Relational | Pilosa ---------------------------------------------|------------------------------------ - `select ID from People where Name = 'Bob'` | `Bitmap(frame=Name, row=[Bob])` - `select ID from People where Age > 30` | `Range(frame=Default, Age > 30)` - `select ID from People where Member = true` | `Bitmap(frame=Member, row=[true])` + `select ID from People where Name = 'Bob'` | `Row(Name="Bob")` + `select ID from People where Age > 30` | `Range(Age > 30)` + `select ID from People where Member = true` | `Row(Member=0)` # TODO this is unfortunate In the relational model, joins are often necessary. Because Pilosa supports extremely high cardinality in both rows and columns, many types of joins are accomplished with basic Pilosa queries across multiple fields. For example, this SQL join: @@ -80,7 +82,7 @@ where c.Make = 'Ford' can be accomplished with a Pilosa query like this (note that [Sum](../query-language/#sum) returns a json object containing both the sum and count, from which the average is easily computed): ```pql -Sum(Bitmap(frame="Car-Make", row=[Ford]), frame=Default, field=Age) +Sum(Row(Car-Make="Ford"), field=Age) ``` This is one major component of Pilosa's ability to combine relationships from multiple data stores. @@ -125,11 +127,11 @@ The standard View contains the same Row/Column format as the input data. #### Time Quantums -If a Field has a time quantum, then Views are generated for each of the defined time segments. For example, for a field with a time quantum of `YMD`, the following `SetBit()` queries will result in the data described in the diagram below: +If a Field has a time quantum, then Views are generated for each of the defined time segments. For example, for a field with a time quantum of `YMD`, the following `Set()` queries will result in the data described in the diagram below: ``` -SetBit(frame="A", row=8, col=3, timestamp="2017-05-18T00:00") -SetBit(frame="A", row=8, col=3, timestamp="2017-05-19T00:00") +Set(3, A=8, 2017-05-18T00:00) +Set(3, A=8, 2017-05-19T00:00) ``` ![time quantum field diagram](/img/docs/field-time-quantum.svg) @@ -141,15 +143,15 @@ Bit-Sliced Indexing (BSI) is the storage method Pilosa uses to represent multi-b Internally Pilosa stores each BSI (TODO!!!!!) `field` as a `view` within a `frame`. The rows of the `view` contain the base-2 representations of the integer values. Pilosa manages the base-2 offset and translation that efficiently packs the integer value within the minimum set of rows. -For example, the following `SetFieldValue()` queries will result in the data described in the diagram below: +For example, the following `Set()` queries executed against BSI fields will result in the data described in the diagram below: ``` -SetFieldValue(col=1, frame="A", field0=1) -SetFieldValue(col=2, frame="A", field0=2) -SetFieldValue(col=3, frame="A", field0=3) -SetFieldValue(col=4, frame="A", field0=7) -SetFieldValue(col=2, frame="A", field1=1) -SetFieldValue(col=3, frame="A", field1=6) +Set(1, A=1) +Set(2, A=2) +Set(3, A=3) +Set(4, A=7) +Set(2, B=1) +Set(3, B=6) ``` ![BSI field diagram](/img/docs/field-bsi.svg) diff --git a/docs/examples.md b/docs/examples.md index 6c4f70cc1..1660e8592 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -33,9 +33,9 @@ The NYC taxi data is comprised of a number of csv files listed here: http://www. * Dropoff time: timestamp * Pickup time: timestamp -We import these fields, creating one or more Pilosa frames from each of them: +We import these fields, creating one or more Pilosa fields from each of them: -frame |mapping +field |mapping ------------|--------------------- cab_type |direct map of enum int → row ID dist_miles |round(dist) → row ID @@ -52,24 +52,24 @@ pickup_month |month(timestamp) → row ID pickup_day |day(timestamp) → row ID pickup_time |time of day mapped to one of 48 half-hour buckets → row ID -We also created two extra frames that represent the duration and average speed of each ride: +We also created two extra fields that represent the duration and average speed of each ride: -frame |mapping +field |mapping --------------------|------------- duration_minutes |round(drop_timestamp - pickup_timestamp) → row ID speed_mph |round(dist_miles / (drop_timestamp - pickup_timestamp)) → row ID #### Mapping -Each column that we want to use must be mapped to a combination of frames and row IDs according to some rule. There are many ways to approach this mapping, and the taxi dataset gives us a good overview of possibilities. +Each column that we want to use must be mapped to a combination of fields and row IDs according to some rule. There are many ways to approach this mapping, and the taxi dataset gives us a good overview of possibilities. -##### 0 columns → 1 frame +##### 0 columns → 1 field -**cab_type**: contains one row for each type of cab. Each column, representing one ride, has a bit set in exactly one row of this frame. The mapping is a simple enumeration, for example yellow=0, green=1, etc. The values of the bits in this frame are determined by the source of the data. That is, we're importing data from several disparate sources: NYC yellow taxi cabs, NYC green taxi cabs, and Uber cars. For each source, the single row to be set in the cab_type frame is constant. +**cab_type**: contains one row for each type of cab. Each column, representing one ride, has a bit set in exactly one row of this field. The mapping is a simple enumeration, for example yellow=0, green=1, etc. The values of the bits in this field are determined by the source of the data. That is, we're importing data from several disparate sources: NYC yellow taxi cabs, NYC green taxi cabs, and Uber cars. For each source, the single row to be set in the cab_type field is constant. -##### 1 column → 1 frame +##### 1 column → 1 field -The following three frames are mapped in a simple direct way from single columns of the original data. +The following three fields are mapped in a simple direct way from single columns of the original data. **dist_miles:** each row represents rides of a certain distance. The mapping is simple: as an example, row 1 represents rides with a distance in the interval [0.5, 1.5]. That is, we round the floating point value of distance to an integer, and use that as the row ID directly. Generally, the mapping from a floating point value to a row ID could be arbitrary. The rounding mapping is concise to implement, which simplifies importing and analysis. As an added bonus, it's human-readable. We'll see this pattern used several times. @@ -84,7 +84,7 @@ lfm := pdk.LinearFloatMapper{ `Min` and `Max` define the linear function, and `Res` determines the maximum allowed value for the output row ID - we chose these values to produce a “round to nearest integer” behavior. Other predefined mappers have their own specific parameters, usually two or three. -This mapper function is the core operation, but we need a few other pieces to define the overall process, which is encapsulated in the BitMapper object. This object defines which field(s) of the input data source to use (`Fields`), how to parse them (`Parsers`), what mapping to use (`Mapper`), and the name of the frame to use (`Frame`). +This mapper function is the core operation, but we need a few other pieces to define the overall process, which is encapsulated in the BitMapper object. This object defines which field(s) of the input data source to use (`Fields`), how to parse them (`Parsers`), what mapping to use (`Mapper`), and the name of the field to use (`Frame`). TODO update so this makes sense ```go pdk.BitMapper{ Frame: "dist_miles", @@ -129,27 +129,27 @@ Here, we define a list of Mappers, each including a name, which we use to refer **passenger_count:** This column contains small integers, so we use one of the simplest possible mappings: the column value is the row ID. -##### 1 column → multiple frames +##### 1 column → multiple fields When working with a composite data type like a timestamp, there are plenty of mapping options. In this case, we expect to see interesting periodic trends, so we want to encode the cyclic components of time in a way that allows us to look at them independently during analysis. -We do this by storing time data in four separate frames for each timestamp: one each for the year, month, day, and time of day. The first three are mapped directly. For example, a ride with a date of 2015/06/24 will have a bit set in row 2015 of frame "year", row 6 of frame "month", and row 24 of frame "day". +We do this by storing time data in four separate fields for each timestamp: one each for the year, month, day, and time of day. The first three are mapped directly. For example, a ride with a date of 2015/06/24 will have a bit set in row 2015 of field "year", row 6 of field "month", and row 24 of field "day". -We might continue this pattern with hours, minutes, and seconds, but we don't have much use for that level of precision here, so instead we use a "bucketing" approach. That is, we pick a resolution (30 minutes), divide the day into buckets of that size, and create a row for each one. So a ride with a time of 6:45AM has a bit set in row 13 of frame "time_of_day". +We might continue this pattern with hours, minutes, and seconds, but we don't have much use for that level of precision here, so instead we use a "bucketing" approach. That is, we pick a resolution (30 minutes), divide the day into buckets of that size, and create a row for each one. So a ride with a time of 6:45AM has a bit set in row 13 of field "time_of_day". -We do all of this for each timestamp of interest, one for pickup time and one for dropoff time. That gives us eight total frames for two timestamps: pickup_year, pickup_month, pickup_day, pickup_time, drop_year, drop_month, drop_day, drop_time. +We do all of this for each timestamp of interest, one for pickup time and one for dropoff time. That gives us eight total fields for two timestamps: pickup_year, pickup_month, pickup_day, pickup_time, drop_year, drop_month, drop_day, drop_time. -##### Multiple columns → 1 frame +##### Multiple columns → 1 field The ride data also contains geolocation data: latitude and longitude for both pickup and dropoff. We just want to be able to produce a rough overview heatmap of ride locations, so we use a grid mapping. We divide the area of interest into a 100x100 grid in latitude-longitude space, label each cell in this grid with a single integer, and use that integer as the row ID. -We do all of this for each location of interest, one for pickup and one for dropoff. That gives us two frames for two locations: pickup_grid_id, drop_grid_id. +We do all of this for each location of interest, one for pickup and one for dropoff. That gives us two fields for two locations: pickup_grid_id, drop_grid_id. Again, there are many mapping options for location data. For example, we might convert to a different coordinate system, apply a projection, or aggregate locations into real-world regions such as neighborhoods. Here, the simple approach is sufficient. ##### Complex mappings -We also anticipate looking for trends in ride duration and speed, so we want to capture this information during the import process. For the frame `duration_minutes`, we compute a row ID as `round((drop_timestamp - pickup_timestamp).minutes)`. For the frame `speed_mph`, we compute row ID as `round(dist_miles / (drop_timestamp - pickup_timestamp).minutes)`. These mapping calculations are straightforward, but because they require arithmetic operations on multiple columns, they are a bit too complex to capture in the basic mappers available in PDK. Instead, we define custom mappers to do the work: +We also anticipate looking for trends in ride duration and speed, so we want to capture this information during the import process. For the field `duration_minutes`, we compute a row ID as `round((drop_timestamp - pickup_timestamp).minutes)`. For the field `speed_mph`, we compute row ID as `round(dist_miles / (drop_timestamp - pickup_timestamp).minutes)`. These mapping calculations are straightforward, but because they require arithmetic operations on multiple columns, they are a bit too complex to capture in the basic mappers available in PDK. Instead, we define custom mappers to do the work: ```go durm := pdk.CustomMapper{ Func: func(fields ...interface{}) interface{} { @@ -172,7 +172,7 @@ Now we can run some example queries. Count per cab type can be retrieved, sorted, with a single PQL call. ```request -TopN(frame=cab_type) +TopN(cab_type) ``` ```response {"results":[[{"id":1,"count":1992943},{"id":0,"count":7057}]]} @@ -181,7 +181,7 @@ TopN(frame=cab_type) High traffic location IDs can be retrieved with a similar call. These IDs correspond to latitude, longitude pairs, which can be recovered from the mapping that generates the IDs. ```request -TopN(frame=pickup_grid_id) +TopN(pickup_grid_id) ``` ```response {"results":[[{"id":5060,"count":40620},{"id":4861,"count":38145},{"id":4962,"count":35268},...]]} @@ -193,7 +193,7 @@ Average of `total_amount` per `passenger_count` can be computed with some postpr queries = '' pcounts = range(10) for i in pcounts: - queries += "TopN(Bitmap(id=%d, frame='passenger_count'), frame=total_amount_dollars)" % i + queries += "TopN(Row(passenger_count=%d), total_amount_dollars)" % i resp = requests.post(qurl, data=queries) average_amounts = [] @@ -209,6 +209,8 @@ Note that the BSI-powered @@ -326,3 +328,6 @@ python benchmarks.py -id 6223 As Matt Swain’s blog post also did a great job using mongoDB for chemical similarity search, we compared benchmark on 500000 molecules between mongoDB aggregation framework with Pilosa. Both using the same molecule, Morgan fingerprint folded to fixed lengths of 4096 bits and were run on a MacBook Pro with a 2.8 GHz 2-core Intel Core i7 processor, memory of 16 GB 1600 MHz DDR3, single host cluster + + +--> diff --git a/docs/getting-started.md b/docs/getting-started.md index 7f5759844..6f935f735 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -47,7 +47,7 @@ Although Pilosa doesn't keep the data in a tabular format, we still use the term #### Create the Schema Note: -The queries in this section which are used to set up the indexes in Pilosa just return the empty object on success: `{}` - if you would like to verify that a query worked as you expected, you can request the schema as follows: +If at any time you want to verify the data structure, you can request the schema as follows: ``` request curl localhost:10101/schema @@ -61,7 +61,7 @@ Before we can import data or run queries, we need to create our indexes and the curl localhost:10101/index/repository -X POST ``` ``` response -{} +{"success":true} ``` Let's create the `stargazer` field which has user IDs of stargazers as its rows: @@ -71,10 +71,10 @@ curl localhost:10101/index/repository/field/stargazer \ -d '{"options": {"type": "time", "timeQuantum": "YMD"}}' ``` ``` response -{} +{"success":true} ``` -Since our data contains time stamps for the time users starred repos, we set the field type to `time`. Time quantum is the resolution of the time we want to use, and we set it to `YMD` (year, month, day) for `stargazer`. +Since our data contains time stamps whcih represent the time users starred repos, we set the field type to `time`. Time quantum is the resolution of the time we want to use, and we set it to `YMD` (year, month, day) for `stargazer`. Next up is the `language` field, which will contain IDs for programming languages: ``` request @@ -82,7 +82,7 @@ curl localhost:10101/index/repository/field/language \ -X POST ``` ``` response -{} +{"success":true} ``` The `language` is a `set` field, but since the default field type is `set`, we didn't specify it in field options. @@ -119,7 +119,7 @@ Which repositories did user 14 star: ``` request curl localhost:10101/index/repository/query \ -X POST \ - -d 'Bitmap(field="stargazer", row=14)' + -d 'Row(stargazer=14)' ``` ``` response { @@ -136,7 +136,7 @@ What are the top 5 languages in the sample data: ``` request curl localhost:10101/index/repository/query \ -X POST \ - -d 'TopN(field="language", n=5)' + -d 'TopN(language, n=5)' ``` ``` response { @@ -157,8 +157,8 @@ Which repositories were starred by user 14 and 19: curl localhost:10101/index/repository/query \ -X POST \ -d 'Intersect( - Bitmap(field="stargazer", row=14), - Bitmap(field="stargazer", row=19) + Row(stargazer=14), + Row(stargazer=19) )' ``` ``` response @@ -177,8 +177,8 @@ Which repositories were starred by user 14 or 19: curl localhost:10101/index/repository/query \ -X POST \ -d 'Union( - Bitmap(field="stargazer", row=14), - Bitmap(field="stargazer", row=19) + Row(stargazer=14), + Row(stargazer=19) )' ``` ``` response @@ -197,9 +197,9 @@ Which repositories were starred by user 14 and 19 and also were written in langu curl localhost:10101/index/repository/query \ -X POST \ -d 'Intersect( - Bitmap(field="stargazer", row=14), - Bitmap(field="stargazer", row=19), - Bitmap(field="language", row=1) + Row(stargazer=14), + Row(stargazer=19), + Row(language=1) )' ``` ``` response @@ -217,7 +217,7 @@ Set user 99999 as a stargazer for repository 77777: ``` request curl localhost:10101/index/repository/query \ -X POST \ - -d 'SetBit(field="stargazer", col=77777, row=99999)' + -d 'Set(77777, stargazer=99999)' ``` ``` response {"results":[true]} diff --git a/docs/glossary.md b/docs/glossary.md index 78ca7c1fe..eb5dd780e 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -12,19 +12,17 @@ nav = [] [Bit](../data-model/#overview): Bits are the fundamental unit of data in Pilosa. A bit lives in a [field](#field), at the intersection of a [row](#row) and [column](#column). -[Bitmap](../data-model/#overview): The on-disk and in-memory representation of a [row](#row). Implemented with [Roaring](#roaring-bitmap). `Bitmap` is also the basic [PQL](#pql) query for reading a Bitmap. +[Bitmap](../data-model/#overview): The on-disk and in-memory representation of a [row](#row). Implemented with [Roaring](#roaring-bitmap). [BSI](../data-model/#bsi-range-encoding) Bit-sliced indexing is the method Pilosa uses to represent multi-bit integers. Integer values are stored in [fields](#field), and can be used for [Range](#range-bsi), [Min](#min), [Max](#max), and [Sum](#sum) queries. -Cluster: A cluster consists of one or more [nodes](#node) which share a cluster configuration. The cluster also defines how data is [replicated](#replica) throughout and how internode communication is coordinated. Pilosa does not have a leader node, all data is evenly distributed, and any node can respond to queries. +Cluster: A cluster consists of one or more [nodes](#node) which share a cluster configuration. The cluster also defines how data is [replicated](#replica) and how internode communication is coordinated. Pilosa does not have a leader node, all data is evenly distributed, and any node can respond to queries. [Column](../data-model/#column): Columns are the fundamental horizontal data axis within Pilosa. Columns are global to all [fields](#field) within an [index](#index). -[Field](../data-model/#bsi-range-encoding): A group of rows used to store integer values with [BSI](#bsi), for use in [Range](#range-bsi) and [Sum](#sum) queries. - Fragment: A Fragment is the intersection of a [field](#field) and a [shard](#shard) in an [index](#index). -[Field](../data-model/#field): Fields are used to group [rows](#row) into different categories. Row IDs are namespaced by field such that the same row ID in a different field refers to a different row. For [ranked](#topn) fields, rows are kept in sorted order within the field. +[Field](../data-model/#field): Fields are used to group [rows](#row) into different categories. Row IDs are namespaced by field such that the same row ID in a different field refers to a different row. For [ranked](#topn) fields, rows are kept in sorted order within the field. Fields are one of three types: set, [int](#bsi), and time. For more information, see [data model](../data-model/) and [Creating fields](../api-reference/#create-field). [Frame](../data-model/#field): Prior to Pilosa 1.0, fields were known as frames. @@ -34,11 +32,11 @@ nav = [] [Jump Consistent Hash](https://arxiv.org/pdf/1406.2294v1.pdf): A fast, minimal memory, consistent hash algorithm that evenly distributes the workload even when the number of buckets changes. -[Max](../query-language/#max): A [PQL](#pql) query that returns the maximum integer value stored in [BSI](#bsi) [fields](#field). +[Max](../query-language/#max): A [PQL](#pql) query that returns the maximum integer value stored in an [integer](#bsi) [field](#field). -MaxShard: The total number of [shards](#shard) allocated to handle the current set of [columns](#column). This value is important for all [nodes](#node) to efficiently distribute queries. +MaxShard: The total number of [shards](#shard) allocated to handle the current set of [columns](#column). This value is important for all [nodes](#node) to efficiently distribute queries. MaxShard is zero-indexed, so if an index contains six shards, its MaxShard will be 5. -[Min](../query-language/#min): A [PQL](#pql) query that returns the minimum integer value stored in [BSI](#bsi) [fields](#field). +[Min](../query-language/#min): A [PQL](#pql) query that returns the minimum integer value stored in an [integer](#bsi) [field](#field). Node: An individual running instance of Pilosa server which belongs to a [cluster](#cluster). @@ -64,14 +62,12 @@ nav = [] ShardWidth: This is the number of [columns](#column) in a [shard](#shard). `ShardWidth` defaults to 220 or about one million. It can be modified, but only at compile time, and before ingesting any data. -[Sum](../query-language/#sum): A [PQL](#pql) query that returns the sum of integers stored in [BSI](#bsi) [fields](#field). +[Sum](../query-language/#sum): A [PQL](#pql) query that returns the sum of integers stored in an [integer](#bsi) [field](#field). -[Tanimoto](../examples/#chemical-similarity-search): Used for similarity queries on Pilosa data. The [Tanimoto Coefficient](https://en.wikipedia.org/wiki/Jaccard_index#Tanimoto_similarity_and_distance) between two [Bitmaps](#bitmap) A and B is the ratio of the size of their intersection to the size of their union (|A∩B|/|A∪B|). - -[Time quantum](../data-model/#time-quantum): Defines the granularity to be used for time [Range](#range) queries. +[Time quantum](../data-model/#time-quantum): Defines the granularity to be used for [Range](#range) queries on time [fields](#field). [TOML](https://github.com/toml-lang/toml): the language used for Pilosa's [configuration file](../configuration/). -[TopN](../query-language/#topn): A [PQL](#pql) query that returns a list of row IDs, sorted by the count of [bits](#bit) set in the [row](#row), within a specified [field](#field). +[TopN](../query-language/#topn): A [PQL](#pql) query that returns a list of rows, sorted by the count of [columns](#column) set in the [row](#row), within a specified [field](#field). [View](../data-model/#view): Views separate the different data layouts within a [Field](#field). The primary view is standard, which represents the typical [row](#row)/[column](#column) data. Time based field views are automatically generated for each [time quantum](#time-quantum). Views are internally managed by Pilosa, and never exposed directly via the API. This simplifies the functional interface by separating it from the physical data representation. diff --git a/docs/query-language.md b/docs/query-language.md index a598bf351..62c912ba4 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -1,4 +1,4 @@ -+++ +v+++ title = "Query Language" weight = 6 nav = [ @@ -29,13 +29,13 @@ There will be one item in the `results` array for each PQL query in the request. ##### Examples -Before running any of the example queries below, follow the instructions in the [Getting Started](../getting-started/) section to set up an index, frames, and populate them with some data. +Before running any of the example queries below, follow the instructions in the [Getting Started](../getting-started/) section to set up an index and fields, and to populate them with some data. -The examples just show the PQL quer(ies) needed - to run the query `SetBit(frame="stargazer", col=10, row=1)` against a server using curl, you would: +The examples just show the PQL quer(ies) needed - to run the query `Set(10, stargazer=1)` against a server using curl, you would: ``` request curl localhost:10101/index/repository/query \ -X POST \ - -d 'SetBit(frame="stargazer", col=10, row=1)' + -d 'Set(10, stargazer=1)' ``` ``` response {"results":[true]} @@ -43,28 +43,27 @@ curl localhost:10101/index/repository/query \ #### Arguments and Types -* `frame` The frame specifies on which Pilosa [frame](../glossary/#frame) the query will operate. Valid frame names are lower case strings; they start with an alphanumeric character, and contain only alphanumeric characters and `_-`. They must be 64 characters or less in length. -* `TIMESTAMP` This is a timestamp in quotes with the following format `"YYYY-MM-DDTHH:MM"` (e.g. "2006-01-02T15:04") +* `field` The field specifies on which Pilosa [field](../glossary/#field) the query will operate. Valid field names are lower case strings; they start with an alphanumeric character, and contain only alphanumeric characters and `_-`. They must be 64 characters or less in length. +* `TIMESTAMP` This is a timestamp in the following format `YYYY-MM-DDTHH:MM` (e.g. 2006-01-02T15:04) * `UINT` An unsigned integer (e.g. 42839) * `ATTR_NAME` Must be a valid identifier `[A-Za-z][A-Za-z0-9._-]*` * `ATTR_VALUE` Can be a string, float, integer, or bool. -* `BITMAP_CALL` Any query which returns a bitmap, such as `Bitmap`, `Union`, `Difference`, `Xor`, `Intersect`, `Range` +* `ROW_CALL` Any query which returns a row, such as `Row`, `Union`, `Difference`, `Xor`, `Intersect`, `Range` * `[]ATTR_VALUE` Denotes an array of `ATTR_VALUE`s. (e.g. `["a", "b", "c"]`) ### Write Operations -#### SetBit +#### Set **Spec:** ``` -SetBit(, , , - [timestamp=TIMESTAMP]) +Set(, field=, [TIMESTAMP]) ``` **Description:** -`SetBit` assigns a value of 1 to a bit in the binary matrix, thus associating the given row in the given frame with the given column. +`Set` assigns a value of 1 to a bit in the binary matrix, thus associating the given row (the `` value) in the given field with the given column. **Result Type:** boolean @@ -77,17 +76,17 @@ A return value of `false` indicates that the bit was already set to 1 and nothin Set the bit at row 1, column 10: ```request -SetBit(frame="stargazer", col=10, row=1) +Set(10, stargazer=1) ``` ```response {"results":[true]} ``` -This sets a bit in the stargazer frame, representing that the user with id=1 has starred the repository with id=10. +This sets a bit in the stargazer field, representing that the user with id=1 has starred the repository with id=10. -SetBit also supports providing a timestamp. To write the date that a user starred a repository: +Set also supports providing a timestamp. To write the date that a user starred a repository: ```request -SetBit(frame="stargazer", col=10, row=1, timestamp="2016-01-01T00:00") +Set(10, stargazer=1, 2016-01-01T00:00) ``` ```response {"results":[true]} @@ -95,24 +94,32 @@ SetBit(frame="stargazer", col=10, row=1, timestamp="2016-01-01T00:00") Set multiple bits in a single request: ```request -SetBit(frame="stargazer", col=10, row=1) SetBit(frame="stargazer", col=10, row=2) SetBit(frame="stargazer", col=20, row=1) SetBit(frame="stargazer", col=30, row=2) +Set(1, stargazer=10) Set(2, stargazer=10) Set(1, stargazer=20) Set(2, stargazer=30) ``` ```response {"results":[false,true,true,true]} ``` +Set the field "pullrequests" to integer value 2 at column 10: +```request +Set(10, pullrequests=2) +``` +```response +{"results":[true]} +``` + #### SetRowAttrs **Spec:** ``` -SetRowAttrs(, , +SetRowAttrs(, , , [ATTR_NAME=ATTR_VALUE ...]) ``` **Description:** -`SetRowAttrs` associates arbitrary key/value pairs with a row in a frame. Setting a value of `null`, without quotes, deletes an attribute. +`SetRowAttrs` associates arbitrary key/value pairs with a row in a field. Setting a value of `null`, without quotes, deletes an attribute. **Result Type:** null @@ -122,17 +129,17 @@ SetRowAttrs queries always return `null` upon success. Set attributes `username` and `active` on row 10: ```request -SetRowAttrs(frame="stargazer", row=10, username="mrpi", active=true) +SetRowAttrs(stargazer, 10, username="mrpi", active=true) ``` ```response {"results":[null]} ``` -Set username value and active status for user 10. These are arbitrary key/value pairs which have no meaning to Pilosa. You can see the attributes you've set on a row with a [Bitmap](../query-language/#bitmap) query like so `Bitmap(frame="stargazer", row=10)`. +Set username value and active status for user 10. These are arbitrary key/value pairs which have no meaning to Pilosa. You can see the attributes you've set on a row with a [Row](../query-language/#row) query like so `Row(stargazer=10)`. Delete attribute `username` on row 10: ```request -SetRowAttrs(frame="stargazer", row=10, username=null) +SetRowAttrs(stargazer, 10, username=null) ``` ```response {"results":[null]} @@ -143,7 +150,7 @@ SetRowAttrs(frame="stargazer", row=10, username=null) **Spec:** ``` -SetColumnAttrs(, , +SetColumnAttrs(, , [ATTR_NAME=ATTR_VALUE ...]) ``` @@ -154,13 +161,13 @@ SetColumnAttrs(, , **Result Type:** null -SetColumnAttrs queries always return `null` upon success. Setting a value of `null`, without quotes, deletes an attribute. To avoid confusion, `frame` cannot be used as an attribute name. +SetColumnAttrs queries always return `null` upon success. Setting a value of `null`, without quotes, deletes an attribute. **Examples:** Set attributes `stars`, `url`, and `active` on column 10: ```request -SetColumnAttrs(col=10, stars=123, url="http://projects.pilosa.com/10", active=true) +SetColumnAttrs(10, stars=123, url="http://projects.pilosa.com/10", active=true) ``` ```response {"results":[null]} @@ -170,13 +177,13 @@ Set url value and active status for project 10. These are arbitrary key/value pa ColumnAttrs can be requested by adding the URL parameter `columnAttrs=true` to a query. For example: ```request -curl localhost:10101/index/repository/query?columnAttrs=true -XPOST -d 'Bitmap(frame="stargazer", row=1)Bitmap(frame="stargazer", row=2)' +curl localhost:10101/index/repository/query?columnAttrs=true -XPOST -d 'Row(stargazer=1) Row(stargazer=2)' ``` ```response { "results":[ - {"attrs":{},"bits":[10,20]}, - {"attrs":{},"bits":[10,30]} + {"attrs":{},"cols":[10,20]}, + {"attrs":{},"cols":[10,30]} ], "columnAttrs":[ {"id":10,"attrs":{"active":true,"stars":123,"url":"http://projects.pilosa.com/10"}}, @@ -189,7 +196,7 @@ In this example, ColumnAttrs have been set on columns 10 and 20, but not column Delete the `url` attribute on column 10: ```request -SetColumnAttrs(col=10, url=null) +SetColumnAttrs(10, url=null) ``` ```response {"results":[null]} @@ -200,14 +207,14 @@ SetColumnAttrs(col=10, url=null) **Spec:** ``` -ClearBit(, , ) +Clear(, field=) ``` **Description:** -`ClearBit` assigns a value of 0 to a bit in the binary matrix, thus disassociating the given row in the given frame from the given column. +`Clear` assigns a value of 0 to a bit in the binary matrix, thus disassociating the given row in the given field from the given column. -Note that clearing bits from time views is not supported. +Note that clearing a column on a time field will remove all data for that column. **Result Type:** boolean @@ -217,9 +224,9 @@ A return value of `false` indicates that the bit was already set to 0 and nothin **Examples:** -Clear the bit at row 1 and column 10 in the stargazer frame: +Clear the bit at row 1 and column 10 in the stargazer field: ```request -ClearBit(frame="stargazer", col=10, row=1) +Clear(10, stargazer=1) ``` ```response {"results":[true]} @@ -227,79 +234,48 @@ ClearBit(frame="stargazer", col=10, row=1) This represents removing the relationship between the user with id=1 and the repository with id=10. -#### SetFieldValue - -**Spec:** - -``` -SetFieldValue(, , ) -``` - -**Description:** - -`SetFieldValue` assigns an integer value with the specified field name to the `col` in the given `frame`. - -**Result Type:** null - -SetFieldValue returns `null` upon success. - -**Examples:** - -Set the field value `pullrequest` to the value 2, on column 10 in frame `stats`: -```request -SetFieldValue(col=10, frame="stats", pullrequests=2) -``` -```response -{"results":[null]} -``` - -This represents setting the number of pull requests of repository 10 to 2. - -This example assumes the existence of the frame `stats` and the field `pullrequests`. See [frame creation](../api-reference/#create-frame) and [field creation](../api-reference/#create-field) for more information. - - ### Read Operations -#### Bitmap +#### Row **Spec:** ``` -Bitmap(, ( | =UINT)) +Row(field=) ``` **Description:** -`Bitmap` retrieves the indices of all the set bits in a row or column based on whether the row or column argument is provided in the query. It also retrieves any attributes set on that row or column. +`Row` retrieves the indices of all the columns in a row. It also retrieves any attributes set on that row. -**Result Type:** object with attrs and bits. +**Result Type:** object with attrs and columns. -e.g. `{"attrs":{"username":"mrpi","active":true},"bits":[10, 20]}` +e.g. `{"attrs":{"username":"mrpi","active":true},"columns":[10, 20]}` **Examples:** -Query all columns with a bit set in row 1 of the frame `stargazer` (repositories that are starred by user 1): +Query all columns with a bit set in row 1 of the field `stargazer` (repositories that are starred by user 1): ```request -Bitmap(frame="stargazer", row=1) +Row(stargazer=1) ``` ```response -{"attrs":{"username":"mrpi","active":true},"bits":[10, 20]} +{"attrs":{"username":"mrpi","active":true},"columns":[10, 20]} ``` * attrs are the attributes for user 1 -* bits are the repositories which user 1 has starred. +* columns are the repositories which user 1 has starred. #### Union **Spec:** ``` -Union([BITMAP_CALL ...]) +Union([ROW_CALL ...]) ``` **Description:** -Union performs a logical OR on the results of all `BITMAP_CALL` queries passed to it. +Union performs a logical OR on the results of all `ROW_CALL` queries passed to it. **Result Type:** object with attrs and bits @@ -309,28 +285,27 @@ attrs will always be empty Query columns with a bit set in either of two rows (repositories that are starred by either of two users): ```request -Union(Bitmap(frame="stargazer", stargazer_id=1), Bitmap(frame="stargazer", stargazer_id=2)) +Union(Row(stargazer=1), Row(stargazer=2)) ``` ```response -{"attrs":{},"bits":[10, 20, 30]} +{"attrs":{},"columns":[10, 20, 30]} ``` -* bits are repositories that were starred by user 1 OR user 2 +* columns are repositories that were starred by user 1 OR user 2 #### Intersect - **Spec:** ``` -Intersect(, [BITMAP_CALL ...]) +Intersect(, [ROW_CALL ...]) ``` **Description:** -Intersect performs a logical AND on the results of all `BITMAP_CALL` queries passed to it. +Intersect performs a logical AND on the results of all `ROW_CALL` queries passed to it. -**Result Type:** object with attrs and bits +**Result Type:** object with attrs and columns attrs will always be empty @@ -339,27 +314,27 @@ attrs will always be empty Query columns with a bit set in both of two rows (repositories that are starred by both of two users): ```request -Intersect(Bitmap(frame="stargazer", row=1), Bitmap(frame="stargazer", row=2)) +Intersect(Row(stargazer=1), Row(stargazer=2)) ``` ```response -{"attrs":{},"bits":[10]} +{"attrs":{},"columns":[10]} ``` -* bits are repositories that were starred by user 1 AND user 2 +* columns are repositories that were starred by user 1 AND user 2 #### Difference **Spec:** ``` -Difference(, [BITMAP_CALL ...]) +Difference(, [ROW_CALL ...]) ``` **Description:** -Difference returns all of the bits from the first `BITMAP_CALL` argument passed to it, without the bits from each subsequent `BITMAP_CALL`. +Difference returns all of the bits from the first `ROW_CALL` argument passed to it, without the bits from each subsequent `ROW_CALL`. -**Result Type:** object with attrs and bits +**Result Type:** object with attrs and columns attrs will always be empty @@ -367,37 +342,37 @@ attrs will always be empty Query columns with a bit set in one row and not another (repositories that are starred by one user and not another): ```request -Difference(Bitmap(frame="stargazer", row=1), Bitmap( frame="stargazer", row=2)) +Difference(Row(stargazer=1), Row(stargazer=2)) ``` ```response -{"results":[{"attrs":{},"bits":[20]}]} +{"results":[{"attrs":{},"columns":[20]}]} ``` -* bits are repositories that were starred by user 1 BUT NOT user 2 +* columns are repositories that were starred by user 1 BUT NOT user 2 Query for the opposite difference: ```request -Difference(Bitmap(frame="stargazer", row=2), Bitmap( frame="stargazer", row=1)) +Difference(Row(stargazer=2), Row(stargazer=1)) ``` ```response -{"attrs":{},"bits":[30]} +{"attrs":{},"columns":[30]} ``` -* Bits are repositories that were starred by user 2 BUT NOT user 1 +* columnss are repositories that were starred by user 2 BUT NOT user 1 #### Xor **Spec:** ``` -Xor(, [BITMAP_CALL ...]) +Xor(, [ROW_CALL ...]) ``` **Description:** -Xor performs a logical XOR on the results of each `BITMAP_CALL` query passed to it. +Xor performs a logical XOR on the results of each `ROW_CALL` query passed to it. -**Result Type:** object with attrs and bits +**Result Type:** object with attrs and columns attrs will always be empty @@ -406,24 +381,24 @@ attrs will always be empty Query columns with a bit set in exactly one of two rows (repositories that are starred by only one of two users): ```request -Xor(Bitmap(frame="stargazer", row=1), Bitmap(frame="stargazer", row=2)) +Xor(Row(stargazer=2), Row(stargazer=1)) ``` ```response -{"results":[{"attrs":{},"bits":[10,20,30]}]} +{"results":[{"attrs":{},"columns":[10,20,30]}]} ``` -* bits are repositories that were starred by user 1 XOR user 2 (user 1 or user 2, but not both) +* columns are repositories that were starred by user 1 XOR user 2 (user 1 or user 2, but not both) #### Count **Spec:** ``` -Count() +Count() ``` **Description:** -Returns the number of set bits in the `BITMAP_CALL` passed in. +Returns the number of set bits in the `ROW_CALL` passed in. **Result Type:** int @@ -431,7 +406,7 @@ Returns the number of set bits in the `BITMAP_CALL` passed in. Query the number of bits set in a row (the number of repositories a user has starred): ```request -Count(Bitmap(frame="stargazer", row=1)) +Count(Row(stargazer=1)) ``` ```response {"results":[1]} @@ -444,34 +419,34 @@ Count(Bitmap(frame="stargazer", row=1)) **Spec:** ``` -TopN([BITMAP_CALL], , [n=UINT], - [, ]) +TopN([ROW_CALL], , [n=UINT], + [attrName=, attrValues=<[]ATTR_VALUE>]) ``` **Description:** -Return the id and count of the top `n` bitmaps (by count of bits) in the frame. -The `field` and `filters` arguments work together to only return Bitmaps which -have the attribute specified by `field` with one of the values specified in -`filters`. +Return the id and count of the top `n` bitmaps (by count of bits) in the field. +The `attrName` and `attrValues` arguments work together to only return rows which +have the attribute specified by `attrName` with one of the values specified in +`attrValues`. **Result Type:** array of key/count objects **Caveats:** -* Performing a TopN() query on a frame with cache type ranked will return the top bitmaps sorted by count in descending order. -* Frames with cache type lru will maintain an LRU (Least Recently Used replacement policy) cache, thus a TopN query on this type of frame will return bitmaps sorted in order of most recently set bit. -* The frame's cache size determines the number of sorted bitmaps to maintain in the cache for purposes of TopN queries. There is a tradeoff between performance and accuracy; increasing the cache size will improve accuracy of results at the cost of performance. -* Once full, the cache will truncate the set of bitmaps according to the frame option CacheSize. Bitmaps that straddle the limit and have the same count will be truncated in no particular order. +* Performing a TopN() query on a field with cache type ranked will return the top bitmaps sorted by count in descending order. +* Fields with cache type lru will maintain an LRU (Least Recently Used replacement policy) cache, thus a TopN query on this type of field will return bitmaps sorted in order of most recently set bit. +* The field's cache size determines the number of sorted bitmaps to maintain in the cache for purposes of TopN queries. There is a tradeoff between performance and accuracy; increasing the cache size will improve accuracy of results at the cost of performance. +* Once full, the cache will truncate the set of bitmaps according to the field option CacheSize. Bitmaps that straddle the limit and have the same count will be truncated in no particular order. * The TopN query's attribute filter is applied to the existing sorted cache of bitmaps. Bitmaps that fall outside of the sorted cache range, even if they would normally pass the filter, are ignored. -See [frame creation](../api-reference/#create-frame) for more information about the cache. +See [field creation](../api-reference/#create-field) for more information about the cache. **Examples:** Basic TopN query: ```request -TopN(frame="stargazer") +TopN(stargazer) ``` ```response {"results":[[{"id":1240,"count":102},{"id":4734,"count":100},{"id":12709,"count":93},...]]} @@ -479,11 +454,11 @@ TopN(frame="stargazer") * `id` is a row ID (user ID) * `count` is a count of columns (repositories) -* Results are the number of bits set in the corresponding row (repositories that each user starred) in descending order for all rows (users) in the stargazer frame. For example user 1240 starred 102 repositories, user 4734 starred 100 repositories, user 12709 starred 93 repository. +* Results are the number of bits set in the corresponding row (repositories that each user starred) in descending order for all rows (users) in the stargazer field. For example user 1240 starred 102 repositories, user 4734 starred 100 repositories, user 12709 starred 93 repository. Limit the number of results: ```request -TopN(frame="stargazer", n=2) +TopN(stargazer, n=2) ``` ```response {"results":[[{"id":1240,"count":102},{"id":4734,"count":100}]]} @@ -493,17 +468,17 @@ TopN(frame="stargazer", n=2) Filter based on an existing Bitmap: ```request -TopN(Bitmap(frame="language", row=1), frame="stargazer", n=2) +TopN(Row(language=1), stargazer, n=2) ``` ```response {"results":[[{"id":1240,"count":35},{"id":7508,"count":32}]]} ``` -* Results are the top two users (rows) sorted by the number of bits set in the intersection with row 1 of the language frame (repositories that they've starred which are written in language 1). +* Results are the top two users (rows) sorted by the number of bits set in the intersection with row 1 of the language field (repositories that they've starred which are written in language 1). Filter based on attributes: ```request -TopN(frame="stargazer", n=2, field=active, filters=[true]) +TopN(stargazer, n=2, attrName=active, attrValues=[true]) ``` ```response {"results":[[{"id":10,"count":1},{"id":13,"count":1}]]} @@ -516,31 +491,30 @@ TopN(frame="stargazer", n=2, field=active, filters=[true]) **Spec:** ``` -Range(, , - , ) +Range(field=, , ) ``` **Description:** -Similar to `Bitmap`, but only returns bits which were set with timestamps -between the given `start` and `end` timestamps. +Similar to `Row`, but only returns bits which were set with timestamps +between the given `start` (first) and `end` (second) timestamps. **Result Type:** object with attrs and bits **Examples:** -Query all columns with a bit set in row 1 of a frame (repositories that a user has starred), within a date range: +Query all columns with a bit set in row 1 of a field (repositories that a user has starred), within a date range: ```request -Range(frame="stargazer", row=1, start="2010-01-01T00:00", end="2017-03-02T03:00") +Range(stargazer=1, 2010-01-01T00:00, 2017-03-02T03:00) ``` ```response -{{"attrs":{},"bits":[10]} +{{"attrs":{},"columns":[10]} ``` This example assumes timestamps have been set on some bits. -* bits are repositories which were starred by user 1 in the time range 2010-01-01 to 2017-03-02. +* columns are repositories which were starred by user 1 in the time range 2010-01-01 to 2017-03-02. #### Range (BSI) @@ -548,16 +522,15 @@ This example assumes timestamps have been set on some bits. **Spec:** ``` -Range(, ) +Range([ ] ) ``` **Description:** -The `Range` query is overloaded to work on `field` values as well as `timestamp` values. +The `Range` query is overloaded to work on `integer` values as well as `timestamp` values. Returns bits that are true for the comparison operator. -**Result Type:** object with attrs and bits - +**Result Type:** object with attrs and columns **Examples:** @@ -565,13 +538,13 @@ In our source data, commitactivity was counted over the last year. The following greater-than `Range` query returns all columns with a field value greater than 100 (repositories having more than 100 commits): ```request -Range(frame="stats", commitactivity > 100) +Range(commitactivity > 100) ``` ```response -{{"attrs":{},"bits":[10]} +{{"attrs":{},"columns":[10]} ``` -* bits are repositories which had at least 100 commits in the last year. +* columns are repositories which had at least 100 commits in the last year. BSI range queries support the following operators: @@ -583,35 +556,37 @@ BSI range queries support the following operators: `>=` | greater-than-or-equal-to, GTE | integer `==` | equal-to, EQ | integer `!=` | not-equal-to, NEQ | integer or `null` - `><` | between, BETWEEN | [integer, integer] -The `BETWEEN` form specifies an interval with both bounds, using the `><` operator, and a two-element list containing the lower and upper bounds of the interval: +`<`, and `<=` can be chained together to represent a bounded interval. For example: -```pql -Range(frame="stats", commitactivity >< [100, 200]) +```request +Range(50 < commitactivity < 150) +``` +```response +{{"attrs":{},"columns":[10]} ``` -This is conceptually equivalent to the interval 100 <= commitactivity <= 200, but this chained comparison syntax is not currently supported. `BETWEEN` query syntax is restricted to greater-than-or-equal-to and less-than-or-equal-to, but any valid interval on the integers can be represented this way. +As of Pilosa 1.0, the "between" syntax `Range(frame=stats, commitactivity >< [50, 150])` is no longer supported. #### Min **Spec:** ``` -Min([BITMAP_CALL], , ) +Min([ROW_CALL], field=) ``` **Description:** -Returns the minimum value of all BSI integer values in the `field` in this `frame`. If the optional `Bitmap` call is supplied, only columns with set bits are considered, otherwise all columns are considered. +Returns the minimum value of all BSI integer values in this `field`. If the optional `Row` call is supplied, only columns with set bits are considered, otherwise all columns are considered. **Result Type:** object with the min and count of columns containing the min value. **Examples:** -Query the minimum value of all fields in a frame (minimum size of all repositories): +Query the minimum value of a field (minimum size of all repositories): ```request -Min(frame="stats", field="diskusage") +Min(field="diskusage") ``` ```response {"value":4,"count":2} @@ -624,20 +599,20 @@ Min(frame="stats", field="diskusage") **Spec:** ``` -Max([BITMAP_CALL], , ) +Max([ROW_CALL], field=) ``` **Description:** -Returns the maximum value of all BSI integer values in the `field` in this `frame`. If the optional `Bitmap` call is supplied, only columns with set bits are considered, otherwise all columns are considered. +Returns the maximum value of all BSI integer values in this `field`. If the optional `Row` call is supplied, only columns with set bits are considered, otherwise all columns are considered. **Result Type:** object with the max and count of columns containing the max value. **Examples:** -Query the maximum value of all fields in a frame (maximum size of all repositories): +Query the maximum value of a field (maximum size of all repositories): ```request -Max(frame="stats", field="diskusage") +Max(field="diskusage") ``` ```response {"value":88,"count":13} @@ -650,12 +625,12 @@ Max(frame="stats", field="diskusage") **Spec:** ``` -Sum([BITMAP_CALL], , ) +Sum([ROW_CALL], field=) ``` **Description:** -Returns the count and computed sum of all BSI integer values in the `field` and `frame`. If the optional `Bitmap` call is supplied, columns with set bits are summed, otherwise the sum is across all columns. +Returns the count and computed sum of all BSI integer values in the `field`. If the optional `Row` call is supplied, columns with set bits are summed, otherwise the sum is across all columns. **Result Type:** object with the computed sum and count of the bitmap field. @@ -663,7 +638,7 @@ Returns the count and computed sum of all BSI integer values in the `field` and Query the size of all repositories. ```request -Sum(frame="stats", field="diskusage") +Sum(field="diskusage") ``` ```response {"value":10,"count":3} diff --git a/docs/webui.md b/docs/webui.md index 94377fbe8..a07132852 100644 --- a/docs/webui.md +++ b/docs/webui.md @@ -43,14 +43,14 @@ In addition to standard PQL, the console supports a few special commands, prefix - `:create index ` - `:delete index ` - `:use ` -- `:create frame ` -- `:delete frame ` +- `:create field ` +- `:delete field ` -Frame creation also supports options like `timeQuantum`. When creating a new frame, add options by using the keys documented in [API reference](../api-reference/#create-frame). +Field creation also supports options like `timeQuantum`. When creating a new field, add options by using the keys documented in [API reference](../api-reference/#create-field). -- `:create frame cacheSize=10000` +- `:create field cacheSize=10000` ### Cluster Admin -Use the Cluster Admin tab to view the current status of your cluster. This contains information on each node in the cluster, plus the list of Indexes and Frames. +Use the Cluster Admin tab to view the current status of your cluster. This contains information on each node in the cluster, plus the list of Indexes and Fields.