scriptoriumadmin/featurebase
1
0
Fork 0
mirror of https://github.com/featurebasedb/featurebase.git synced 2026-09-07 17:15:56 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

remove docs directory

Browse source
This commit is contained in:
Travis 2021-02-23 10:13:16 -06:00
parent 5794a4af69
commit 295101ecbd
No known key found for this signature in database
GPG key ID: 37080CC2042BA34E
17 changed files with 0 additions and 5338 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
docs/README.md
View file

@ -1,5 +0,0 @@
Pilosa docs are maintained here, to stay in sync with the codebase. The format is [Blackfriday](https://github.com/russross/blackfriday) markdown, with some Hugo [front matter](https://gohugo.io/content-management/front-matter/).
Please visit [our website](https://www.pilosa.com/docs/) to view the docs complete with styles, diagrams, and comprehensive search. Internal links will only work on the website.
Have you found a discrepancy, typo, or other problem? Please submit an [issue](https://github.com/pilosa/pilosa/issues/new) or a pull request!

327
docs/administration.md
View file

@ -1,327 +0,0 @@
+++
title = "Administration"
weight = 13
nav = [
"Installing in production",
"Importing and Exporting Data",
"Versioning",
"Resizing the Cluster",
"Backup/restore",
]
+++
## Administration Guide
### Installing in production
#### Hardware
Pilosa is a standalone, compiled Go application, so there is no need to worry about running and configuring a Java VM. Pilosa can run on very small machines and works well with even a medium sized dataset on a personal laptop. If you are reading this section, you are likely ready to deploy a cluster of Pilosa servers handling very large datasets or high velocity data. These are guidelines for running a cluster; specific needs may differ.
#### Memory
Pilosa holds all row/column bitmap data in main memory. While this data is compressed more than a typical database, available memory is a primary concern. In a production environment, we recommend choosing hardware with a large amount of memory >= 64GB. Prefer a small number of hosts with lots of memory per host over a larger number with less memory each. Larger clusters tend to be less efficient overall due to increased inter-node communication.
#### CPUs
Pilosa is a concurrent application written in Go and can take full advantage of multicore machines. The main unit of parallelism is the [shard](../data-model/#shard), so a single query will only use a number of cores up to the number of shards stored on that host. Multiple queries can still take advantage of multiple cores as well, so tuning in this area is dependent upon the expected workload.
#### Disk
Even though the main dataset is in memory Pilosa backs up to disk frequently. We recommend SSDs—especially if you have a write-heavy application.
#### Network
Pilosa is designed to be a distributed application, with data replication replicated across the cluster. As such, every write and read needs to communicate with several nodes. Therefore fast internode communication is essential. If using a service like AWS we recommend that all nodes exist in the same region and availability zone. The inherent latency of spreading a Pilosa cluster across physical regions is not usually worth the redundancy protection. Since Pilosa is designed to be an indexing service there should already be a system of record, or ability to rebuild a cluster quickly from backups.
#### Overview
While Pilosa does have some high system requirements it is not a best practice to set up a cluster with the fewest, largest machines available. You want an evenly distributed load across several nodes in a cluster to easily recover from a single node failure, and have the resource capacity to handle a missing node until it's repaired or replaced. Nor is it advisable to have many small machines, as the internode network traffic will become a bottleneck. You can always add nodes later, but that does require some down time.
### Open File Limits
Pilosa requires a large number of open files to support its memory-mapped file storage system. Most operating systems put limits on the maximum number of files that may be opened concurrently by a process. On Linux systems, this limit is controlled by a utility called [ulimit](https://ss64.com/bash/ulimit.html). Pilosa will automatically attempt to raise the limit to `262144` during startup, but it may fail due to access limitations. If you see errors related to open file limits when starting Pilosa, it is recommended that you run `sudo ulimit -n 262144` before starting Pilosa.
On Mac OS X, `ulimit` does not behave predictably. The Mac OS X system has a utility called csrutil that prevents you from changing the open file limit easily. One workaround that may work for you involves disabling the csrutil program. To disable the csrutil program, restart your laptop and when the start up screen pops up, hold down command + R to enter Recovery Mode. Open a terminal and enter `csrutil disable`, then restart your computer as you normally would. Now that the csrutil is disabled, you can change the open file limit. The open file limit can be changed by creating the following files and changing their ownership:
Copy the contents of [this](https://github.com/wilsonmar/mac-setup/blob/master/configs/limit.maxfiles.plist) file into a new file on your system located at /Library/LaunchDaemons/limit.maxfiles.plist, then run:
```
sudo chown root:wheel /Library/LaunchDaemons/limit.maxfiles.plist
```
Copy the contents of [this](https://github.com/wilsonmar/mac-setup/blob/master/configs/limit.maxproc.plist) file into a new file on your system located at /Library/LaunchDaemons/limit.maxproc.plist, then run:
```
sudo chown root:wheel /Library/LaunchDaemons/limit.maxproc.plist
```
To ensure the open file limit has successfully changed, run `ulimit -a`. Your open files should be set to a number greater than 256 (in the range of 524288) and your max users processes should be greater than 709 (in the range of 2048).
### Importing and Exporting Data
#### Importing
The import API expects a csv of the format `Row,Column`.
When importing large datasets remember it is much faster to pre sort the data by row ID and then by column ID in ascending order. You can use the `--sort` flag to do that. Also, avoid querying Pilosa until the import is complete, otherwise you will experience inconsistent results.
```
pilosa import --sort -i project -f stargazer project-stargazer.csv
```
We recommend importing data using official Pilosa client libraries. You can find the corresponding documentation at:
* [Go client imports documentation](https://github.com/pilosa/go-pilosa/blob/master/docs/imports-exports.md)
* [Java client imports documentation](https://github.com/pilosa/java-pilosa/blob/master/docs/imports.md)
* [Python client imports documentation](https://github.com/pilosa/python-pilosa/blob/master/docs/imports.md)
##### Importing Integer Values
If you are using [integer](../data-model/#bsi-range-encoding) field values, the CSV file should be in the format `Column,Value`.
```
pilosa import -i project -f stargazer-counts project-stargazer-counts.csv
```
##### Importing Boolean Values
If you are using a [boolean](../data-model/#boolean) field, the CSV file should be in the format `Boolean,Value`, where `Boolean` is either `0` (false) or `1` (true).
For example, importing a file with the following contents will result in columns 3 and 9 being set in the `false` row, and columns 1, 2, 4, and 8 being set in the `true` row.
```
0,3
0,9
1,1
1,2
1,4
1,8
```
<div class="note">
<p>Note that you must first create a field. View <a href="../api-reference/#create-field">Create Field</a> for more details. The `-e` flag can create the necessary schema when using a field of type "set".</p>
</div>
#### Clearing Data via Import
By using the `--clear` flag with the import command, Pilosa will clear the values provided in the import payload.
For example, importing a file with the following contents along with the `--clear` flag will result in data being cleared from row 0, column 9; row 1, columns 2 and 8; and row 3, column 12. Clearing a value that doesn't exists is allowed.
```
0,9
1,2
1,8
3,12
```
#### Exporting
Exporting data to csv can be performed on a live instance of Pilosa. You need to specify the index and the field. The API also expects the shard number, but the `pilosa export` sub command will export all shards within a field. The data will be in csv format `Row,Column` and sorted by column.
```request
curl "http://localhost:10101/export?index=repository&field=stargazer&shard=0" \
--header "Accept: text/csv"
```
```response
2,10
2,30
3,426
4,2
...
```
### Versioning
Pilosa follows [Semantic Versioning](http://semver.org/).
MAJOR.MINOR.PATCH:
* MAJOR version when you make incompatible API changes,
* MINOR version when you add functionality in a backwards-compatible manner, and
* PATCH version when you make backwards-compatible bug fixes.
#### PQL versioning
The Pilosa server should support PQL versioning using HTTP headers. On each request, the client should send a Content-Type header and an Accept header. The server should respond with a Content-Type header that matches the client Accept header. The server should also optionally respond with a Warning header if a PQL version is in a deprecation period, or an HTTP 400 error if a PQL version is no longer supported.
#### Upgrading
To upgrade Pilosa:
1. First, upgrade the [client libraries](../client-libraries/) you are using in your application. Generally, a client version `X` will be compatible with the Pilosa server version `X` and earlier. For example, `python-pilosa 0.9.0` is compatible with both `pilosa 0.8.0` and `pilosa 0.9.0`.
2. Next, download the latest release from our [installation page](/docs/latest/installation/) or from the [release page on Github](https://github.com/pilosa/pilosa/releases).
3. Shut down the Pilosa cluster.
4. Make a backup of the [data directory](../configuration/#data-dir) on each cluster node.
5. Upgrade the Pilosa server binaries and any configuration changes. See the following sections on any version-specific changes you must make.
6. Start Pilosa. It is recommended to start the cluster coordinator node first, followed by any other nodes.
##### Version 1.4
Pilosa 1.4.0 changes the way that integer fields are stored. The upgrade from old format to new is handled automatically, however you will not be able to downgrade to 1.3 should you wish to do so. We *always* recommend taking a backup of your Pilosa data directory before upgrading Pilosa, but doubly so with this release.
### Resizing the Cluster
If you need to increase (or decrease) the capacity of a Pilosa server, you can add or remove nodes to a running cluster at any time. Note that you can only add or remove one node at a time; if you attempt to add multiple nodes at once, those requests will be enqueued and processed serially. Also note that during any resize process, the cluster goes into state `RESIZING` during which all read/write requests are denied. When the cluster returns to state `NORMAL` then read/write operations can resume. The amount of time that the cluster stays in state `RESIZING` depends on the amount of data that needs to be moved during the resize process.
#### Adding a Node
You can add a new, empty node to an existing cluster by starting `pilosa server` on the new node with the correct configuration options. Specifically, you must specify the [cluster coordinator](../configuration/#cluster-coordinator) to be the same as the coordinator on the existing nodes. You must also specify at least one valid [gossip seed](../configuration/#gossip-seeds) (preferably multiple for redundancy). When the new node starts, the coordinator node will receive a `nodeJoin` event indicating that a new node is joining the cluster. At this point, the coordinator will put the cluster into state `RESIZING` and kick off a resize job that instructs all of the nodes in the cluster how to rebalance data to accomodate the additional capacity of the new node. Once the resize job is complete, the coordinator will put the cluster back to state `NORMAL` and ensure that the new node is included in future queries.
If the node is being added to a cluster which contains no data (for example, during startup of a new cluster), the coordinator will bypass the `RESIZING` state and allow the node to join the cluster immediately.
#### Removing a Node
In order to remove a node from a cluster, your cluster must be configured to have a [cluster replicas](../configuration/#cluster-replicas) value of at least 2; if you're removing a node that no longer exists (for example a node that has died), there must be at least one additional replica of the data owned by the dead node in order for the cluster to correctly rebalance itself.
To remove node `localhost:10102` from a cluster having coordinator `localhost:10101`, first determine the ID of the node to be removed. If the node to be removed is still available, you can find the ID by issuing a `/status` request to the node. The node's ID is in the `localID` field:
``` request
curl localhost:10101/status
```
``` response
{
"state":"NORMAL",
"nodes":[
{"id":"24824777-62ec-4151-9fbd-67e4676e317d","uri":{"scheme":"http","host":"localhost","port":10101}}
{"id":"40a891fa-243b-4d71-ae24-4f5c78a0f4b1","uri":{"scheme":"http","host":"localhost","port":10102}}
{"id":"9fab09cc-3c26-4202-9622-d167c84684d9","uri":{"scheme":"http","host":"localhost","port":10103}}
],
"localID": "40a891fa-243b-4d71-ae24-4f5c78a0f4b1"
}
```
If the node to be removed is no longer available, you can get the IDs of the nodes in the cluster by issuing a `/status` request to any available node:
``` request
curl localhost:10101/status
```
``` response
{
"state":"NORMAL",
"nodes":[
{"id":"24824777-62ec-4151-9fbd-67e4676e317d","uri":{"scheme":"http","host":"localhost","port":10101}}
{"id":"40a891fa-243b-4d71-ae24-4f5c78a0f4b1","uri":{"scheme":"http","host":"localhost","port":10102}}
{"id":"9fab09cc-3c26-4202-9622-d167c84684d9","uri":{"scheme":"http","host":"localhost","port":10103}}
],
"localID": "40a891fa-243b-4d71-ae24-4f5c78a0f4b1"
}
```
Once you have the ID of the node that you want to remove from the cluster, issue the following request:
```
curl localhost:10101/cluster/resize/remove-node \
-X POST \
-d '{"id": "40a891fa-243b-4d71-ae24-4f5c78a0f4b1"}'
```
At this point, the coordinator will put the cluster into state `RESIZING` and kick off a resize job that instructs all of the nodes in the cluster how to rebalance data to accomodate the reduced capacity of the cluster. Once the resize job is complete, the coordinator will put the cluster back to state `NORMAL` and ensure that the removed node is no longer included in future queries.
Note that you can't directly remove the coordinator node. If you need to remove the coordinator node from the cluster, you must first [make one of the other nodes the coordinator](#changing-the-coordinator).
#### Aborting a Resize Job
If at any point you need to abort an active resize job, you can issue a `POST` request to the `/cluster/resize/abort` endpoint on the coordinator node.
For example, if your coordinator node is `localhost:10101`, then you can run:
```
curl localhost:10101/cluster/resize/abort -X POST
```
This will immediately abort the resize job and return the cluster to state `NORMAL`. Because data is never removed from a node during a resize job (only once a resize job has successfully completed), aborting a resize job will return the cluster back to the state it was in before the resize began.
#### Changing the Coordinator
In order to assign a different node to be the coordinator, you can issue a `/cluster/resize/set-coordinator` request to any node in the cluster. The payload should indicate the ID of the node to be made coordinator.
```
curl localhost:10101/cluster/resize/set-coordinator \
-X POST \
-d '{"id": "9fab09cc-3c26-4202-9622-d167c84684d9"}'
```
### Backup/restore
Pilosa continuously writes out the in-memory bitmap data to disk. This data is organized by Index->Field->Views->Fragment->numbered shard files. These data files can be routinely backed up to restore nodes in a cluster.
Depending on the size of your data you have two options. For a small dataset you can rely on the periodic anti-entropy sync process to replicate existing data back to this node.
For larger datasets and to make this process faster you could copy the relevant data files from the other nodes to the new one before startup.
Note: This will only work when the replication factor is >= 2
#### Using Index Sync
- Shutdown the cluster.
- Modify config file to replace existing node address with new node.
- Restart all nodes in the cluster.
- Wait for auto Index sync to replicate data from existing nodes to new node.
#### Copying data files manually
- To accomplish this you will first need:
- List of all indexes on your cluster
- List of all fields in your indexes
- Max shard per index, listed in the `/internal/shards/max` endpoint
- With this information you can query the `/internal/fragment/nodes` endpoint and iterate over each shard
- Using the list of shards owned by this node you will then need to manually:
- setup a directory structure similar to the other nodes with a path for each Index/Field
- copy each owned shard for an existing node to this new node
- Modify the cluster config file to replace the previous node address with the new node address.
- Restart the cluster
- Wait for the first sync (10 minutes) to validate Index connections
### Diagnostics
Each Pilosa cluster is configured by default to share anonymous usage details with Pilosa Corp. These metrics allow us to understand how Pilosa is used by the community and improve the technology to suit your needs. Diagnostics are sent to Pilosa every hour. Each of the metrics are detailed below as well as opt-out instructions.
- **Version:** Version string of the build.
- **Host:** Host URI.
- **Cluster:** List of nodes in the cluster.
- **NumNodes:** Number of nodes in the cluster.
- **NumCPU:** Number of cores per node
- **BSIEnabled:** Bit Sliced Index Fields in use.
- **TimeQuantumEnabled:** Time Quantum Fields in use.
- **NumIndexes:** Number of indexes in the Cluster.
- **NumFields:** Number of fields in the Cluster.
- **NumShards:** Number of shards in the Cluster.
- **NumViews:** Number of views in the Cluster.
- **OpenFiles:** Open file handle count.
- **GoRoutines:** Go routine count.
You can opt-out of the Pilosa diagnostics reporting by setting the command line configuration option `--metric.diagnostics=false`, the `PILOSA_METRIC_DIAGNOSTICS` environment variable, or the TOML configuration file `[metric]` `diagnostics` option.
### Metrics
Pilosa can be configured to emit metrics pertaining to its internal processes in one of three formats: Expvar, StatsD, or Prometheus. Metric recording is disabled by default.
The metrics configuration options are:
- [Host](../configuration/#metric-host): specify host that receives metric events
- [Poll Interval](../configuration/#metric-poll-interval): specify polling interval for runtime metrics
- [Service](../configuration/#metric-service): declare type StatsD or Expvar
#### Tags
StatsD Tags adhere to the DataDog format (key:value), and we tag the following:
- NodeID
- Index
- Field
- View
- Shard
#### Events
We currently track the following events
- **Index:** The creation of a new index.
- **Field:** The creation of a new field.
- **MaxShard:** The creation of a new Shard.
- **SetBit:** Count of set bits.
- **ClearBit:** Count of cleared bits.
- **ImportBit:** During a bulk data import this represents the count of bits created.
- **SetRowAttrs:** Count of attributes set per row.
- **SetColumnAttrs:** Count of attributes set per column.
- **Bitmap:** Count of Bitmap queries.
- **TopN:** Count of TopN queries.
- **Union:** Count of Union queries.
- **Intersection:** Count of Intersection queries.
- **Difference:** Count of Difference queries.
- **Xor:** Count of Xor queries.
- **Not:** Count of Not queries.
- **Count:** Count of Count queries.
- **Range:** Count of ranged Row queries.
- **Snapshot:** Event count when the snapshot process is triggered.
- **BlockRepair:** Count of data blocks that were out of sync and repaired.
- **GarbageCollection:** Event count when garbage collection occurs.
- **Goroutines:** Number of running goroutines.
- **OpenFiles:** Number of open file handles associated with running Pilosa process ID.

415
docs/api-reference.md
View file

@ -1,415 +0,0 @@
+++
title = "API Reference"
weight = 10
nav = []
+++
## API Reference
### List all index schemas
`GET /index`
Is equivalent to `GET /schema` and returns the same response.
### List index schema
`GET /index/{index-name}`
Returns the schema of the specified index in JSON.
``` request
curl -XGET localhost:10101/index/user
```
``` response
{
"name": "user",
"createdAt": 1591178953061239000,
"options": {
"keys": false,
"trackExistence": true
},
"fields": [
{
"name": "event",
"createdAt": 1591178962332452000,
"options": {
"type": "set",
"cacheType": "ranked",
"cacheSize": 50000,
"keys": false
}
}
],
"shardWidth": 1048576
}
```
### Create index
`POST /index/{index-name}`
Creates an index with the given name.
The request payload is in JSON, and may contain the `options` field. The `options` field is a JSON object with the following options:
* `keys` (bool): Enables using column keys instead of column IDs.
* `trackExistence` (bool): Enables or disables existence tracking on the index. Required for [Not](../query-language/#not) queries. It is `true` by default.
``` request
curl -XPOST localhost:10101/index/user -d '{"options":{"keys":true}}'
```
``` response
{"success":true,"name":"user","createdAt":1591179042178854000}
```
### Remove index
`DELETE /index/index-name`
Removes the given index.
``` request
curl -XDELETE localhost:10101/index/user
```
``` response
{"success":true}
```
### Query index
`POST /index/{index-name}/query`
Sends a [query](../query-language/) to the Pilosa server with the given index. The request body is UTF-8 encoded text and response body is in JSON by default.
``` request
curl localhost:10101/index/user/query \
-X POST \
-d 'Row(language=5)'
```
``` response
{
"results": [
{
"attrs": {},
"columns": [
100
]
}
]
}
```
In order to send protobuf binaries in the request and response, set `Content-Type` and `Accept` headers to: `application/x-protobuf`.
The response doesn't include column attributes by default. To return them, set the `columnAttrs` query argument to `true`.
The query is executed for all [shards](../data-model/#shard) by default. To use specified shards only, set the `shards` query argument to a comma-separated list of slice indices.
``` request
curl "localhost:10101/index/user/query?columnAttrs=true&shards=0,1" \
-X POST \
-d 'Row(language=5)'
```
``` response
{
"columnAttrs": [
{
"attrs": {
"name": "Klingon"
},
"id": 100
}
],
"results": [
{
"attrs": {},
"columns": [
100
]
}
]
}
```
By default, all bits and attributes (*for `Row` queries only*) are returned. In order to suppress returning bits, set `excludeBits` query argument to `true`; to suppress returning attributes, set `excludeAttrs` query argument to `true`.
### Import Data
`POST /index/{index-name}/field/{field-name}/import`
Supports high-rate data ingest to a particular shard of a particular field. The
official client libraries use this endpoint for their import functionality - it
is not usually necessary to use this endpoint directly. See the documentation for
imports for
<a href="https://github.com/pilosa/go-pilosa/blob/master/docs/imports-exports.md">Go</a>,
<a href="https://github.com/pilosa/java-pilosa/blob/master/docs/imports.md">Java</a>,
and <a href="https://github.com/pilosa/python-pilosa/tree/master/docs/imports.md">Python</a>.
The request payload is protobuf encoded with the following schema. The RowKeys
and/or ColumnKeys fields are used if the pilosa field or index are configured
for keys respectively. Otherwise, the RowIDs and ColumnIDs fields are used. They
must have the same number of items, and each index into those two lists
represents a particular bit to be set. Timestamps are optional, but if they
exist must also contain the same number of items as rows and columns. The
column IDs must all be in the shard specified in the request.
Some endpoints and data structures include a `CreatedAt` fields.
This is typically stored as a timestamp, but it's purpose is not to inform of the creation date of a particular index or field,
but to serve as a unique identifier for use in cache invalidation.
The problem is that users of Pilosa (such as ingesters e.g. the [IDK](https://github.com/molecula/idk))
can usually assume that translation keys for records and field values never change - they are only appended to, and can therefore be trivially cached.
This is true except in cases where an index or field gets deleted and then recreated,
or if Pilosa is restored from a backup.
So the ingesters must send their current `CreatedAt` value which will have changed if either of those two conditions has occured (or if Pilosa was just restarted),
and the ingester will know that it needs to drop its cache.
```
message ImportRequest {
string Index = 1;
string Field = 2;
uint64 Shard = 3;
repeated uint64 RowIDs = 4;
repeated uint64 ColumnIDs = 5;
repeated int64 Timestamps = 6;
repeated string RowKeys = 7;
repeated string ColumnKeys = 8;
int64 IndexCreatedAt = 9;
int64 FieldCreatedAt = 10;
}
```
### Create field
`POST /index/{index-name}/field/{field-name}`
Creates a field in the given index with the given name.
The request payload is in JSON, and may contain the `options` field. The `options` field is a JSON object which must contain a `type`:
* `type` (string): Sets the field type and type options.
* `keys` (bool): Enables using column keys instead of column IDs (optional).
Valid `type`s and correspondonding options are listed below:
* `set`
* `cacheType` (string): [ranked](../data-model/#ranked) or [LRU](../data-model/#lru) caching on this field. Default is `ranked`.
* `cacheSize` (int): Number of rows to keep in the cache. Default is 50,000.
* `int`
* `min` (int): Minimum integer value allowed for the field.
* `max` (int): Maximum integer value allowed for the field.
* `bool`
* (boolean fields take no arguments)
* `time`
* `timeQuantum` (string): [Time Quantum](../data-model/#time-quantum) for this field.
* `mutex`
* `cacheType` (string): [ranked](../data-model/#ranked) or [LRU](../data-model/#lru) caching on this field. Default is `ranked`.
* `cacheSize` (int): Number of rows to keep in the cache. Default is 50,000.
The following example creates an `int` field called "quantity" capable of storing values from -1000 to 2000:
``` request
curl localhost:10101/index/user/field/quantity \
-X POST \
-d '{"options": {"type": "int", "min": -1000, "max":2000}}'
```
``` response
{"success":true,"name":"quantity","createdAt":1591180110914425000}
```
Integer fields are stored as n-bit range-encoded values. Pilosa supports 63-bit, signed integers with values between `min` and `max`.
``` request
curl localhost:10101/index/user/field/language -X POST
```
``` response
{"success":true,"name":"language","createdAt":1591180128294321000}
```
``` request
curl localhost:10101/index/repository/field/stats \
-X POST \
-d '{"options":{"type": "int", "min": 0, "max": 1000000}}'
```
``` response
{"success":true,"name":"stats","createdAt":1591180737881627000}
```
### Remove field
`DELETE /index/{index-name}/field/{field-name}`
Removes the given field.
``` request
curl -XDELETE localhost:10101/index/user/field/language
```
``` response
{"success":true}
```
### List all index schemas
`GET /schema`
Returns the schema of all indexes in JSON.
``` request
curl -XGET localhost:10101/schema
```
``` response
{
"indexes": [
{
"name": "user",
"createdAt": 1591178953061239000,
"options": {
"keys": false,
"trackExistence": true
},
"fields": [
{
"name": "event",
"createdAt": 1591178962332452000,
"options": {
"type": "set",
"cacheType": "ranked",
"cacheSize": 50000,
"keys": false
}
},
{
"name": "language",
"createdAt": 1591180128294321000,
"options": {
"type": "set",
"cacheType": "ranked",
"cacheSize": 50000,
"keys": false
}
},
{
"name": "quantity",
"createdAt": 1591180110914425000,
"options": {
"type": "int",
"base": 0,
"bitDepth": 0,
"min": -1000,
"max": 2000,
"keys": false,
"foreignIndex": ""
}
}
],
"shardWidth": 1048576
}
]
}
```
### Duplicate schema into empty Pilosa cluster
`POST /schema`
To duplicate one Pilosa cluster's schema to another, it's possible to
pass the output of `GET /schema` as the request body of `POST /schema`
and all the indexes and fields in the schema will be created in
Pilosa. As of this writing, the behavior of POSTing a schema to a
non-empty Pilosa cluster is undefined. These semantics will likely be
ironed out in a future version.
``` request
# after (e.g.) curl -XGET localhost:10101/schema > schema.json
curl -XPOST localhost:10101/schema --data-binary @schema.json
```
Response: `204 No Content`
### Get version
`GET /version`
Returns the version of the Pilosa server.
``` request
curl -XGET localhost:10101/version
```
``` response
{"version":"2.0.0-alpha.20-6-gb9d8d6b4"}
```
### Get status
`GET /status`
Returns the status of the cluster.
```request
curl -XGET localhost:10101/status
```
```response
{
"state": "NORMAL",
"nodes": [
{
"id": "1b018ce0-5de5-4da9-9285-6c4c0d8106f9",
"uri": {
"scheme": "http",
"host": "localhost",
"port": 10101
},
"grpc-uri": {
"scheme": "http",
"host": "localhost",
"port": 20101
},
"isCoordinator": true,
"state": "READY"
}
],
"localID": "1b018ce0-5de5-4da9-9285-6c4c0d8106f9"
}
```
### Get active queries
`GET /queries`
Returns the set of active queries. Supports pretty printing in `text/plain` format or JSON output in `application/json` format.
Also includes the amount of time that the query has been running (in nanoseconds when using JSON).
```request
curl -XGET localhost:10101/queries
```
```response
182.412µs All()
```
```request
curl -XGET -H "Accept: application/json" localhost:10101/queries
```
```response
[{"query":"All()","age":135123}]
```
### Recalculate Caches
`POST /recalculate-caches`
Recalculates the caches on demand. The cache is recalculated every 10
seconds by default. This endpoint can be used to recalculate the cache
before the 10 second interval. This should probably only be used in
integration tests and not in a typical production workflow. Note that
in a multi-node cluster, the cache is only recalculated on the node
that receives the request.
``` request
curl -XPOST localhost:10101/recalculate-caches
```
Response: `204 No Content`

25
docs/architecture.md
View file

@ -1,25 +0,0 @@
+++
title = "Architecture"
weight = 6
nav = []
+++
## Architecture
### Roaring bitmap storage format
Bitmaps are persisted to disk using a file format very similar to the [Roaring Bitmap format spec](https://github.com/RoaringBitmap/RoaringFormatSpec). Pilosa's format uses 64-bit IDs, so it is not binary-compatible with the spec. Some parts of the format are simpler, and an additional section is included. Specific differences include:
* The cookie is always bytes 0-3; the container count is always bytes 4-7, never bytes 2-3.
* The cookie includes file format version in bytes 2-3 (currently equal to zero).
* The descriptive header includes, for each container, a 64-bit key, a 16-bit cardinality, and a 16-bit container type (which only uses two bits now). This makes the runFlag bitset unnecessary. This is in contrast to the spec, which stores a 16-bit key and a 16-bit cardinality.
* The offset header section is always included.
* RLE runs are serialized as [start, last], not [start, length].
* After the container storage section is an operation log, of unspecified length.
![roaring file format diagram](/img/docs/pilosa-roaring-storage-diagram.png)
*Pilosa Roaring storage format diagram*
All values are little-endian. The first two bytes of the cookie is 12348, to reflect incompatibility with the spec, which uses 12346 or 12347. Container types are NOT inferred from their cardinality as in the spec. Instead, the container type is read directly from the descriptive header.
Check out this [blog post](/blog/adding-rle-support/) for some more details about Roaring in Pilosa.

18
docs/client-libraries.md
View file

@ -1,18 +0,0 @@
+++
title = "Client Libraries"
weight = 12
nav = [
"Go",
"Python",
"Java",
]
+++
## Client Libraries
We have the following official client libraries. You can find more information in their repositories:
* [Go client repository](https://github.com/pilosa/go-pilosa)
* [Java client repository](https://github.com/pilosa/java-pilosa)
* [Python client repository](https://github.com/pilosa/python-pilosa)
Check out our [Getting Started](https://github.com/pilosa/getting-started) repository for sample code for the official clients.

648
docs/configuration.md
View file

@ -1,648 +0,0 @@
+++
title = "Configuration"
weight = 7
nav = [
"Command line flags",
"Environment variables",
"Config file",
"All Options",
]
+++
## Configuration
Pilosa can be configured through command line flags, environment variables, and/or a configuration file; configured options take precedence in that order. So if an option is specified in a command line flag, it will take precedence over the same option specified in the environment, which will take precedence over that same option specified in the configuration file.
All options are available in all three configuration types with the exception of the `--config` option which specifies the location of the config file, and therefore will not be used if it is present in the config file.
The syntax for each option is slightly different between each of the configuration types, but follows a simple formula. See the following three sections for an explanation of each configuration type.
### Command line flags
Pilosa uses GNU/POSIX style flags. Most flags you specify as `--flagname=value` although some have a short form that is a single character and can be specified with a single dash like `-f value`. Running `pilosa server --help` will give an overview of the available flags as well as their short forms (if applicable).
### Environment variables
Every command line flag has a corresponding environment variable. The environment variable is the flag name in all caps, prefixed by `PILOSA_`, and with dots and dashes replaced by underscores. For example: `--scope.flag-name` becomes `PILOSA_SCOPE_FLAG_NAME`.
### Config file
The config file is in the [toml format](https://github.com/toml-lang/toml) and has exactly the same options available as the flags and environment variables. Any flag which contains a dot (".") denotes nesting within the config file, so the two flags `--cluster.coordinator` and `--cluster.replicas=1` look like this in the config file:
```toml
[cluster]
coordinator = true
replicas = 1
```
### All Options
#### Advertise
* Description: Address advertised by the server to other nodes in the cluster and to clients via the `/status` endpoint. Host defaults to the IP address represented by `bind` and port to 10101. If `bind` is set to `0.0.0.0` and `advertise` is not specified, then Pilosa will try to determine a reasonable, external IP address to use for `advertise`.
* Flag: `--advertise="192.168.1.100:10101"`
* Env: `PILOSA_BIND="192.168.1.100:10101"`
* Config:
```toml
advertise = 192.168.1.100:10101
```
#### Anti Entropy Interval
* Description: Interval at which the cluster will run its anti-entropy routine which ensures that all replicas of each fragment are in sync.
* Flag: `--anti-entropy.interval="10m0s"`
* Env: `PILOSA_ANTI_ENTROPY_INTERVAL="10m0s"`
* Config:
```toml
[anti-entropy]
interval = "10m0s"
```
#### Bind
* Description: host:port on which the Pilosa server will listen for requests. Host defaults to localhost and port to 10101. If `bind` is set to `0.0.0.0` then Pilosa will listen on all available interfaces.
* Flag: `--bind="localhost:10101"`
* Env: `PILOSA_BIND="localhost:10101"`
* Config:
```toml
bind = localhost:10101
```
#### CORS (Cross-Origin Resource Sharing) Allowed Origins
* Description: List of allowed origin URIs for CORS
* Flag: `--handler.allowed-origins="https://myapp.com,https://myapp.org"`
* Env: `PILOSA_HANDLER_ALLOWED_ORIGINS="https://myapp.com,https://myapp.org"`
* Config:
```toml
[handler]
allowed-origins = ["https://myapp.com", "https://myapp.org"]
```
#### Data Dir
* Description: Directory to store Pilosa data files.
* Flag: `--data-dir="~/.pilosa"`
* Env: `PILOSA_DATA_DIR="~/.pilosa"`
* Config:
```toml
data-dir = "~/.pilosa"
```
#### Log Path
* Description: Path of log file.
* Flag: `--log-path="/path/to/logfile"`
* Env: `PILOSA_LOG_PATH="/path/to/logfile"`
* Config:
```toml
log-path = "/path/to/logfile"
```
#### Verbose
* Description: Enable verbose logging.
* Flag: `--verbose`
* Env: `PILOSA_VERBOSE`
* Config:
```toml
verbose = true
```
#### Long Query Time
* Description: Duration that will trigger log and stat messages for slow queries.
* Flag: `long-query-time="1m0s"`
* Env: `PILOSA_CLUSTER_LONG_QUERY_TIME="1m0s"`
* Config:
```toml
long-query-time = "1m0s"
```
#### Max Map Count
* Description: Maximum number of active memory maps Pilosa will use for fragment
files (actual total usage may be slightly higher). Best practice is to set
this ~10% lower than your system's maximum map count (obtained via `sysctl
vm.max_map_count` on Linux). If you plan on having lots of fragments per host,
it's a good idea to raise both the system's max map count, and Pilosa's. The
number of fragments is a function of the number of shards, fields, and time
quantums. Using, for example, YMDH time quantum fields with a wide range of
timestamps will create lots of fragments. When Pilosa exhausts the
max-map-count it falls back to reading files directly into memory. This can be
a bit slower, and cause slower restarts, but is generally fine.
* Flag: `--max-map-count=1000000`
* Env: `PILOSA_MAX_MAP_COUNT=1000000`
* Config:
```toml
max-map-count = 1000000
```
#### Max Writes Per Request
* Description: Maximum number of mutating commands allowed per request. This includes Set, Clear, SetRowAttrs, and SetColumnAttrs.
* Flag: `--max-writes-per-request=5000`
* Env: `PILOSA_MAX_WRITES_PER_REQUEST=5000`
* Config:
```toml
max-writes-per-request = 5000
```
#### Max File Count
* Description: A soft limit on the maximum number of files that Pilosa will keep
open simultaneously. When past this limit, Pilosa will only keep files open
for as long as it needs to write updates. This will negatively affect
performance in cases where Pilosa is doing lots of small updates.
* Flag: `--max-file-count=1000000`
* Env: `PILOSA_MAX_FILE_COUNT=1000000`
* Config:
```toml
max-file-count = 1000000
```
#### Gossip Advertise Host
* Description: Host on which memberlist should advertise. Defaults to `advertise` host.
* Flag: `--gossip.advertise-host=192.168.1.100`
* Env: `PILOSA_GOSSIP_ADVERTISE_HOST=192.168.1.100
* Config:
```toml
[gossip]
advertise-host = 192.168.1.100
```
#### Gossip Advertise Port
* Description: Port on which memberlist should advertise. Defaults to `advertise` port.
* Flag: `--gossip.advertise-port=15001`
* Env: `PILOSA_GOSSIP_ADVERTISE_PORT=15001`
* Config:
```toml
[gossip]
advertise-port = 15001
```
#### Gossip Port
* Description: Port to which Pilosa should bind for internal communication. If more than one Pilosa server is running on the same host, the gossip port for each server must be unique.
* Flag: `--gossip.port=11101`
* Env: `PILOSA_GOSSIP_PORT=11101`
* Config:
```toml
[gossip]
port = 11101
```
#### Gossip Seeds
* Description: This specifies which internal host(s) should be used to initialize membership in the cluster. Typically this can be the address of any available host in the cluster. For example, when starting a three-node cluster made up of `node0`, `node1`, and `node2`, the `gossip.seeds` for all three nodes can be configured to be the address of `node0`. Multiple seeds should be comma-separated in the flag and env forms.
* Flag: `--gossip.seeds="localhost:11101,localhost:11110"`
* Env: `PILOSA_GOSSIP_SEEDS="localhost:11101,localhost:11110"`
* Config:
```toml
[gossip]
seeds = ["localhost:11101", "localhost:11110"]
```
#### Gossip Key
* Description: Path to the file which contains the key to encrypt gossip communication. The contents of the file should be either 16, 24, or 32 bytes to select AES-128, AES-192, or AES-256 encryption. You can read from `/dev/random` device on UNIX-like systems to create the key file; e.g., `head -c 32 /dev/random > gossip.key32` creates a key file to use AES-256.
* Flag: `--gossip.key="/var/secret/gossip.key32"`
* Env: `PILOSA_GOSSIP_KEY="/var/secret/gossip.key32"`
* Config:
```toml
[gossip]
key = "/var/secret/gossip.key32"
```
#### Cluster Long Query Time
* Description (DEPRICATED, see Long Query Time): Duration that will trigger log and stat messages for slow queries.
* Flag: `cluster.long-query-time="1m0s"`
* Env: `PILOSA_CLUSTER_LONG_QUERY_TIME="1m0s"`
* Config:
```toml
[cluster]
long-query-time = "1m0s"
```
#### Cluster Coordinator
* Description: Indicates whether the node should act as the coordinator for the cluster. Only one node per cluster should be the coordinator.
* Flag: `cluster.coordinator`
* Env: `PILOSA_CLUSTER_COORDINATOR`
* Config:
```toml
[cluster]
coordinator = true
```
#### Cluster Replicas
* Description: Number of hosts each piece of data should be stored on.
* Flag: `cluster.replicas=1`
* Env: `PILOSA_CLUSTER_REPLICAS=1`
* Config:
```toml
[cluster]
replicas = 1
```
#### Cluster Type
* Description: Determine how the cluster handles membership and state sharing. Choose from [static, gossip].
* static - Messaging between nodes is disabled. This is primarily used for testing.
* gossip - Messages are transmitted over TCP. Cluster status and node state are kept in sync via internode gossip.
* Flag: `cluster.type="gossip"`
* Env: `PILOSA_CLUSTER_TYPE="gossip"`
* Config:
```toml
[cluster]
type = "gossip"
```
#### Profile CPU
* Description: If this is set to a path, collect a cpu profile and store it there.
* Flag: `--profile.cpu="/path/to/somewhere"`
* Env: `PILOSA_PROFILE_CPU="/path/to/somewhere"`
* Config:
```toml
[profile]
cpu = "/path/to/somewhere"
```
#### Profile CPU Time
* Description: Amount of time to collect cpu profiling data at startup if `profile.cpu` is set.
* Flag: `--profile.cpu-time="30s"`
* Env: `PILOSA_PROFILE_CPU_TIME="30s"`
* Config:
```toml
[profile]
cpu-time = "30s"
```
#### Metric Service
* Description: Which stats service to use for collecting [metrics](../administration/#metrics). Choose from [statsd, expvar, prometheus, none].
* Flag: `--metric.service=statsd`
* Env: `PILOSA_METRIC_SERVICE=statsd`
* Config:
```toml
[metric]
service = "statsd"
```
#### Metric Host
* Description: Address of the StatsD service host.
* Flag: `--metric.host=localhost:8125`
* Env: `PILOSA_METRIC_HOST=localhost:8125`
* Config:
```toml
[metric]
host = "localhost:8125"
```
#### Metric Poll Interval
* Description: Rate at which runtime metrics (such as open file handles and memory usage) are collected.
* Flag: `metric.poll-interval="0m15s"`
* Env: `PILOSA_METRIC_POLL_INTERVAL=0m15s`
* Config:
```toml
[metric]
poll-interval = "0m15s"
```
#### Metric Diagnostics
* Description: Enable [reporting](../administration/#diagnostics) of limited usage statistics to Pilosa developers. To disable, set to false.
* Flag: `metric.diagnostics`
* Env: `PILOSA_METRIC_DIAGNOSTICS`
* Config:
```toml
[metric]
diagnostics = true
```
#### TLS Certificate
* Description: Path to the TLS certificate to use for serving HTTPS. Usually has one of `.crt` or `.pem` extensions.
* Flag: `tls.certificate=/srv/pilosa/certs/server.crt`
* Env: `PILOSA_TLS_CERTIFICATE=/srv/pilosa/certs/server.crt`
* Config:
```toml
[tls]
certificate = "/srv/pilosa/certs/server.crt"
```
#### TLS Certificate Key
* Description: Path to the TLS certificate key to use for serving HTTPS. Usually has the `.key` extension.
* Flag: `tls.key=/srv/pilosa/certs/server.key`
* Env: `PILOSA_TLS_KEY=/srv/pilosa/certs/server.key`
* Config:
```toml
[tls]
key = "/srv/pilosa/certs/server.key"
```
#### TLS CA Certificate
* Description: Path to the TLS certificate key to use for serving HTTPS. Usually has one of `.crt` or `.pem` extensions.
* Flag: `tls.ca-certificate=/srv/pilosa/certs/ca-chain.pem`
* Env: `PILOSA_TLS_CA_CERTIFICATE=/srv/pilosa/certs/ca-chain.pem`
* Config:
```toml
[tls]
ca-certificate = "/srv/pilosa/certs/ca-chain.pem"
```
#### TLS Skip Verify
* Description: Disables verification for checking TLS certificates. This configuration item is mainly useful for using self-signed certificates for a Pilosa cluster. Do not use in production since it makes man-in-the-middle attacks trivial.
* Flag: `tls.skip-verify`
* Env: `PILOSA_TLS_SKIP_VERIFY`
* Config:
```toml
[tls]
skip-verify = true
```
#### TLS Enable Client Certificate Verification
* Description: Enables verification of client certificates on incoming HTTPS requests for mutual TLS authentication.
* Flag: `tls.enable-client-verification`
* Env: `PILOSA_TLS_ENABLE_CLIENT_VERIFICATION`
* Config:
```toml
[tls]
enable-client-verification = true
```
#### Tracing Sampler Type
* Description: Jaeger sampler type (const, probabilistic, ratelimiting, or remote). Set to 'off' to disable tracing completely. Default is 'off'.
* Flag: `tracing.sampler-type`
* Env: `PILOSA_TRACING_SAMPLER_TYPE`
* Config:
```toml
[tracing]
sampler-type = "remote"
```
#### Tracing Sampler Parameter
* Description: Jaeger sampler parameter (number)
* Flag: `tracing.sampler-param`
* Env: `PILOSA_TRACING_SAMPLER_PARAM`
* Config:
```toml
[tracing]
sampler-param = 0.001
```
#### Tracing Agent Host/Port
* Description: Jaeger agent host:port
* Flag: `tracing.agent-host-port`
* Env: `PILOSA_TRACING_AGENT_HOST_PORT`
* Config:
```toml
[tracing]
agent-host-port = "localhost:6831"
```
#### Profile Block Rate
* Description: Block Rate is passed directly to Go's
[runtime.SetBlockProfileRate](https://golang.org/pkg/runtime/#SetBlockProfileRate). Goroutine blocking events will be sampled at 1
per `rate` nanoseconds. A value of "1" samples every event, and 0 disables
profiling.
* Flag: `--profile.block-rate=10000000`
* Env: `PILOSA_PROFILE_BLOCK_RATE=10000000`
* Config:
```toml
[profile]
block-rate = 10000000
```
#### Profile Mutex Fraction
* Description: Mutex Fraction is passed directly to Go's
[runtime.SetMutexProfileFraction](https://golang.org/pkg/runtime/#SetMutexProfileFraction). 1/`fraction` of events will be sampled.
* Flag: `--profile.mutex-fraction=100`
* Env: `PILOSA_PROFILE_MUTEX_FRACTION=100`
* Config:
```toml
[profile]
mutex-fraction = 100
```
#### Translation Map Size
* Description: Size in bytes of mmap to allocate for key translation
* Flag: `translation.map-size`
* Env: `PILOSA_TRANSLATION_MAP_SIZE`
* Config:
```toml
[translation]
map-size = 10737418240
```
### Example Cluster Configuration
A three node cluster running on different hosts could be minimally configured as follows:
#### Node 0
data-dir = "/home/pilosa/data"
bind = "node0.pilosa.com:10101"
[gossip]
port = 12000
seeds = ["node0.pilosa.com:12000"]
[cluster]
replicas = 1
coordinator = true
#### Node 1
data-dir = "/home/pilosa/data"
bind = "node1.pilosa.com:10101"
[gossip]
port = 12000
seeds = ["node0.pilosa.com:12000"]
[cluster]
replicas = 1
coordinator = false
#### Node 2
data-dir = "/home/pilosa/data"
bind = "node2.pilosa.com:10101"
[gossip]
port = 12000
seeds = ["node0.pilosa.com:12000"]
[cluster]
replicas = 1
coordinator = false
### Example Cluster Configuration (HTTPS)
The same cluster which uses HTTPS instead of HTTP can be configured as follows. Note that we explicitly specify `https` as the protocol in `bind` and `cluster.hosts` configuration. It is not required to use a gossip key but it is highly recommended:
#### Node 0
data-dir = "/home/pilosa/data"
bind = "https://node0.pilosa.com:10101"
[gossip]
port = 12000
seeds = ["node0.pilosa.com:12000"]
key = "/home/pilosa/private/gossip.key32"
[cluster]
replicas = 1
coordinator = true
[tls]
certificate = "/home/pilosa/private/server.crt"
key = "/home/pilosa/private/server.key"
#### Node 1
data-dir = "/home/pilosa/data"
bind = "https://node1.pilosa.com:10101"
[gossip]
port = 12000
seeds = ["node0.pilosa.com:12000"]
key = "/home/pilosa/private/gossip.key32"
[cluster]
replicas = 1
coordinator = false
[tls]
certificate = "/home/pilosa/private/server.crt"
key = "/home/pilosa/private/server.key"
#### Node 2
data-dir = "/home/pilosa/data"
bind = "https://node2.pilosa.com:10101"
[gossip]
port = 12000
seeds = ["node0.pilosa.com:12000"]
key = "/home/pilosa/private/gossip.key32"
[cluster]
replicas = 1
coordinator = false
[tls]
certificate = "/home/pilosa/private/server.crt"
key = "/home/pilosa/private/server.key"
### Example Cluster Configuration (HTTPS, same host)
You can run a cluster on the same host using the configuration above with a few changes. Gossip port and bind address should be different for each node and a data directory should be accessed only by a single node.
#### Node 0
data-dir = "/home/pilosa/data0"
bind = "https://localhost:10100"
[gossip]
port = 12000
seeds = ["localhost:12000"]
key = "/home/pilosa/private/gossip.key32"
[cluster]
replicas = 1
coordinator = true
[tls]
certificate = "/home/pilosa/private/server.crt"
key = "/home/pilosa/private/server.key"
#### Node 1
data-dir = "/home/pilosa/data1"
bind = "https://localhost:10101"
[gossip]
port = 12001
seeds = ["localhost:12000"]
key = "/home/pilosa/private/gossip.key32"
[cluster]
replicas = 1
coordinator = false
[tls]
certificate = "/home/pilosa/private/server.crt"
key = "/home/pilosa/private/server.key"
#### Node 2
data-dir = "/home/pilosa/data2"
bind = "https://localhost:10102"
[gossip]
port = 12002
seeds = ["localhost:12000"]
key = "/home/pilosa/private/gossip.key32"
[cluster]
replicas = 1
coordinator = false
[tls]
certificate = "/home/pilosa/private/server.crt"
key = "/home/pilosa/private/server.key"

58
docs/console.md
View file

@ -1,58 +0,0 @@
+++
title = "Console"
weight = 9
nav = [
"Installation",
"Query",
"Cluster Admin",
]
+++
## Console
A web-based app called Pilosa Console is available in a separate package. This can be used for constructing queries and viewing the cluster status.
### Installation
Releases are [available on Github](https://github.com/pilosa/console/releases) as well as on [Homebrew](https://brew.sh/) for Mac.
Installing on a Mac with Homebrew is simple; just run:
```
brew tap pilosa/homebrew-pilosa
brew install pilosa-console
```
You may also build from source by checking out the [repo on Github](https://github.com/pilosa/console) and running:
```
make install
```
### Query
The Query tab allows you to enter [PQL](../query-language/) queries and run them against your locally running server. First you must select an Index with the Select index dropdown.
Each query's result will be displayed in the Output section along with the query time.
The Console will keep a record of each query and its result with the latest query on top.
![Console screenshot](/img/docs/webui-console.png)
*Console query screenshot*
In addition to standard PQL, the console supports a few special commands, prefixed with `:`.
- `:create index <indexname>`
- `:delete index <indexname>`
- `:use <indexname>`
- `:create field <fieldname>`
- `:delete field <fieldname>`
Field creation also supports options like `timeQuantum`. When creating a new field, add options by using the keys documented in [API reference](../api-reference/#create-field).
- `:create field <fieldname> cacheSize=10000`
### Cluster Admin
Use the Cluster Admin tab to view the current status of your cluster. This contains information on each node in the cluster, plus the list of Indexes and Fields.

217
docs/data-model.md
View file

@ -1,217 +0,0 @@
+++
title = "Data Model"
weight = 5
nav = [
"Overview",
"Index",
"Column",
"Row",
"Field",
"Time Quantum",
"Attribute",
"Shard",
]
+++
## Data Model
### Overview
The central component of Pilosa's data model is a boolean matrix. Each cell in the matrix is a single bit; if the bit is set, it indicates that a relationship exists between that particular row and column.
Rows and columns can represent anything (they could even represent the same set of things as in a [bigraph](https://en.wikipedia.org/wiki/Bigraph)). Pilosa can associate arbitrary key/value pairs (referred to as attributes) to rows and columns, but queries and storage are optimized around the core matrix.
Pilosa lays out data first in rows, so queries which get all the set bits in one or many rows, or compute a combining operation—such as Intersect or Union—on multiple rows, are the fastest. Pilosa categorizes rows into different *fields* and quickly retrieves the top rows in a field sorted by the number of columns set in each row.
Please note that Pilosa is most performant when row and column IDs are sequential starting from 0. You can deviate from this to some degree, but setting a bit with column ID 2<sup>63</sup> on a single-node cluster, for example, will not work well due to memory limitations.
![basic data model diagram](/img/docs/data-model.png)
*Basic data model diagram*
### Index
The purpose of the Index is to represent a data namespace. You cannot perform cross-index queries.
### Column
Column ids are sequential, increasing integers and they are common to all Fields within an Index. A single column often corresponds to a record in a relational table, although other configurations are possible, and sometimes preferable.
### Row
Row ids are sequential, increasing integers namespaced to each Field within an Index.
### Field
Fields are used to segment rows within an index, for example to define different functional groups. A Pilosa field might correspond to a single field in a relational table, where each row in a standard Pilosa field represents a single possible value of the relational field. Similarly, an integer field could represent all possible integer values of a relational field.
#### Relational Analogy
The Pilosa index is a flexible structure; it can represent any sort of high-cardinality binary matrix. We have explored a number of modeling patterns in Pilosa use cases; one accessible example is a direct analogy to the relational model, summarized here.
Entities:
Relational | Pilosa
-------------|----------------------------------------------
Database | N/A *(internal: Holder)*
Table | Index
Row | Column
Column | Field
Value | Row
Value (int) | Field.Value (see [BSI](#bsi-range-encoding))
Simple queries:
Relational | Pilosa
-----------------------------------------------|------------------------------------
`select ID from People where Name = 'Bob'` | `Row(Name="Bob")`
`select ID from People where Age > 30` | `Row(Age > 30)`
`select ID from People where Member = true` | `Row(Member=0)`
Note that `Row(Member=0)` selects all entities with a bit set in row 0 of the Member field. We could just as well use row 1 to store this, in which case we would use `Row(Member=1)`, which looks a bit more intuitive. In the relational model, joins are often necessary. Because Pilosa supports extremely high cardinality in both rows and columns, many types of joins are accomplished with basic Pilosa queries across multiple fields. For example, this SQL join:
```sql
select AVG(p.Age) from People p
inner join PersonCar pc on pc.PersonID=p.ID
inner join Cars c on pc.CarID=c.ID
where c.Make = 'Ford'
```
can be accomplished with a Pilosa query like this (note that [Sum](../query-language/#sum) returns a json object containing both the sum and count, from which the average is easily computed):
```pql
Sum(Row(Car-Make="Ford"), field=Age)
```
This is one major component of Pilosa's ability to combine relationships from multiple data stores.
#### Ranked
Ranked Fields maintain a sorted cache of column counts by Row ID (yielding the top rows by columns with a bit set in each). This cache facilitates the TopN query. The cache size defaults to 50,000 and can be set at Field creation.
![ranked field diagram](/img/docs/field-ranked.png)
*Ranked field diagram*
#### LRU
The LRU cache maintains the most recently accessed Rows.
![lru field diagram](/img/docs/field-lru.png)
*LRU field diagram*
### Time Quantum
Setting a time quantum on a field creates extra views which allow ranged Row queries down to the time interval specified. For example, if the time quantum is set to `YMD`, ranged Row queries down to the granularity of a day are supported.
### Attribute
Attributes are arbitrary key/value pairs that can be associated with either rows or columns. This metadata is stored in a separate BoltDB data structure.
Column-level attributes are common across an index. That is, each column attribute applies to all bits in the corresponding column, across all fields in an index. Row attributes apply to all bits in the corresponding row.
### Shard
Indexes are segmented into groups of columns called shards (previously known as slices). Each shard contains a fixed number of columns, which is the ShardWidth. ShardWidth is a constant that can only be modified at compile time, and before ingesting data. The default value is 2<sup>20</sup>.
Query operations run in parallel, and they are evenly distributed across a cluster via a consistent hash algorithm.
### Field Type
Upon creation, fields are configured to be of a certain type. Pilosa supports the following field types: `set`, `int`, `bool`, `time`, and `mutex`.
#### Set
Set is the default field type in Pilosa. Set fields represent a standard, binary matrix of rows and columns where each row key represents a possible field value. The following example creates a `set` field called "info" with a ranked cache containing up to 100,000 records.
Row and/or column key can be a string literal (e.g. "value"). This mapping is also stored in a separate BoltDB data structure. Becauase BoltDB does not allow to have empty strings as keys, in pilosa we translate an empty string key into sentinel byte slice:
```go
[]byte{
0x00, 0x00, 0x00,
0x4d, 0x54, 0x4d, 0x54, // MTMT
0x00,
0xc2, 0xa0, // NO-BREAK SPACE
0x00,
}
```
(where the first three bytes are _zero_ bytes, next four bytes stands for `MTMT` literal and the rest four bytes represent NBSP prefixed and suffixed with _zero_ byte).
In reverse translation, if we get from BoltDB the sentinel key, pilosa will rewrite it into an empty string (`""`).
``` request
curl localhost:10101/index/repository/field/info \
-X POST \
-d '{"options": {"type": "set", "cacheType": "ranked", "cacheSize":100000}}'
```
``` response
{"success":true}
```
#### Int
Fields of type `int` are used to store integer values. Integer fields share the same columns as the other fields in the index, but values for the field must be integers that fall between the `min` and `max` values specified when creating the field. The following example creates an `int` field called "quantity" capable of storing values from -1000 to 2000:
``` request
curl localhost:10101/index/repository/field/quantity \
-X POST \
-d '{"options": {"type": "int", "min": -1000, "max":2000}}'
```
``` response
{"success":true}
```
##### BSI Range-Encoding
Bit-Sliced Indexing (BSI) is the storage method Pilosa uses to represent multi-bit integers in a bitmap index. Integers are stored as n-bit, range-encoded bit-sliced indexes of base-2, along with an additional row indicating "not null". This means that a 16-bit integer will require 17 rows: one for each 0-bit of the 16 bit-slice components (the 1-bit does not need to be stored because with range-encoding the highest bit position is always 1) and one for the non-null row. Pilosa can evaluate `Row`, `Min`, `Max`, and `Sum` queries on these BSI integers. The result of a `Sum` query includes a count, which can be used to compute an average with no other overhead.
Internally Pilosa stores each BSI `field` as a `view`. The rows of the `view` contain the base-2 representations of the integer values. Pilosa manages the base-2 offset and translation that efficiently packs the integer value within the minimum set of rows.
For example, the following `Set()` queries executed against BSI fields will result in the data described in the diagram below:
```
Set(1, A=1)
Set(2, A=2)
Set(3, A=3)
Set(4, A=7)
Set(2, B=1)
Set(3, B=6)
```
![BSI field diagram](/img/docs/field-bsi.png)
*BSI field diagram*
Check out this [blog post](/blog/range-encoded-bitmaps/) for some more details about BSI in Pilosa.
###### BSI Deprecated Format
The original implementation of BSI required a fixed bit depth when creating fields because the existence bit was written to the bit above the highest bit. The second version of BSI moves the existence bit to the beginning, adds a negative bit as the second bit, and shifts all remaining bits up by two.
Pilosa automatically converts all old data to the new format on startup, however, this can cause issues when upgrading Pilosa and then reverting back to an old version. This documentation section exists as a record for anyone who experiences unusual behavior in BSI between versions.
#### Time
Time fields are similar to `set` fields, but in addition to row and column information, they also store a per-bit time value down to a defined granularity. The following example creates a `time` field called "event" which stores timestamp information down to a day granularity.
``` request
curl localhost:10101/index/repository/field/event \
-X POST \
-d '{"options": {"type": "time", "timeQuantum": "YMD"}}'
```
``` response
{"success":true}
```
With `time` fields, data views are generated for each of the defined time segments. For example, for a field with a time quantum of `YMD`, the following `Set()` queries will result in the data described in the diagram below:
```
Set(3, A=8, 2017-05-18T00:00)
Set(3, A=8, 2017-05-19T00:00)
```
![time quantum field diagram](/img/docs/field-time-quantum.png)
*Time quantum field diagram*
#### Mutex
Mutex fields are similar to `set` fields, with the distinction of requiring the row value for each column to be mutually exclusive. In other words, each column can only have a single value for the field. If the field value for a column is updated on a `mutex` field, then the previous field value for that column will be cleared. This field type is like a field in an RDBMS table where every record contains a single value for a particular field.
#### Boolean
A boolean field is similar to a `mutex` field tracking only two values: `true` and `false`. Boolean fields do not maintain a sorted cache, nor do they support key values.

221
docs/examples.md
View file

@ -1,221 +0,0 @@
+++
title = "Examples"
weight = 4
nav = [
"Transportation",
]
+++
## Examples
### Transportation
#### Introduction
New York City released an extremely detailed data set of over 1 billion taxi rides taken in the city - this data has become a popular target for analysis by tech bloggers and has been very well studied. For this reason, we thought it would be interesting to import this data to Pilosa in order to compare with other data stores and techniques on the exact same data set.
Transportation in general is a compelling use case for Pilosa as it often involves multiple disparate data sources, as well as high rate, real time, and extremely large amounts of data (particularly if one wants to draw reasonable conclusions).
We've written a tool to help import the NYC taxi data into Pilosa - this tool is part of the [PDK](../pdk/) (Pilosa Development Kit), and takes advantage of a number of reusable modules that may help you import other data as well. Follow along and we'll explain the whole process step by step.
After initial setup, the PDK import tool does everything we need to define a Pilosa schema, map data to bitmaps accordingly, and import it into Pilosa.
#### Data Model
The NYC taxi data is comprised of a number of csv files listed here: http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml. These data files have around 20 columns, about half of which are relevant to the benchmark queries we're looking at:
* Distance: miles, floating point
* Fare: dollars, floating point
* Number of passengers: integer
* Dropoff location: latitude and longitude, floating point
* Pickup location: latitude and longitude, floating point
* Dropoff time: timestamp
* Pickup time: timestamp
We import these fields, creating one or more Pilosa fields from each of them:
field |mapping
------------|---------------------
cab_type |direct map of enum int → row ID
dist_miles |round(dist) → row ID
total_amount_dollars |round(dist) → row ID
passenger_count |direct map of integer value → row ID
drop_grid_id |(lat, lon) → 100x100 rectangular grid → cell ID
drop_year |year(timestamp) → row ID
drop_month |month(timestamp) → row ID
drop_day |day(timestamp) → row ID
drop_time |time of day mapped to one of 48 half-hour buckets
pickup_grid_id |(lat, lon) → 100x100 rectangular grid → cell ID
pickup_year |year(timestamp) → row ID
pickup_month |month(timestamp) → row ID
pickup_day |day(timestamp) → row ID
pickup_time |time of day mapped to one of 48 half-hour buckets → row ID
We also created two extra fields that represent the duration and average speed of each ride:
field |mapping
--------------------|-------------
duration_minutes |round(drop_timestamp - pickup_timestamp) → row ID
speed_mph |round(dist_miles / (drop_timestamp - pickup_timestamp)) → row ID
#### Mapping
Each column that we want to use must be mapped to a combination of fields and row IDs according to some rule. There are many ways to approach this mapping, and the taxi dataset gives us a good overview of possibilities.
##### 0 columns → 1 field
**cab_type**: contains one row for each type of cab. Each column, representing one ride, has a bit set in exactly one row of this field. The mapping is a simple enumeration, for example yellow=0, green=1, etc. The values of the bits in this field are determined by the source of the data. That is, we're importing data from several disparate sources: NYC yellow taxi cabs, NYC green taxi cabs, and Uber cars. For each source, the single row to be set in the cab_type field is constant.
##### 1 column → 1 field
The following three fields are mapped in a simple direct way from single columns of the original data.
**dist_miles:** each row represents rides of a certain distance. The mapping is simple: as an example, row 1 represents rides with a distance in the interval [0.5, 1.5]. That is, we round the floating point value of distance to an integer, and use that as the row ID directly. Generally, the mapping from a floating point value to a row ID could be arbitrary. The rounding mapping is concise to implement, which simplifies importing and analysis. As an added bonus, it's human-readable. We'll see this pattern used several times.
In PDK parlance, we define a Mapper, which is simply a function that returns integer row IDs. PDK has a number of predefined mappers that can be described with a few parameters. One of these is LinearFloatMapper, which applies a linear function to the input, and casts it to an integer, so the rounding is handled implicitly. In code:
```go
lfm := pdk.LinearFloatMapper{
Min: -0.5,
Max: 3600.5,
Res: 3601,
}
```
`Min` and `Max` define the linear function, and `Res` determines the maximum allowed value for the output row ID - we chose these values to produce a "round to nearest integer" behavior. Other predefined mappers have their own specific parameters, usually two or three.
This mapper function is the core operation, but we need a few other pieces to define the overall process, which is encapsulated in the ColumnMapper object. This object defines which field(s) of the input data source to use (`Fields`), how to parse them (`Parsers`), what mapping to use (`Mapper`), and the name of the field to use (`Field`). <!-- TODO update so this makes sense -->
```go
pdk.ColumnMapper{
Field: "dist_miles",
Mapper: lfm,
Parsers: []pdk.Parser{pdk.FloatParser{}},
Fields: []int{fields["trip_distance"]},
},
```
These same objects are represented in the JSON definition file:
```go
{
"Fields": {
"Trip_distance": 10
},
"Mappers": [
{
"Name": "lfm0",
"Min": -0.5,
"Max": 3600.5,
"Res": 3600
}
],
"ColumnMappers": [
{
"Field": "dist_miles",
"Mapper": {
"Name": "lfm0"
},
"Parsers": [
{"Name": "FloatParser"}
],
"Fields": "Trip_distance"
}
]
}
```
Here, we define a list of Mappers, each including a name, which we use to refer to the mapper later, in the list of ColumnMappers. We can also do this with Parsers, but a few simple Parsers that need no configuration are available by default. We also have a list of Fields, which is simply a map of field names (in the source data) to column indices (in Pilosa). We use these names in the ColumnMapper definitions to keep things human-readable.
**total_amount_dollars:** Here we use the rounding mapping again, so each row represents rides with a total cost that rounds to the row's ID. The ColumnMapper definition is very similar to the previous one.
**passenger_count:** This column contains small integers, so we use one of the simplest possible mappings: the column value is the row ID.
##### 1 column → multiple fields
When working with a composite data type like a timestamp, there are plenty of mapping options. In this case, we expect to see interesting periodic trends, so we want to encode the cyclic components of time in a way that allows us to look at them independently during analysis.
We do this by storing time data in four separate fields for each timestamp: one each for the year, month, day, and time of day. The first three are mapped directly. For example, a ride with a date of 2015/06/24 will have a bit set in row 2015 of field "year", row 6 of field "month", and row 24 of field "day".
We might continue this pattern with hours, minutes, and seconds, but we don't have much use for that level of precision here, so instead we use a "bucketing" approach. That is, we pick a resolution (30 minutes), divide the day into buckets of that size, and create a row for each one. So a ride with a time of 6:45AM has a bit set in row 13 of field "time_of_day".
We do all of this for each timestamp of interest, one for pickup time and one for dropoff time. That gives us eight total fields for two timestamps: pickup_year, pickup_month, pickup_day, pickup_time, drop_year, drop_month, drop_day, drop_time.
##### Multiple columns → 1 field
The ride data also contains geolocation data: latitude and longitude for both pickup and dropoff. We just want to be able to produce a rough overview heatmap of ride locations, so we use a grid mapping. We divide the area of interest into a 100x100 grid in latitude-longitude space, label each cell in this grid with a single integer, and use that integer as the row ID.
We do all of this for each location of interest, one for pickup and one for dropoff. That gives us two fields for two locations: pickup_grid_id, drop_grid_id.
Again, there are many mapping options for location data. For example, we might convert to a different coordinate system, apply a projection, or aggregate locations into real-world regions such as neighborhoods. Here, the simple approach is sufficient.
##### Complex mappings
We also anticipate looking for trends in ride duration and speed, so we want to capture this information during the import process. For the field `duration_minutes`, we compute a row ID as `round((drop_timestamp - pickup_timestamp).minutes)`. For the field `speed_mph`, we compute row ID as `round(dist_miles / (drop_timestamp - pickup_timestamp).minutes)`. These mapping calculations are straightforward, but because they require arithmetic operations on multiple columns, they are a bit too complex to capture in the basic mappers available in PDK. Instead, we define custom mappers to do the work:
```go
durm := pdk.CustomMapper{
Func: func(fields ...interface{}) interface{} {
start := fields[0].(time.Time)
end := fields[1].(time.Time)
return end.Sub(start).Minutes()
},
Mapper: lfm,
}
```
#### Import process
After designing this schema and mapping, we capture it in a JSON definition file that can be read by the PDK import tool. Running `pdk taxi` runs the import based on the information in this file. For more details, see the [PDK](../pdk/) section, or check out the [code](https://github.com/pilosa/pdk/tree/master/usecase/taxi) itself.
#### Queries
Now we can run some example queries.
Count per cab type can be retrieved, sorted, with a single PQL call.
```request
TopN(cab_type)
```
```response
{"results":[[{"id":1,"count":1992943},{"id":0,"count":7057}]]}
```
High traffic location IDs can be retrieved with a similar call. These IDs correspond to latitude, longitude pairs, which can be recovered from the mapping that generates the IDs.
```request
TopN(pickup_grid_id)
```
```response
{"results":[[{"id":5060,"count":40620},{"id":4861,"count":38145},{"id":4962,"count":35268},...]]}
```
Average of `total_amount` per `passenger_count` can be computed with some postprocessing. We use a small number of `TopN` calls to retrieve counts of rides by passenger_count, then use those counts to compute an average.
```python
import pilosa
client = pilosa.Client()
schema = client.schema()
taxi = schema.index("taxi")
passenger_count = taxi.field("passenger_count")
total_amount_dollars = taxi.field("total_amount_dollars")
queries = []
pcounts = range(10)
for i in pcounts:
queries.append(total_amount_dollars.topn(passenger_count.row(i))
query = taxi.batch_query(**queries)
results = client.query(query)
resp = requests.post(qurl, data=queries)
average_amounts = []
for pcount, result in zip(pcounts, resp.results):
wsum = sum([r.count * r.id for r in result.count_items])
count = sum([r.count for r in result.count_items])
average_amounts.append(float(wsum)/count)
```
<div class="note">
Note that the <a href="../data-model/#bsi-range-encoding">BSI</a>-powered <a href="../query-language/#sum">Sum</a> query now provides an alternative approach to this kind of query.
</div>
<!-- Disabled until we have the time to update the Jupyter notebook --YT
For more examples and details, see this [ipython notebook](https://github.com/pilosa/notebooks/blob/master/taxi-use-case.ipynb).
-->

43
docs/faq.md
View file

@ -1,43 +0,0 @@
+++
title = "FAQ"
weight = 15
nav = []
+++
## FAQ
### What is Pilosa?
Pilosa is an in-memory, distributed index that is layered over persistent storage. It supports fast ad-hoc queries and segmentation. Pilosa does not require the underlying data to be moved, rather it can be populated in conjunction with data writes, or it can be backfilled asynchronously from any other data store or event processing system. This allows Pilosa to support sub-second queries against very large underlying data sets.
### Is Pilosa a database?
Pilosa is not a database in the traditional sense. While Pilosa does store data (both in-memory as well as persisted to disk), it wouldn't typically be used as a primary data store. Instead, one would likely use Pilosa as an index of the data stored in a traditional database or in a data warehouse.
### Where does Pilosa fit in my stack?
Pilosa was designed to index the relationships in your data. Pilosa runs along with your existing stack, integrating with one or more backing data stores. Pilosa can connect through a stream platform like Kafka or application integration via [PDK](../pdk/).
### How is Pilosa different from Elasticsearch since they are both indexes?
Elasticsearch is a search engine based on Lucene, and is therefore very good at indexing and searching large volumes of unstructured text. As it matures, Elasticsearch has continued to move into the analytics space, but its core data object is still the "document". Pilosa is specifically designed to index structured data and improve query speed. By representing data as the relationship between objects, and then storing those relationships in bitmaps, Pilosa can very efficiently search and compare many millions of data points while still maintaining a small memory footprint.
### How do I get my data into Pilosa?
There are typically two methods for getting data into Pilosa: importing large batches of data from an existing data set, and continuously updating Pilosa as data is added or updated.
In the first case, one would use the `pilosa import` command to bulk load structured data into Pilosa. In order to improve this process, one can use the Pilosa Development Kit (PDK) to map structured data in the original data set onto the Pilosa schema.
For the case where data is continually mutating, one would apply a parallel data writer at the point at which data is written to the persistent data store. This new writer would simultaneously write to Pilosa. An example use case would be one where Kafka was employed as the message broker in your data pipeline, you could introduce an additional Kafka consumer to read from the message log and write mutated data to Pilosa.
### What languages can I use with it?
There is currently [client support](../client-libraries/) for [Go](https://github.com/pilosa/go-pilosa), [Python](https://github.com/pilosa/python-pilosa), and [Java](https://github.com/pilosa/java-pilosa). If you want to use Pilosa with a different language, you can access Pilosa via the [Pilosa API](../api-reference/).
### Do you query Pilosa using SQL?
One can access Pilosa directly via the terminal using the [Pilosa Query Language](../query-language/) (PQL), but a typical implementation would use one of the Pilosa client libraries to integrate with an existing codebase. There is currently client support for Go, Python, and Java.
### Replication on each node?
Pilosa supports a replication factor greater than or equal to one. When replication is configured to be greater than one, then all mutations will be replicated to additional nodes in the cluster. For example, in a five-node cluster consisting of nodes A-B-C-D-E and with replication factor of three, then a write to node B will result in data being written to nodes B, C, and D. If the replication factor is greater than the number of nodes in the cluster, the data will be replicated to every node in the cluster only once.

970
docs/getting-started.md
View file

@ -1,970 +0,0 @@
+++
title = "Getting Started"
weight = 3
nav = [
"Starting Pilosa",
"Sample Project",
"Using Curl",
"Using Go",
"Using Java",
"Using Python",
"What's Next?",
]
+++
## Getting Started
Pilosa supports an HTTP interface which uses JSON by default.
Any HTTP tool can be used to interact with the Pilosa server. The examples in this documentation will use [curl](https://curl.haxx.se/) which is available by default on many UNIX-like systems including Linux and MacOS. However, the best way to interface with the Pilosa server is through one of our three official client libraries. Pilosa currently supports [Go](https://github.com/pilosa/go-pilosa), [Java](https://github.com/pilosa/java-pilosa), and [Python](https://github.com/pilosa/python-pilosa).
<div class="note">
<p>Note that Pilosa server requires a high limit for open files. Check the documentation of your system to see how to increase it in case you hit that limit. See <a href="/docs/administration/#open-file-limits">Open File Limits</a> for more details.</p>
</div>
### Starting Pilosa
Follow the steps in the [Installation](../installation/) document to install Pilosa.
Execute the following in a terminal to run Pilosa with the default configuration (Pilosa will be available at [localhost:10101](http://localhost:10101)):
```
pilosa server
```
Let's make sure Pilosa is running:
``` request
curl localhost:10101/status
```
``` response
{"state":"NORMAL","nodes":[{"id":"91715a50-7d50-4c54-9a03-873801da1cd1","uri":{"scheme":"http","host":"localhost","port
":10101},"isCoordinator":true}],"localID":"91715a50-7d50-4c54-9a03-873801da1cd1"}
```
### Sample Project
In order to better understand Pilosa's capabilities, we will create a sample project called "Star Trace" containing information about 1,000 popular Github repositories which have "go" in their name. The Star Trace index will include data points such as programming language and stargazers—people who have starred a project.
Although Pilosa doesn't keep the data in a tabular format, we still use the terms "columns" and "rows" when describing the data model. We put the primary objects in columns, and the properties of those objects in rows. For example, the Star Trace project will contain an index called "repository" which contains columns representing Github repositories, and rows representing properties like programming languages and stargazers. We can better organize the rows by grouping them into sets called Fields. So the "repository" index might have a "languages" field as well as a "stargazers" field. You can learn more about indexes and fields in the [Data Model](../data-model/) section of the documentation.
<div class="note">
<p>If at any time you want to verify the data structure, you can request the schema as follows:</p>
</div>
```request
curl localhost:10101/schema
```
```response
{
"indexes": [
{
"name": "repository",
"options": {
"keys": false,
"trackExistence": true
},
"fields": [
{
"name": "language",
"options": {
"type": "set",
"cacheType": "ranked",
"cacheSize": 50000,
"keys": false
}
},
{
"name": "stargazer",
"options": {
"type": "time",
"timeQuantum": "YMDH",
"keys": false,
"noStandardView": false
}
}
],
"shardWidth": 1048576
}
]
}
```
<div class="note">
<p>Note: This is the response you should receive once completing this project. It has also been formatted using <a href="https://stedolan.github.io/jq/"><code>jq</code></a>. </p>
</div>
#### Using Curl
##### Create the Schema
Before we can import data or run queries, we need to create our indexes and the fields within them. Let's create the `repository` index first:
``` request
curl localhost:10101/index/repository -X POST
```
``` response
{"success":true}
```
The index name must be 230 characters or fewer, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. The same goes for field names.
Let's create the `stargazer` field which has user IDs of stargazers as its rows:
``` request
curl localhost:10101/index/repository/field/stargazer \
-X POST \
-d '{"options": {"type": "time", "timeQuantum": "YMD"}}'
```
``` response
{"success":true}
```
Since our data contains time stamps which represent the time users starred repos, we set the field type to `time`. Time quantum is the resolution of the time we want to use, and we set it to `YMD` (year, month, day) for `stargazer`.
Next up is the `language` field, which will contain IDs for programming languages:
``` request
curl localhost:10101/index/repository/field/language \
-X POST
```
``` response
{"success":true}
```
The `language` is a `set` field, but since the default field type is `set`, we didn't specify it in field options.
##### Import Data From CSV Files
Download the `stargazer.csv` and `language.csv` files here:
```
curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/stargazer.csv
curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/language.csv
```
Run the following commands to import the data into Pilosa:
```
pilosa import -i repository -f stargazer stargazer.csv
pilosa import -i repository -f language language.csv
```
If you are using a Docker container for Pilosa (with name `pilosa`), you should instead copy the `*.csv` file into the container and then import them:
```
docker cp stargazer.csv pilosa:/stargazer.csv
docker exec -it pilosa /pilosa import -i repository -f stargazer /stargazer.csv
docker cp language.csv pilosa:/language.csv
docker exec -it pilosa /pilosa import -i repository -f language /language.csv
```
Note that both the user IDs and the repository IDs were remapped to sequential integers in the data files, they don't correspond to actual Github IDs anymore. You can check out [languages.txt](https://github.com/pilosa/getting-started/blob/master/languages.txt) to see the mapping for languages.
##### Make Some Queries
Which repositories did user 14 star:
``` request
curl localhost:10101/index/repository/query \
-X POST \
-d 'Row(stargazer=14)'
```
``` response
{
"results":[
{
"attrs":{},
"columns":[1,2,3,362,368,391,396,409,416,430,436,450,454,460,461,464,466,469,470,483,484,486,490,491,503,504,514]
}
]
}
```
What are the top 5 languages in the sample data:
``` request
curl localhost:10101/index/repository/query \
-X POST \
-d 'TopN(language, n=5)'
```
``` response
{
"results":[
[
{"id":5,"count":119},
{"id":1,"count":50},
{"id":4,"count":48},
{"id":9,"count":31},
{"id":13,"count":25}
]
]
}
```
Which repositories were starred by user 14 and 19:
``` request
curl localhost:10101/index/repository/query \
-X POST \
-d 'Intersect(
Row(stargazer=14),
Row(stargazer=19)
)'
```
``` response
{
"results":[
{
"attrs":{},
"columns":[2,3,362,396,416,461,464,466,470,486]
}
]
}
```
Which repositories were starred by user 14 or 19:
``` request
curl localhost:10101/index/repository/query \
-X POST \
-d 'Union(
Row(stargazer=14),
Row(stargazer=19)
)'
```
``` response
{
"results":[
{
"attrs":{},
"columns":[1,2,3,361,362,368,376,377,378,382,386,388,391,396,398,400,409,411,412,416,426,428,430,435,436,450,452,453,454,456,460,461,464,465,466,469,470,483,484,486,487,489,490,491,500,503,504,505,512,514]
}
]
}
```
Which repositories were starred by user 14 and 19 and also were written in language 1:
``` request
curl localhost:10101/index/repository/query \
-X POST \
-d 'Intersect(
Row(stargazer=14),
Row(stargazer=19),
Row(language=1)
)'
```
``` response
{
"results":[
{
"attrs":{},
"columns":[2,362,416,461]
}
]
}
```
Set user 99999 as a stargazer for repository 77777:
``` request
curl localhost:10101/index/repository/query \
-X POST \
-d 'Set(77777, stargazer=99999)'
```
``` response
{"results":[true]}
```
Please note that while user ID 99999 may not be sequential with the other column IDs, it is still a relatively low number.
Don't try to use arbitrary 64-bit integers as column or row IDs in Pilosa - this will lead to problems such as poor performance and out of memory errors.
#### Using Go
Pilosa follows the Go policy of supporting the two most recent major versions of Go.
##### Create the Environment
Interacting with Pilosa in your go program is best accomplished using our client, go-pilosa. To install go-pilosa, open a new terminal and download the library to your `GOPATH` using:
```
go get github.com/pilosa/go-pilosa
```
Create a project folder:
```
mkdir getting-started && cd getting-started
```
In this folder, we will download two CSV files to provide data to our fields later on. Download the `stargazer.csv` and `language.csv` files here:
```
curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/stargazer.csv
curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/language.csv
```
We will also create a file called `startrace.go` as follows:
```
touch startrace.go
```
This file will be used in the following sections.
##### Create the Schema
Before we can import data or run queries, we need to create our schema. You can see two imports from the go-pilosa repo, go-pilosa for the client, and csv for the CSV reader. Create the schema by creating a client (which will communicate our schema to Pilosa), creating a schema locally (which will contain our indexes and fields), and syncing with Pilosa. This is all done in the `startrace.go` file:
```
package main
import (
"bytes"
"fmt"
"github.com/pilosa/go-pilosa"
"github.com/pilosa/go-pilosa/csv"
"io/ioutil"
"log"
)
func main() {
// Create the Schema
client := pilosa.DefaultClient()
schema, _ := client.Schema()
// This is where the index will go later
// This is where the fields will go later
err := client.SyncSchema(schema)
if err != nil {
log.Fatal(err)
}
}
```
Next, let's create the `repository` index:
```
repository := schema.Index("repository")
```
The index name must be 230 characters or fewer, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. The same goes for field names.
Let's create the `stargazer` field which has user IDs of stargazers as its rows:
```
stargazer := repository.Field("stargazer")
```
Next up is the `language` field, which will contain IDs for programming languages:
```
language := repository.Field("language")
```
Your `startrace.go` file should look like:
```
package main
import (
"bytes"
"fmt"
"github.com/pilosa/go-pilosa"
"github.com/pilosa/go-pilosa/csv"
"io/ioutil"
"log"
)
func main() {
// Create the Schema
client := pilosa.DefaultClient()
schema, _ := client.Schema()
repository := schema.Index("repository")
stargazer := repository.Field("stargazer")
language := repository.Field("language")
err := client.SyncSchema(schema)
if err != nil {
log.Fatal(err)
}
}
```
##### Import Data From CSV Files
Now that we have our index and our fields, we can import the data we downloaded earlier and be on our way to making our own queries.
First, we will load our data into the `stargazer` field:
```
stargazerFile, err := ioutil.ReadFile("stargazer.csv")
if err != nil {
log.Fatal(err)
}
format := "2006-01-02T15:04"
iterator = csv.NewColumnIteratorWithTimestampFormat(csv.RowIDColumnID, bytes.NewReader(stargazerFile), format)
err = client.ImportField(stargazer, iterator)
if err != nil {
log.Fatal(err)
}
```
Since our `stargazer` data contains time stamps, which represent the time users starred repos, we will be using the `csv.NewColumnIteratorWithTimeStampFormat` function from the go-pilosa/csv package. This function takes the format of the csv files (`csv.RowIDColumnID`), an `io.Reader` (`bytes.NewReader(stargazerFile)`), and the time quantum format (`format`) and translates the csv file into a format Pilosa can read. Time quantum is the resolution of the time we want to use.
Next, we will load our data into the `language` field:
```
languageFile, err := ioutil.ReadFile("language.csv")
if err != nil {
log.Fatal(err)
}
iterator := csv.NewColumnIterator(csv.RowIDColumnID, bytes.NewReader(languageFile))
err = client.ImportField(language, iterator)
if err != nil {
log.Fatal(err)
}
```
Since our `language` data doesn't contain time stamps, we will use the `csv.NewColumnIterator` function in place of `csv.NewColumnIteratorWithTimeStampFormat`.
Note that both the user IDs and the repository IDs were remapped to sequential integers in the data files, they don't correspond to actual Github IDs anymore. You can check out [languages.txt](https://github.com/pilosa/getting-started/blob/master/languages.txt) to see the mapping for languages.
For more information on imports in go-pilosa, please see the go-pilosa [site](https://github.com/pilosa/go-pilosa/blob/master/docs/imports-exports.md).
##### Make Some Queries
Now that we have a working schema, we can query it.
Which repositories did user 14 star:
``` request
response, err := client.Query(stargazer.Row(14))
if err != nil {
log.Fatal(err)
}
fmt.Println("User 14 starred: ", response.Result().Row().Columns)
```
``` response
User 14 starred: [1 2 3 362 368 391 396 409 416 430 436 450 454 460 461 464 466 469 470 483 484 486 490 491 503 504 514]
```
What are the top 5 languages in the sample data:
``` request
response, err = client.Query(language.TopN(5))
if err != nil {
log.Fatal(err)
}
fmt.Println("Top Languages: ", response.Result().CountItems())
```
``` response
Top Languages: [{5 119} {1 50} {4 48} {9 31} {13 25}]
```
Which repositories were starred by user 14 and 19:
``` request
response, err = client.Query(repository.Intersect(stargazer.Row(14), stargazer.Row(19)))
if err != nil {
log.Fatal(err)
}
fmt.Println("Both user 14 and 19 starred: ", response.Result().Row().Columns)
```
``` response
Both user 14 and 19 starred: [2 3 362 396 416 461 464 466 470 486]
```
Which repositories were starred by user 14 or 19:
``` request
response, err = client.Query(repository.Union(stargazer.Row(14), stargazer.Row(19)))
if err != nil {
log.Fatal(err)
}
fmt.Println("User 14 or 19 starred: ", response.Result().Row().Columns)
```
``` response
User 14 or 19 starred: [1 2 3 361 362 368 376 377 378 382 386 388 391 396 398 400 409 411 412 416 426 428 430 435 436 450 452 453 454 456 460 461 464 465 466 469 470 483 484 486 487 489 490 491 500 503 504 505 512 514]
```
Which repositories were starred by user 14 and 19 and also were written in language 1:
``` request
response, err = client.Query(repository.Intersect(stargazer.Row(14), stargazer.Row(19), language.Row(1)))
if err != nil {
log.Fatal(err)
}
fmt.Println("Both user 14 and 19 starred and were written in language 1: ", response.Result().Row().Columns)
```
``` response
Both user 14 and 19 starred and were written in language 1: [2 362 416 461]
```
Set user 99999 as a stargazer for repository 77777:
``` request
client.Query(stargazer.Set(99999, 77777))
response, err = client.Query(stargazer.Row(99999))
if err != nil {
log.Fatal(err)
}
fmt.Println("Set user 99999 as a stargazer for repository 77777")
```
``` response
Set user 99999 as a stargazer for repository 77777
```
Please note that while user ID 99999 may not be sequential with the other column IDs, it is still a relatively low number.
Don't try to use arbitrary 64-bit integers as column or row IDs in Pilosa - this will lead to problems such as poor performance and out of memory errors.
For more information about go-pilosa, please see our Go client library at [go-pilosa](https://github.com/pilosa/go-pilosa) or checkout the go-pilosa [Data Model and Queries](https://github.com/pilosa/go-pilosa/blob/master/docs/data-model-queries.md) section for more query options.
#### Using Java
Pilosa requires Java 8 or higher and Maven 3 or higher.
##### Create the Environment
Create a project folder:
```
mkdir getting-started && cd getting-started
```
In this folder, we will download two CSV files to provide data to our fields later on. Download the `stargazer.csv` and `language.csv` files here:
```
curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/stargazer.csv
curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/language.csv
```
We will now create the java directory that will contain our `pom.xml` file and create the `pom.xml` file:
```
mkdir startrace && cd startrace
touch pom.xml
```
For this specific project, the `pom.xml` file needs to contain:
```
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.pilosa</groupId>
<artifactId>getting-started</artifactId>
<version>1.0.0</version>
<dependencies>
<dependency>
<groupId>com.pilosa</groupId>
<artifactId>pilosa-client</artifactId>
<version>1.3.1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.6.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<!-- Build an executable JAR -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>3.0.2</version>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>lib/</classpathPrefix>
<mainClass>main.java.StarTrace</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<!-- create an uber JAR -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
```
We will now create the java directory that will contain our `StarTrace.java` file and create the `StarTrace.java` file:
```
mkdir -p src/main/java && cd src/main/java
touch StarTrace.java
```
This file will be used in the following sections.
##### Create the Schema
Before we can import data or run queries, we need to create our schema. You can see the first six dependencies are imported from the java-pilosa library. Create the schema by creating a client which will communicate our schema to Pilosa, creating a schema which will contain our indexes and fields, and syncing with Pilosa. This is all done in the `StarTrace.java` file:
```
package main.java;
import com.pilosa.client.PilosaClient;
import com.pilosa.client.QueryResponse;
import com.pilosa.client.exceptions.PilosaException;
import com.pilosa.client.orm.*;
import com.pilosa.client.csv.FileRecordIterator;
import com.pilosa.client.TimeQuantum;
import java.io.IOException;
import java.text.SimpleDateFormat;
public class StarTrace {
public static void main(String []args) throws IOException {
// Create the Schema
PilosaClient client = PilosaClient.defaultClient();
Schema schema = client.readSchema();
// This is were the index will go later
// This is were the fields will go later
client.syncSchema(schema);
}
}
```
Next, let's create the `repository` index:
```
Index repository = schema.index("repository");
```
The index name must be 230 characters or fewer, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. The same goes for field names.
Let's create the `stargazer` field which has user IDs of stargazers as its rows:
```
FieldOptions stargazerOptions = FieldOptions.builder()
.fieldTime(TimeQuantum.YEAR_MONTH_DAY)
.build();
Field stargazer = repository.field("stargazer", stargazerOptions);
```
Since our data contains time stamps which represent the time users starred repos, we set the field type to `time` using `fieldTime()`. Time quantum is the resolution of the time we want to use, and we set it to `YEAR_MONTH_DAY` for `stargazer`.
Next up is the `language` field, which will contain IDs for programming languages:
```
Field language = repository.field("language");
```
The `language` field is a `set` field, but since the default field type is `set`, we don't need to specify it
Your `StarTrace.java` file should look like:
```
package main.java;
import com.pilosa.client.PilosaClient;
import com.pilosa.client.QueryResponse;
import com.pilosa.client.exceptions.PilosaException;
import com.pilosa.client.orm.*;
import com.pilosa.client.csv.FileRecordIterator;
import com.pilosa.client.TimeQuantum;
import java.io.IOException;
import java.text.SimpleDateFormat;
public class StarTrace {
public static void main(String []args) throws IOException {
// Create the Schema
PilosaClient client = PilosaClient.defaultClient();
Schema schema = client.readSchema();
Index repository = schema.index("repository");
FieldOptions stargazerOptions = FieldOptions.builder()
.fieldTime(TimeQuantum.YEAR_MONTH_DAY)
.build();
Field stargazer = repository.field("stargazer", stargazerOptions);
Field language = repository.field("language");
client.syncSchema(schema);
}
}
```
##### Import Data From CSV Files
Now that we have our index and our fields, we can import the data we downloaded earlier and be on our way to making our own queries.
First, we will load our data into the `stargazer` field:
```
SimpleDateFormat timestampFormat = new SimpleDateFormat("yyyy-MM-dd'T'hh:mm");
FileRecordIterator iterator = FileRecordIterator.fromPath("stargazer.csv", stargazer, timestampFormat);
client.importField(stargazer, iterator);
```
Due to the time aspect of the `stargazer` csv file, we have to specify the time stamp format in the `fromPath` function. We set the variable `timestampFormat` to the format present in the csv file using the function `SimpleDateFormat()` and pass the variable to the `fromPath` function, which will take the csv file name, the field name, and the time stamp format and translate the csv file into a format Pilosa can read.
Next, we will load our data into the `language` field:
```
iterator = FileRecordIterator.fromPath("language.csv", language);
client.importField(language, iterator);
```
Since our `language` data doesn't have a time aspect, the time stamp format doesn't need to be specified.
Note that both the user IDs and the repository IDs were remapped to sequential integers in the data files, they don't correspond to actual Github IDs anymore. You can check out [languages.txt](https://github.com/pilosa/getting-started/blob/master/languages.txt) to see the mapping for languages.
For more information on imports in java-pilosa, please see the java-pilosa [site](https://github.com/pilosa/java-pilosa/blob/master/docs/imports.md).
##### Make Some Queries
Now that we have a working schema, we can query it.
Which repositories did user 14 star:
``` request
QueryResponse response = client.query(stargazer.row(14));
System.out.println("User 14 starred: " + response.getResult().getRow().getColumns());
```
``` response
User 14 starred: [1, 2, 3, 362, 368, 391, 396, 409, 416, 430, 436, 450, 454, 460, 461, 464, 466, 469, 470, 483, 484, 486, 490, 491, 503, 504, 514]
```
What are the top 5 languages in the sample data:
``` request
response = client.query(language.topN(5));
System.out.println("Top Languages: " + response.getResult().getCountItems());
```
``` response
Top Languages: [CountResultItem(id=5, count=119), CountResultItem(id=1, count=50), CountResultItem(id=4, count=48), CountResultItem(id=9, count=31), CountResultItem(id=13, count=25)]
```
Which repositories were starred by user 14 and 19:
``` request
response = client.query(repository.intersect(stargazer.row(14), stargazer.row(19)));
System.out.println("Both user 14 and 19 starred: " + response.getResult().getRow().getColumns());
```
``` response
Both user 14 and 19 starred: [2, 3, 362, 396, 416, 461, 464, 466, 470, 486]
```
Which repositories were starred by user 14 or 19:
``` request
response = client.query(repository.union(stargazer.row(14), stargazer.row(19)));
System.out.println("User 14 or 19 starred: " + response.getResult().getRow().getColumns());
```
``` response
User 14 or 19 starred: [1, 2, 3, 361, 362, 368, 376, 377, 378, 382, 386, 388, 391, 396, 398, 400, 409, 411, 412, 416, 426, 428, 430, 435, 436, 450, 452, 453, 454, 456, 460, 461, 464, 465, 466, 469, 470, 483, 484, 486, 487, 489, 490, 491, 500, 503, 504, 505, 512, 514]
```
Which repositories were starred by user 14 and 19 and also were written in language 1:
``` request
response = client.query(repository.intersect(stargazer.row(14), stargazer.row(19), language.row(1)));
System.out.println("Both user 14 and 19 starred and were written in language 1: " + response.getResult().getRow().getColumns());
```
``` response
Both user 14 and 19 starred and were written in language 1: [2, 362, 416, 461]
```
Set user 99999 as a stargazer for repository 77777:
``` request
client.query(stargazer.set(99999, 77777));
System.out.println("Set user 99999 as a stargazer for repository 77777");
```
``` response
Set user 99999 as a stargazer for repository 77777
```
Please note that while user ID 99999 may not be sequential with the other column IDs, it is still a relatively low number.
Don't try to use arbitrary 64-bit integers as column or row IDs in Pilosa - this will lead to problems such as poor performance and out of memory errors.
For more information about java-pilosa, please see our Java client library at [java-pilosa](https://github.com/pilosa/java-pilosa) or checkout the java-pilosa [Data Model and Queries](https://github.com/pilosa/java-pilosa/blob/master/docs/data-model-queries.md) section for more query options.
#### Python Users
Pilosa requires Python 2.7 or higher or Python 3.4 or higher.
##### Create the Environment
Create a new project folder:
```
mkdir getting-started && cd getting-started
```
In this folder, we will download two CSV files to provide data to our fields later on. Download the `stargazer.csv` and `language.csv` files here:
```
curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/stargazer.csv
curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/language.csv
```
We will also download two text files. One is the `requirements.txt` that will install python-pilosa later on and the other is `languages.txt` which will provide context to the `language` field.
```
curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/python/requirements.txt
curl -O https://raw.githubusercontent.com/pilosa/getting-started/master/language.txt
```
We will now create the python environment:
```
python3 -m venv startrace
```
Next, we activate the python environment we created and install the single dependency, python-pilosa:
```
source startrace/bin/activate
pip install -r requirements.txt
```
We will also create a file called `startrace.py` as follows:
```
touch startrace.py
```
This file will be used in the following sections.
##### Create the Schema
Before we can import data or run queries, we need to create our schema. You can see the dependencies dealing with `pilosa` are from the python-pilosa library. Create the schema by creating a client which will communicate our schema to Pilosa, creating a schema which will contain our indexes and fields, and syncing with Pilosa. This is all done in the `startrace.py` file:
```
from __future__ import print_function
import os
import sys
import time
import pilosa
from pilosa import Client, Index, TimeQuantum
from pilosa.imports import csv_column_reader, csv_row_id_column_id
try:
# Python 2.7 and 3
from io import StringIO
except ImportError:
# Python 2.6 and 2.7
from StringIO import StringIO
# Create the Schema
client = pilosa.Client()
schema = client.schema()
# This is where the index will go later
# This is where the fields will go later
client.sync_schema(schema)
```
Next, let's create the `repository` index:
```
repository = schema.index("repository")
```
The index name must be 230 characters or fewer, start with a letter, and consist only of lowercase alphanumeric characters or `_-`. The same goes for field names.
Let's create the `stargazer` field which has user IDs of stargazers as its rows:
```
stargazer = repository.field("stargazer", time_quantum=pilosa.TimeQuantum.YEAR_MONTH_DAY)
```
Since our data contains time stamps which represent the time users starred repos, we establish the time aspect by using `time_quantum`. Time quantum is the resolution of the time we want to use, and we set it to `YEAR_MONTH_DAY` for `stargazer`.
Next up is the `language` field, which will contain IDs for programming languages:
```
language = repository.field("language")
```
The `language` field is a `set` field, but since the defualt field is `set`, we didn't need to specify any options.
Your `StarTrace.py` file should look like:
```
from __future__ import print_function
import os
import sys
import time
import pilosa
from pilosa import Client, Index, TimeQuantum
from pilosa.imports import csv_column_reader, csv_row_id_column_id
try:
# Python 2.7 and 3
from io import StringIO
except ImportError:
# Python 2.6 and 2.7
from StringIO import StringIO
# Create the Schema
client = pilosa.Client()
schema = client.schema()
repository = schema.index("repository")
stargazer = repository.field("stargazer", time_quantum=pilosa.TimeQuantum.YEAR_MONTH_DAY)
language = repository.field("language")
client.sync_schema(schema)
```
##### Import Data From CSV Files
Now that we have our index and our fields, we can import the data we downloaded earlier and be on our way to making our own queries.
First, we will load our data into the `stargazer` field:
```
time_func = lambda s: int(time.mktime(time.strptime(s, "%Y-%m-%dT%H:%M")))
with open("stargazer.csv") as f:
stargazer_reader = csv_column_reader(f, timefunc=time_func)
client.import_field(stargazer, stargazer_reader)
```
Due to the time aspect of the `stargazer` csv file, we have to specify the time stamp format in the `csv_column_reader` function. We set the variable `time_func` to the format present in the csv file and call it in the `csv_column_reader` function, which will take the csv file and the time stamp format and translate the csv file into a format Pilosa can read
Next, we will load our data into the `language` field:
```
with open("language.csv") as f:
language_reader = csv_column_reader(f, csv_row_id_column_id)
client.import_field(language, language_reader)
```
The `language` is a `set` field, but since the default field type is `set`, we didn't need to specify it.
For more information on imports in python-pilosa, please see the python-pilosa [site](https://github.com/pilosa/python-pilosa/blob/master/docs/imports.md).
Note that both the user IDs and the repository IDs were remapped to sequential integers in the data files, they don't correspond to actual Github IDs anymore. You can check out [languages.txt](https://github.com/pilosa/getting-started/blob/master/languages.txt) to see the mapping for languages.
##### Make Some Queries
Now that we have a working schema, we can query it.
Which repositories did user 14 star:
``` request
response = client.query(stargazer.row(14))
print("User 14 starred: ", response.result.row.columns)
```
``` response
User 14 starred: [1, 2, 3, 362, 368, 391, 396, 409, 416, 430, 436, 450, 454, 460, 461, 464, 466, 469, 470, 483, 484, 486, 490, 491, 503, 504, 514]
```
What are the top 5 languages in the sample data:
``` request
def load_language_names():
with open("languages.txt") as f:
return [line.strip() for line in f]
def print_topn(items):
lines = ["\t{i}. {s[0]}: {s[1]} stars".format(s=s, i=i + 1) for i, s in enumerate(items)]
print("\n".join(lines))
language_names = load_language_names()
top_languages = client.query(language.topn(5)).result.count_items
language_items = [(language_names[item.id], item.count) for item in top_languages]
print("Top languages: ")
print_topn(language_items)
```
``` response
Top languages:
1. Go: 119 stars
2. Shell: 50 stars
3. Makefile: 48 stars
4. HTML: 31 stars
5. JavaScript: 25 stars
```
Which repositories were starred by user 14 and 19:
``` request
repsonse = client.query(repository.intersect(stargazer.row(14), stargazer.row(19)))
print("Both user 14 and 19 starred: ", response.result.row.columns)
```
``` resposne
Both user 14 and 19 starred: [1, 2, 3, 362, 368, 391, 396, 409, 416, 430, 436, 450, 454, 460, 461, 464, 466, 469, 470, 483, 484, 486, 490, 491, 503, 504, 514]
```
Which repositories were starred by user 14 or 19:
``` request
response = client.query(repository.union(stargazer.row(14), stargazer.row(19)))
print("User 14 or 19 starred: ", response.result.row.columns)
```
``` response
User 14 or 19 starred: [1, 2, 3, 361, 362, 368, 376, 377, 378, 382, 386, 388, 391, 396, 398, 400, 409, 411, 412, 416, 426, 428, 430, 435, 436, 450, 452, 453, 454, 456, 460, 461, 464, 465, 466, 469, 470, 483, 484, 486, 487, 489, 490, 491, 500, 503, 504, 505, 512, 514]
```
Which repositories were starred by user 14 and 19 and also were written in language 1:
``` request
response = client.query(repository.intersect(stargazer.row(14), stargazer.row(19), language.row(1)))
print("Both user 14 and 19 starred and were written in language 1: ", response.result.row.columns)
```
``` response
Both user 14 and 19 starred and were written in language 1: [2, 362, 416, 461]
```
Set user 99999 as a stargazer for repository 77777:
``` request
client.query(stargazer.set(99999, 77777))
print("Set user 99999 as a stargazer for repository 77777")
```
``` response
Set user 99999 as a stargazer for repository 77777
```
Please note that while user ID 99999 may not be sequential with the other column IDs, it is still a relatively low number.
Don't try to use arbitrary 64-bit integers as column or row IDs in Pilosa - this will lead to problems such as poor performance and out of memory errors.
For more information about python-pilosa, please see our Python client library at [python-pilosa](https://github.com/pilosa/python-pilosa) or checkout the python-pilosa [Data Model and Queries](https://github.com/pilosa/python-pilosa/blob/master/docs/data-model-queries.md) section for more query options.
### What's Next?
You can jump to [Data Model](../data-model/) for an in-depth look at Pilosa's data model, or [Query Language](../query-language/) for more details about **PQL**, the query language of Pilosa. Check out the [Examples](../examples/) page for example implementations of real world use cases for Pilosa. Ready to get going in your favorite language? Have a peek at our small but expanding set of official [Client Libraries](../client-libraries/).

77
docs/glossary.md
View file

@ -1,77 +0,0 @@
+++
title = "Glossary"
weight = 14
nav = []
+++
## Glossary
<strong id="anti-entropy">[Anti-entropy](../configuration/#anti-entropy-interval):</strong> A periodic process that compares each [shard](#shard) and its [replicas](#replica) across the [cluster](#cluster) to repair inconsistencies.
<strong id="attribute">[Attribute](../data-model/#attribute):</strong> Attributes can be associated to both [rows](#row) and [columns](#column). This metadata is kept separately from the core binary matrix in a [BoltDB](https://github.com/boltdb/bolt) store.
<strong id="bit">[Bit](../data-model/#overview):</strong> Bits are the fundamental unit of data in Pilosa. A bit lives in a [field](#field), at the intersection of a [row](#row) and [column](#column).
<strong id="bitmap">[Bitmap](../data-model/#overview):</strong> The on-disk and in-memory representation of a [row](#row). Implemented with [Roaring](#roaring-bitmap).
<strong id="bsi">[BSI](../data-model/#bsi-range-encoding):</strong> Bit-sliced indexing is the method Pilosa uses to represent multi-bit integers. Integer values are stored in `int` [fields](#field), and can be used for [Range](#range-bsi), [Min](#min), [Max](#max), and [Sum](#sum) queries.
<strong id="cluster">Cluster:</strong> A cluster consists of one or more [nodes](#node) which share a cluster configuration. The cluster also defines how data is [replicated](#replica) and how internode communication is coordinated. Pilosa does not have a leader node, all data is evenly distributed, and any node can respond to queries.
<strong id="column">[Column](../data-model/#column):</strong> Columns are the fundamental horizontal data axis within Pilosa. Columns are global to all [fields](#field) within an [index](#index).
<strong id="fragment">Fragment:</strong> A Fragment is the intersection of a [field](#field) and a [shard](#shard) in an [index](#index).
<strong id="field">[Field](../data-model/#field):</strong> Fields are used to group [rows](#row) into different categories. Row IDs are namespaced by field such that the same row ID in a different field refers to a different row. For [ranked](#topn) fields, rows are kept in sorted order within the field. Fields are one of five types: set, [int](#bsi), bool, time, and mutex. For more information, see [data model](../data-model/) and [Creating fields](../api-reference/#create-field).
<strong id="frame">[Frame](../data-model/#field):</strong> Prior to Pilosa 1.0, fields were known as frames.
<strong id="gossip">[Gossip](https://en.wikipedia.org/wiki/Gossip_protocol):</strong> A protocol used by Pilosa for internal communication.
<strong id="groupby">[GroupBy](../query-language/#group-by):</strong> A [PQL](#pql) query, with functionality similar to a SQL `GROUP BY` clause, that returns the count of the intersection of every combination of rows taking one row each from the specified `Rows` calls. GroupBy can be thought of as a multi-dimensional version of the [TopN](#topn) query.
<strong id="index">[Index](../data-model/#index):</strong> An Index is a top level container in Pilosa, analogous to a database in an RDBMS. Basic queries cannot operate across multiple indexes.
<strong id="jump-consistent-hash">[Jump Consistent Hash](https://arxiv.org/pdf/1406.2294v1.pdf):</strong> A fast, minimal memory, consistent hash algorithm that evenly distributes the workload even when the number of buckets changes.
<strong id="max">[Max](../query-language/#max):</strong> A [PQL](#pql) query that returns the maximum integer value stored in an [integer](#bsi) [field](#field).
<strong id="maxshard">MaxShard:</strong> The total number of [shards](#shard) allocated to handle the current set of [columns](#column). This value is important for all [nodes](#node) to efficiently distribute queries. MaxShard is zero-indexed, so if an index contains six shards, its MaxShard will be 5.
<strong id="min">[Min](../query-language/#min):</strong> A [PQL](#pql) query that returns the minimum integer value stored in an [integer](#bsi) [field](#field).
<strong id="node">Node:</strong> An individual running instance of Pilosa server which belongs to a [cluster](#cluster).
<strong id="partition">Partition:</strong> The [consistent hash](#jump-consistent-hash) maps keys to partitions (or locations on the unit circle), based on a preset maximum number of partitions. Partitions are then evenly mapped to physical [nodes](#node). To add nodes to the [cluster](#cluster), the partitions must be remapped, and data is then associated across the new cluster topology. `DefaultPartitionN` is 256. It can be modified, but only at compile time, and before ingesting any data.
<strong id="pql">[PQL](../query-language/):</strong> Pilosa Query Language.
<strong id="protobuf">[Protobuf](https://developers.google.com/protocol-buffers/):</strong> Protocol Buffers is a binary serialization format which Pilosa uses for internal messages, and can be used by clients as an alternative to JSON.
<strong id="replica">[Replica](../configuration/#cluster-replicas):</strong> A copy of a [fragment](#fragment) on a different [node](#node) than the original. The `cluster.replicas` configuration parameter determines how many replicas of a fragment exist in the cluster. This includes the original, so a value of 1 means no extra copies are made.
<strong id="roaring-bitmap">[Roaring Bitmap](http://roaringbitmap.org):</strong> the compressed bitmap format which Pilosa uses to [implement bitmaps](../architecture/#roaring-bitmap-storage-format), for both storage and logical query operations.
<strong id="row">[Row](../data-model/#row):</strong> Rows are the fundamental vertical data axis within Pilosa. They are namespaced to each [field](#field) within an [index](#index). Represented as a [Bitmap](#bitmap).
<strong id="range">[Row (Ranged)](../query-language/#row-range):</strong> A [PQL](#pql) query that returns bits based on comparison to timestamps, set according to the [time quantum](#time-quantum).
<strong id="range-bsi">[Row (BSI)](../query-language/#row-bsi):</strong> A [PQL](#pql) query that returns bits based on comparison to integers stored in [BSI](#bsi) [fields](#field).
<strong id="rows">[Rows](../query-language/#rows):</strong> A [PQL](#pql) query that returns a list of row IDs in the given field which have at least one bit set. The field argument is mandatory, the others are optional. `Rows` is the primary argument used with the [GroupBy](#groupby) query.
<strong id="slice">[Slice](../data-model/#shard):</strong> Prior to Pilosa 1.0, shards were known as slices.
<strong id="shard">[Shard](../data-model/#shard):</strong> [Columns](#column) are [sharded](https://en.wikipedia.org/wiki/Shard_(database_architecture)) on a preset [width](#shardwidth). Shards are operated on in parallel and are evenly distributed across the cluster via a [consistent hash](#jump-consistent-hash).
<strong id="shardwidth">ShardWidth:</strong> This is the number of [columns](#column) in a [shard](#shard). `ShardWidth` defaults to 2<sup>20</sup> or about one million. It can be modified, but only at compile time, and before ingesting any data.
<strong id="sum">[Sum](../query-language/#sum):</strong> A [PQL](#pql) query that returns the sum of integers stored in an [integer](#bsi) [field](#field).
<strong id="time-quantum">[Time quantum](../data-model/#time-quantum):</strong> Defines the granularity to be used for [ranged Row](#range) queries on time [fields](#field).
<strong id="toml">[TOML](https://github.com/toml-lang/toml):</strong> the language used for Pilosa's [configuration file](../configuration/).
<strong id="topn">[TopN](../query-language/#topn):</strong> A [PQL](#pql) query that returns a list of rows, sorted by the count of [columns](#column) set in the [row](#row), within a specified [field](#field).
<strong id="view">View:</strong> Views separate the different data layouts within a [Field](#field). The primary view is standard, which represents the typical [row](#row)/[column](#column) data. Time based field views are automatically generated for each [time quantum](#time-quantum). Views are internally managed by Pilosa, and never exposed directly via the API. This simplifies the functional interface by separating it from the physical data representation.

382
docs/installation.md
View file

@ -1,382 +0,0 @@
+++
title = "Installation"
weight = 2
nav = [
"Installing on MacOS",
"Installing on Linux",
]
+++
## Installation
Pilosa is currently available for [MacOS](#installing-on-macos) and [Linux](#installing-on-linux).
### Installing on MacOS
There are four ways to install Pilosa on MacOS: Use [Homebrew](https://brew.sh/) (recommended), download the binary, build from source, or use [Docker](#docker).
#### Use Homebrew
1. Update your Homebrew formulas:
```
brew update
```
2. Install Pilosa
```
brew install pilosa
```
3. Make sure Pilosa is installed successfully:
```
pilosa
```
If you see something like:
```
Pilosa is a fast index to turbocharge your database.
This binary contains Pilosa itself, as well as common
tools for administering pilosa, importing/exporting data,
backing up, and more. Complete documentation is available
at https://www.pilosa.com/docs/.
Version: v1.4.0
Build Time: 2018-05-14T22:14:01+0000
Usage:
pilosa [command]
Available Commands:
check Do a consistency check on a pilosa data file.
config Print the current configuration.
export Export data from pilosa.
generate-config Print the default configuration.
help Help about any command
import Bulk load data into pilosa.
inspect Get stats on a pilosa data file.
server Run Pilosa.
Flags:
-c, --config string Configuration file to read from.
-h, --help help for pilosa
Use "pilosa [command] --help" for more information about a command.
```
You're good to go!
#### Download the Binary
1. Download the latest release:
```
curl -L -O https://github.com/pilosa/pilosa/releases/download/v1.4.0/pilosa-v1.4.0-darwin-amd64.tar.gz
```
Other releases can be downloaded from our Releases page on Github.
2. Extract the binary:
```
tar xfz pilosa-v1.4.0-darwin-amd64.tar.gz
```
3. Move the binary into your PATH so you can run `pilosa` from any shell:
```
cp -i pilosa-v1.4.0-darwin-amd64/pilosa /usr/local/bin
```
4. Make sure Pilosa is installed successfully:
```
pilosa
```
If you see something like:
```
Pilosa is a fast index to turbocharge your database.
This binary contains Pilosa itself, as well as common
tools for administering pilosa, importing/exporting data,
backing up, and more. Complete documentation is available
at https://www.pilosa.com/docs/.
Version: v1.4.0
Build Time: 2018-05-14T22:14:01+0000
Usage:
pilosa [command]
Available Commands:
check Do a consistency check on a pilosa data file.
config Print the current configuration.
export Export data from pilosa.
generate-config Print the default configuration.
help Help about any command
import Bulk load data into pilosa.
inspect Get stats on a pilosa data file.
server Run Pilosa.
Flags:
-c, --config string Configuration file to read from.
-h, --help help for pilosa
Use "pilosa [command] --help" for more information about a command.
```
You're good to go!
#### Build from Source
<div class="note">
<p>For advanced instructions for building from source, view our <a href="https://github.com/pilosa/pilosa/blob/master/CONTRIBUTING.md">Contributor's Guide.</a></p>
</div>
1. Install the prerequisites:
* [Go](https://golang.org/doc/install). Be sure to set the `$GOPATH` and `$PATH` environment variables as described [here](https://golang.org/doc/code.html#GOPATH).
* [Git](https://git-scm.com/)
2. Clone the repo:
```
mkdir -p ${GOPATH}/src/github.com/pilosa && cd $_
git clone https://github.com/pilosa/pilosa.git
```
3. Build the Pilosa repo:
```
cd $GOPATH/src/github.com/pilosa/pilosa
make install-build-deps
make install
```
4. Make sure Pilosa is installed successfully:
```
pilosa
```
If you see something like:
```
Pilosa is a fast index to turbocharge your database.
This binary contains Pilosa itself, as well as common
tools for administering pilosa, importing/exporting data,
backing up, and more. Complete documentation is available
at https://www.pilosa.com/docs/.
Version: v1.4.0
Build Time: 2018-05-14T22:14:01+0000
Usage:
pilosa [command]
Available Commands:
check Do a consistency check on a pilosa data file.
config Print the current configuration.
export Export data from pilosa.
generate-config Print the default configuration.
help Help about any command
import Bulk load data into pilosa.
inspect Get stats on a pilosa data file.
server Run Pilosa.
Flags:
-c, --config string Configuration file to read from.
-h, --help help for pilosa
Use "pilosa [command] --help" for more information about a command.
```
You're good to go!
#### What's next?
Head over to the [Getting Started](../getting-started/) guide to create your first Pilosa index.
### Installing on Linux
There are three ways to install Pilosa on Linux: download the binary (recommended), build from source, or use [Docker](#docker).
#### Download the Binary
1. To install the latest version of Pilosa, download the latest release:
```
curl -L -O https://github.com/pilosa/pilosa/releases/download/v1.4.0/pilosa-v1.4.0-linux-amd64.tar.gz
```
Note: This assumes you are using an `amd64` compatible architecture. Other releases can be downloaded from our Releases page on Github.
2. Extract the binary:
```
tar xfz pilosa-v1.4.0-linux-amd64.tar.gz
```
3. Move the binary into your PATH so you can run `pilosa` from any shell:
```
cp -i pilosa-v1.4.0-linux-amd64/pilosa /usr/local/bin
```
4. Make sure Pilosa is installed successfully:
```
pilosa
```
If you see something like:
```
Pilosa is a fast index to turbocharge your database.
This binary contains Pilosa itself, as well as common
tools for administering pilosa, importing/exporting data,
backing up, and more. Complete documentation is available
at https://www.pilosa.com/docs/.
Version: v1.4.0
Build Time: 2018-05-14T22:14:01+0000
Usage:
pilosa [command]
Available Commands:
check Do a consistency check on a pilosa data file.
config Print the current configuration.
export Export data from pilosa.
generate-config Print the default configuration.
help Help about any command
import Bulk load data into pilosa.
inspect Get stats on a pilosa data file.
server Run Pilosa.
Flags:
-c, --config string Configuration file to read from.
-h, --help help for pilosa
Use "pilosa [command] --help" for more information about a command.
```
You're good to go!
#### Build from Source
<div class="note">
<p>For advanced instructions for building from source, view our <a href="https://github.com/pilosa/pilosa/blob/master/CONTRIBUTING.md">Contributor's Guide.</a></p>
</div>
1. Install the prerequisites:
* [Go](https://golang.org/doc/install). Be sure to set the `$GOPATH` and `$PATH` environment variables as described [here](https://golang.org/doc/code.html#GOPATH).
* [Git](https://git-scm.com/)
2. Clone the repo:
```
mkdir -p ${GOPATH}/src/github.com/pilosa && cd $_
git clone https://github.com/pilosa/pilosa.git
```
3. Build the Pilosa repo:
```
cd $GOPATH/src/github.com/pilosa/pilosa
make install-build-deps
make install
```
4. Make sure Pilosa is installed successfully:
```
pilosa
```
If you see something like:
```
Pilosa is a fast index to turbocharge your database.
This binary contains Pilosa itself, as well as common
tools for administering pilosa, importing/exporting data,
backing up, and more. Complete documentation is available
at https://www.pilosa.com/docs/.
Version: v1.4.0
Build Time: 2018-05-14T22:14:01+0000
Usage:
pilosa [command]
Available Commands:
check Do a consistency check on a pilosa data file.
config Print the current configuration.
export Export data from pilosa.
generate-config Print the default configuration.
help Help about any command
import Bulk load data into pilosa.
inspect Get stats on a pilosa data file.
server Run Pilosa.
Flags:
-c, --config string Configuration file to read from.
-h, --help help for pilosa
Use "pilosa [command] --help" for more information about a command.
```
You're good to go!
#### What's next?
Head over to the [Getting Started](../getting-started/) guide to create your first Pilosa index.
### Windows
Windows is currently not supported as a target deployment platform for Pilosa, but developing and running Pilosa is made possible by Docker. See the [Docker](#docker) documentation for using Docker for Windows and Docker Toolbox.
Windows Subsystem for Linux is currently not supported.
### Docker
1. Install Docker for your platform. On Linux, Docker is available via your package manager. On MacOS, you can use Docker for Mac or Docker Toolbox. On Windows, you can use Docker for Windows or Docker Toolbox.
2. **This step is necessary only if you are using Docker Toolbox**, otherwise skip to step 3:
a. Start the Docker support using `docker-machine start` in a terminal. The environment variables of the terminal should be updated accordingly, run `docker-machine env` to display the necessary commands.
b. Set up port forwarding in the VirtualBox GUI or on the command line. Guest port should be 10101. For the host port, 10101 is recommended. If the `VBoxManage` command is in your `PATH`, you can use the following command (assuming you use the default VM):
```
VBoxManage modifyvm "default" --natpf1 "pilosa,tcp,,10101,,10101"
```
3. Confirm that the Docker daemon is running in the background:
```
docker version
```
If you are getting a "command not found" or similar, check that `docker` command is in your path. If you don't see the server listed, start the Docker application.
4. Pull the official Pilosa image from Docker Hub:
```
docker pull pilosa/pilosa:latest
```
5. Make sure Pilosa is installed successfully, and make it accessible:
```
docker run -d --rm --name pilosa -p 10101:10101 pilosa/pilosa:latest server --bind 0.0.0.0:10101
```
6. Check that it is accessible from outside the container.
Run the following in a separate terminal:
```
curl localhost:10101/schema
```
If that returns `{"indexes":null}` or similar, then Pilosa is accessible from outside the container. Otherwise check that you have correctly typed `-p 10101:10101` when running the Pilosa container and the port mappings in VirtualBox is correct (Docker Toolbox only).
7. When you want to terminate the Pilosa container, you can run the following:
```
docker stop pilosa
```
#### What's next?
Head over to the [Getting Started](../getting-started/) guide to create your first Pilosa index.

19
docs/introduction.md
View file

@ -1,19 +0,0 @@
+++
title = "Introduction"
weight = 1
nav = []
+++
## Introduction
Pilosa is an open source, distributed index.
[//]: # (TODO insert a graphic here?)
It is designed primarily for speed and horizontal scalability. If you have data with billions of objects that can have millions of possible attributes, and you want to explore those relationships, Pilosa can help you.
"What attributes are the most common?", "Which objects have these specific attributes?", "What groups of attributes often appear together?" Pilosa is designed to answer these types of queries in real time, suitable for use with high rate data streams, or to power a user interface.
Once you have Pilosa [installed](../installation/), the [getting started](../getting-started/) guide will show you the basics of interacting with Pilosa and give you some pointers for deeper exploration.

74
docs/pdk.md
View file

@ -1,74 +0,0 @@
+++
title = "PDK"
weight = 11
nav = [
"Examples and Executables",
"Library",
]
+++
## PDK
The [Pilosa Dev Kit](https://github.com/pilosa/pdk) contains executables, examples, and Go libraries to help you use Pilosa effectively.
### Examples and Executables
Running `pdk -h` will give the most up to date list of all the tools and examples that PDK provides. We'll cover a few of the more important ones here.
#### Kafka
`pdk kafka` reads either JSON or Avro encoded records from Kafka (using the
Confluent Schema Registry in the case of Avro), and indexes them in Pilosa. Each
record from Kafka is assigned a Pilosa column, and each value in a record is
assigned a row or field. Pilosa field names are built from the "path" through
the record to arrive at that field. For example:
```json
{
"name": "jill",
"favorite_foods": ["corn chips", "chipotle dip"],
"location": {
"city": "Austin",
"state": "Texas",
"latitude": 3754,
"longitude": 4526
},
"active": true,
"age": 27
}
```
This JSON object would result in the following Pilosa schema:
| Field | Example Value | Type | Cache Size |
|----------------|---------------|--------|------------|
| name | "jill" | ranked | 100000 |
| favorite_foods | "corn chips" | ranked | 100000 |
| default | | ranked | 100000 |
| age | 27 | int | |
| location | | ranked | 1000 |
| latitude | 3754 | int | |
| longitude | 4526 | int | |
| location-city | "Austin" | ranked | 100000 |
| location-state | "Texas" | ranked | 100000 |
All set fields are created as ranked fields by default, with the cache size
listed above. Integer fields are created with a minimum size of zero and a
fixed maximum of 2147483647. Field names are a dash-separated concatenation of
all key values in the path - you can see this with fields like location-city.
Most of the options to `pdk kafka` are self-explanatory (kafka hosts, pilosa hosts,
kafka topics, kafka group, etc.), but there are a few options that give some
control over the way data is indexed, and ingestion performance.
* `--batch-size`: The batch size controls how many set bits or values are batched up to be imported *per field*. So for fields that have one value per record, you have to wait for `batch-size` records to come through before you'll see the data indexed in Pilosa. Fields like `favorite_foods` which can have multiple values could be indexed sooner.
* `--framer.collapse`: This is a list of strings which will be removed from the field names created by dash-concatentating all names in the JSON path to a value. E.G. if "location" were listed in `framer.collapse`, then there would be fields named "city" and "state" rather than "location-city" and "location-state".
* `--framer.ignore`: This allows you to skip indexing on any path containing these strings. If you have a field like email address or some other unique ID, you might not want to index it.
* `--subject-path`: If nothing is passed for this option, then each record will be assigned a unique sequential column ID. If `subject-path` is specified, then the value at this path in the record will be mapped to a column ID. If the same value appears in another record, the same column ID will be used.
* `--proxy`: The PDK ingests data, but also keeps a mapping for string values to row IDs, and from subjects to column ids. Because of this, querying Pilosa directly may not be useful, since it only returns integer row and column ids. The PDK will start a proxy server which intercepts requests to Pilosa using strings for row and column ids, and translates them to the integers that Pilosa understands. It will also translate responses so that (e.g.) a TopN query will return `{"results":[[{"Key":"chipotle dip","Count":1},{"Key":"corn chips","Count":1}]]}`. By default, the mapping is stored in an embedded leveldb.
For more information on running `pdk kafka` and how Pilosa interfaces with Kafka, please see the [kafka directory](https://github.com/pilosa/pdk/tree/master/kafka) in the pdk repository.
### Library
For now, the [Godocs](https://godoc.org/github.com/pilosa/pdk) have the most up to date library documentation.

1060
docs/query-language.md
View file

File diff suppressed because it is too large Load diff

779
docs/tutorials.md
View file

@ -1,779 +0,0 @@
+++
title = "Tutorials"
weight = 4
nav = [
"Setting Up a Secure Cluster",
"Setting Up a Docker Cluster",
"Using Integer Field Values",
"Storing Row and Column Attributes",
]
+++
## Tutorials
<div class="note">
<!-- this is html because there is a problem putting a list inside a shortcode -->
Some of our tutorials work better as standalone repos, since you can <code>git clone</code> the instructions, code, and data all at once. Officially supported tutorials are listed here.<br />
<br />
<ul>
<li><a href="https://github.com/pilosa/cosmosa">Run Pilosa with Microsoft's Azure Cosmos DB</a></li>
</ul>
</div>
### Setting Up a Secure Cluster
#### Introduction
Pilosa supports encrypting all communication with nodes in a cluster using TLS, including [Mutual TLS Authentication](https://en.wikipedia.org/wiki/Mutual_authentication). In this tutorial, we will be setting up a three node Pilosa cluster running on the same computer. The same steps can be used for a multi-computer cluster but that requires setting up firewalls and other platform-specific configuration which is beyond the scope of this tutorial.
This tutorial assumes that you are using a UNIX-like system, such as Linux or MacOS. [Windows Subsystem for Linux (WSL)](https://msdn.microsoft.com/en-us/commandline/wsl/about) works equally well on Windows 10 systems.
#### Installing Pilosa and Creating the Directory Structure
If you haven't already done so, install Pilosa server on your computer. For Linux and WSL (Windows Subsystem for Linux) use the [Installing on Linux](../installation/#installing-on-linux) instructions. For MacOS use the [Installing on MacOS](../installation/#installing-on-macos). We do not support precompiled releases for other platforms, but you can always compile it yourself from source. See [Build from Source](../installation/#build-from-source).
After installing Pilosa, you may have to add it to your `$PATH`. Check that you can run Pilosa from the command line:
``` request
pilosa --help
```
``` response
Pilosa is a fast index to turbocharge your database.
This binary contains Pilosa itself, as well as common
tools for administering pilosa, importing/exporting data,
backing up, and more. Complete documentation is available
at https://www.pilosa.com/docs/.
Pilosa v1.4.0
Build Time: 2019-09-23T14:33:07+0000
Usage:
pilosa [command]
Available Commands:
check Do a consistency check on a pilosa data file.
config Print the current configuration.
export Export data from pilosa.
generate-config Print the default configuration.
help Help about any command
holder Load Pilosa.
import Bulk load data into pilosa.
inspect Get stats on a pilosa data file.
server Run Pilosa.
Flags:
-c, --config string Configuration file to read from.
-h, --help help for pilosa
Use "pilosa [command] --help" for more information about a command.
```
First, create a directory in which to put all of the files for this tutorial. Then switch to that directory:
```
mkdir $HOME/pilosa-tls-tutorial && cd $_
```
#### Creating the TLS Certificate and Gossip Key
Securing a Pilosa cluster consists of securing the communication between nodes using TLS and Gossip encryption.
The first step is acquiring the necessary TLS certificates. Operating your own public key infrastructure (PKI) is outside of the scope of this tutorial, but it is easy to get started with [certstrap](https://github.com/square/certstrap) for testing/development purposes. For production, you can use OpenSSL or any other software that provides PKI using X.509 certificates, including [Hashicorp Vault](https://learn.hashicorp.com/vault/secrets-management/sm-pki-engine). It is not recommended to use certstrap in production.
First, create a certificate authority (CA):
```
$ certstrap init --common-name ca
Created out/ca.key
Created out/ca.crt
Created out/ca.crl
```
The command above creates three files in the `out/` directory:
* `ca.key` is the CA private key file which must be kept as secret.
* `ca.crt` is the CA TLS certificate.
* `ca.crl` is the Certificate Revocation List (CRL).
Next, create and sign a wildcard certificate for pilosa:
```
$ certstrap request-cert --cn "*.pilosa.local"
Created out/*.pilosa.local.key
Created out/*.pilosa.local.csr
$ certstrap sign "*.pilosa.local" --CA ca
Created out/*.pilosa.local.crt from out/*.pilosa.local.csr signed by out/ca.key
```
The commands above create three files in the `out/` directory:
* `*.pilosa.local.key` is the private key file which must be kept as secret.
* `*.pilosa.local.csr` is the certificate signing request (CSR).
* `*.pilosa.local.crt` is the signed TLS certificate.
You can also create a separate client certificate signed by the same CA to test mutual TLS using curl:
```
$ certstrap request-cert --cn "curl"
Created out/curl.key
Created out/curl.csr
$ certstrap sign "curl" --CA ca
Created out/curl.crt from out/curl.csr signed by out/ca.key
```
Having created the TLS certificates, we can now create the gossip encryption key. The gossip encryption key file must be exactly 16, 24, or 32 bytes to select one of AES-128, AES-192, or AES-256 encryption. Reading random bytes from cryptographically secure `/dev/random` serves our purpose very well:
```
head -c 32 /dev/random > pilosa.local.gossip32
```
We now have a file called `pilosa.local.gossip32` in the current directory which contains 32 random bytes.
#### Creating the Configuration Files
Pilosa supports passing configuration items using command line options, environment variables, or a configuration file. For this tutorial, we will use three configuration files; one configuration file for each of our three nodes.
One of the nodes in the cluster must be chosen as the *coordinator*. We choose the first node as the coordinator in this tutorial. The coordinator is only important during cluster resizing operations, and otherwise acts like any other node in the cluster. In the future, the coordinator will be chosen transparently by distributed consensus, and this option will be deprecated.
Create `node1.config.toml` in the project directory and paste the following in it:
```toml
# node1.config.toml
data-dir = "node1_data"
bind = "https://01.pilosa.local:10501"
[cluster]
coordinator = true
[tls]
ca-certificate = "out/ca.crt"
certificate = "out/*.pilosa.local.crt"
key = "out/*.pilosa.local.key"
enable-client-verification = true
[gossip]
seeds = ["01.pilosa.local:15000"]
port = 15000
key = "pilosa.local.gossip32"
```
Create `node2.config.toml` in the project directory and paste the following in it:
```toml
# node2.config.toml
data-dir = "node2_data"
bind = "https://02.pilosa.local:10502"
[tls]
ca-certificate = "out/ca.crt"
certificate = "out/*.pilosa.local.crt"
key = "out/*.pilosa.local.key"
enable-client-verification = true
[gossip]
seeds = ["01.pilosa.local:15000"]
port = 16000
key = "pilosa.local.gossip32"
```
Create `node3.config.toml` in the project directory and paste the following in it:
```toml
# node3.config.toml
data-dir = "node3_data"
bind = "https://03.pilosa.local:10503"
[tls]
ca-certificate = "out/ca.crt"
certificate = "out/*.pilosa.local.crt"
key = "out/*.pilosa.local.key"
enable-client-verification = true
[gossip]
seeds = ["01.pilosa.local:15000"]
port = 17000
key = "pilosa.local.gossip32"
```
Here is some explanation of the configuration items:
* `data-dir` points to the directory where the Pilosa server writes its data. If it doesn't exist, the server will create it.
* `bind` is the address to which the server listens for incoming requests. The address is composed of three parts: scheme, host, and port. The default scheme is `http` so we explicitly specify `https` to use the HTTPS protocol for communication between nodes.
* `[cluster]` section contains the settings for a cluster. We set `coordinator = true` for only the first node to choose that as the coordinator node. See [Cluster Configuration](../configuration/#cluster-coordinator) for other settings.
* `[tls]` section contains the TLS settings, including the path to the TLS certificate and the corresponding key. The `ca-certificate` setting is optional and will default to your system CAs. You may also disable server-to-server verification by setting `skip-verify` to `true`, which we don't recommend for production.
* `[gossip]` section contains settings for the gossip protocol. `seeds` contains the list of nodes from which to seed cluster membership. There must be at least one gossip seed. The `port` setting is the gossip listen address for the node. If all nodes of the cluster are running on the same computer, the gossip listen address should be different for each node. Otherwise, it can be set to the same value. Finally, the `key` points to the gossip encryption key we created earlier.
#### Final Touches Before Running the Cluster
Before running the cluster, let's make sure that `01.pilosa.local`, `02.pilosa.local` and `03.pilosa.local` resolve to an IP address. If you are running the cluster on your computer, it is adequate to add them to your `/etc/hosts`. Below is one of the many ways of doing that (mind the `>>`):
```
sudo sh -c 'printf "\n127.0.0.1 01.pilosa.local 02.pilosa.local 03.pilosa.local\n" >> /etc/hosts'
```
Ensure we can access the hosts in the cluster:
```
ping -c 1 01.pilosa.local
ping -c 1 02.pilosa.local
ping -c 1 03.pilosa.local
```
If any of the commands above return `ping: unknown host`, make sure your `/etc/hosts` contains the failed hostname.
#### Running the Cluster
Let's open three terminal windows and run each node in its own window. This will enable us to better observe what's happening on each node.
Switch to the first terminal window, change to the project directory and start the first node:
```
cd $HOME/pilosa-tls-tutorial
pilosa server -c node1.config.toml
```
Switch to the second terminal window, change to the project directory and start the second node:
```
cd $HOME/pilosa-tls-tutorial
pilosa server -c node2.config.toml
```
Switch to the third terminal window, change to the project directory and start the third node:
```
cd $HOME/pilosa-tls-tutorial
pilosa server -c node3.config.toml
```
Let's ensure that all three Pilosa servers are running and they are connected:
``` request
curl --cacert out/ca.crt --cert out/curl.crt --key out/curl.key \
https://01.pilosa.local:10501/status
```
``` response
{"state":"NORMAL","nodes":[{"id":"98ebd177-c082-4c54-8d48-7e7c75857b52","uri":{"scheme":"https","host":"02.pilosa.local","port":10502},"isCoordinator":false},{"id":"a33dc0d6-c35f-4559-984a-e582bf032a21","uri":{"scheme":"https","host":"03.pilosa.local","port":10503},"isCoordinator":false},{"id":"e24ac014-ee2f-4cb0-b565-74df6c551f0a","uri":{"scheme":"https","host":"01.pilosa.local","port":10501},"isCoordinator":true}]}
```
The `-k` flag is used to tell curl that it shouldn't bother checking the certificate the server provides, and the `--ipv4` flag avoids an issue on MacOS where the curl request takes a long time if the address resolves to `127.0.0.1`. You can leave it out on Linux and WSL.
If everything is set up correctly, the cluster state should be `NORMAL`.
#### Running Queries
Having confirmed that our cluster is running normally, let's perform a few queries. First, we need to create an index and a field:
``` request
curl --cacert out/ca.crt --cert out/curl.crt --key out/curl.key \
https://01.pilosa.local:10501/index/sample-index \
-X POST
```
``` response
{"success":true}
```
This will create index `sample-index` with default options. Let's create the field now:
``` request
curl --cacert out/ca.crt --cert out/curl.crt --key out/curl.key \
https://01.pilosa.local:10501/index/sample-index/field/sample-field \
-X POST
```
``` response
{"success":true}
```
We just created field `sample-field` with default options.
Let's run a `Set` query:
``` request
curl --cacert out/ca.crt --cert out/curl.crt --key out/curl.key \
https://01.pilosa.local:10501/index/sample-index/query \
-X POST \
-d 'Set(100, sample-field=1)'
```
``` response
{"results":[true]}
```
Confirm that the value was indeed set:
``` request
curl --cacert out/ca.crt --cert out/curl.crt --key out/curl.key \
https://01.pilosa.local:10501/index/sample-index/query \
-X POST \
-d 'Row(sample-field=1)'
```
``` response
{"results":[{"attrs":{},"columns":[100]}]}
```
The same response should be returned when querying other nodes in the cluster:
``` request
curl --cacert out/ca.crt --cert out/curl.crt --key out/curl.key \
https://02.pilosa.local:10502/index/sample-index/query \
-X POST \
-d 'Row(sample-field=1)'
```
``` response
{"results":[{"attrs":{},"columns":[100]}]}
```
#### What's Next?
Check out our [Administration Guide](https://www.pilosa.com/docs/latest/administration/) to learn more about making the most of your Pilosa cluster and [Configuration Documentation](https://www.pilosa.com/docs/latest/configuration/) to see the available options to configure Pilosa.
### Setting Up a Docker Cluster
In this tutorial, we will be setting up a 2-node Pilosa cluster using Docker containers.
#### Running a Docker Cluster on a Single Server
The instructions below require Docker 1.13 or better.
Let's first be sure that the Pilosa image is up to date:
```
docker pull pilosa/pilosa:latest
```
Then, create a virtual network to attach our containers. We are going to name our network `pilosanet`:
```
docker network create pilosanet
```
Let's run the first Pilosa node and attach it to that virtual network. We set the first node as the cluster coordinator and use its address as the gossip seed. And also set the server address to `pilosa1`:
```
docker run -it --rm --name pilosa1 -p 10101:10101 --network=pilosanet pilosa/pilosa:latest server --bind pilosa1 --cluster.coordinator=true --gossip.seeds=pilosa1:14000
```
Let's run the second Pilosa node and attach it to the virtual network as well. Note that we set the address of the gossip seed to the address of the first node:
```
docker run -it --rm --name pilosa2 -p 10102:10101 --network=pilosanet pilosa/pilosa:latest server --bind pilosa2 --gossip.seeds=pilosa1:14000
```
Let's test that the nodes in the cluster connected with each other:
``` request
curl localhost:10101/status
```
``` response
{"state":"NORMAL","nodes":[{"id":"2e8332d0-1fee-44dd-a359-e0d6ecbcefc1","uri":{"scheme":"http","host":"pilosa1","port":10101},"isCoordinator":true},{"id":"8c0dbcdc-9503-4265-8ad2-ba85a4bb10fa","uri":{"scheme":"http","host":"pilosa2","port":10101},"isCoordinator":false}],"localID":"2e8332d0-1fee-44dd-a359-e0d6ecbcefc1"}
```
And similarly for the second node:
``` request
curl localhost:10102/status
```
``` response
{"state":"NORMAL","nodes":[{"id":"2e8332d0-1fee-44dd-a359-e0d6ecbcefc1","uri":{"scheme":"http","host":"pilosa1","port":10101},"isCoordinator":true},{"id":"8c0dbcdc-9503-4265-8ad2-ba85a4bb10fa","uri":{"scheme":"http","host":"pilosa2","port":10101},"isCoordinator":false}],"localID":"2e8332d0-1fee-44dd-a359-e0d6ecbcefc1"}
```
The corresponding [Docker Compose](https://docs.docker.com/compose/) file is below:
```yaml
version: '2'
services:
pilosa1:
image: pilosa/pilosa:latest
ports:
- "10101:10101"
environment:
- PILOSA_CLUSTER_COORDINATOR=true
- PILOSA_GOSSIP_SEEDS=pilosa1:14000
networks:
- pilosanet
entrypoint:
- /pilosa
- server
- --bind
- "pilosa1:10101"
pilosa2:
image: pilosa/pilosa:latest
ports:
- "10102:10101"
environment:
- PILOSA_GOSSIP_SEEDS=pilosa1:14000
networks:
- pilosanet
entrypoint:
- /pilosa
- server
- --bind
- "pilosa2:10101"
networks:
pilosanet:
```
#### Running a Docker Swarm
It is very easy to run a Pilosa Cluster on different servers using [Docker Swarm mode](https://docs.docker.com/engine/swarm/). All we have to do is create an overlay network instead of a bridge network.
The instructions in this section require Docker 17.06 or newer. Although it is possible to run a Docker swarm on MacOS or Windows, it is easiest to run it on Linux. The following instructions assume you are running on Linux.
We are going to use two servers: the manager node runs in the first server and a worker node in the second server.
Docker nodes require some ports to be accesible from the outside. Before proceeding, make sure the following ports are open on all nodes: TCP/2377, TCP/7946, UDP/7946, UDP/4789.
Let's initialize the swarm first. Run the following on the manager:
```
docker swarm init --advertise-addr=IP-ADDRESS
```
Virtual machines running on the cloud usually have at least two network interfaces: the external interface and the internal interface. Use the IP of the external interface.
The output of the command above should be similar to:
```
To add a manager to this swarm, run the following command:
docker swarm join --token SOME-TOKEN MANAGER-IP-ADDRESS:2377
```
Let's make the worker node join the manager. Copy/paste the command above in a shell on the worker, replacing the token and IP address with the correct values. You may neeed to add `--advertise-addr=WORKER-EXTERNAL-IP-ADDRESS` parameter if the worker has more than one network interface:
```
docker swarm join --token SOME-TOKEN MANAGER-IP-ADDRESS:2377
```
Run the following on the manager to check that the worker joined to the swarm:
```
docker node ls
```
Which should output:
ID|HOSTNAME|STATUS|AVAILABILITY|MANAGER STATUS|ENGINE VERSION
---|--------|------|------------|--------------|-------------
MANAGER-ID *|swarm1|Ready|Active|Leader|18.05.0-ce|
WORKER-ID|swarm2|Ready|Active||18.05.0-ce|
If you have created the `pilosanet` network before, delete it before carrying on, otherwise skip to the next step:
```
docker network rm pilosanet
```
Let's create the `pilosanet` network, but with `overlay` type this time. We should also make this network attachable in order to be able to attach containers to it. Run the following on the manager:
```
docker network create -d overlay pilosanet --attachable
```
We can now create the Pilosa containers. Let's start the coordinator node first. Run the following on one of the servers:
```
docker run -it --rm --name pilosa1 --network=pilosanet pilosa/pilosa:latest server --bind pilosa1 --cluster.coordinator=true --gossip.seeds=pilosa1:14000
```
And the following on the other server:
```
docker run -it --rm --name pilosa2 --network=pilosanet pilosa/pilosa:latest server --bind pilosa2 --gossip.seeds=pilosa1:14000
```
These were the same commands we used in the previous section except the port mapping! Let's run another container on the same virtual network to read the status from the coordinator:
``` request
docker run -it --rm --network=pilosanet --name shell alpine wget -q -O- pilosa1:10101/status
```
``` response
{"state":"NORMAL","nodes":[{"id":"3e3b0abd-1945-441a-a01f-5a28272972f5","uri":{"scheme":"http","host":"pilosa1","port":10101},"isCoordinator":true},{"id":"71ed27cc-9443-4f41-88fb-1c22f92bf695","uri":{"scheme":"http","host":"pilosa2","port":10101},"isCoordinator":false}],"localID":"3e3b0abd-1945-441a-a01f-5a28272972f5"}
```
You can add additional worker nodes to both the swarm and the Pilosa cluster using the steps above.
#### What's Next?
Check out our [Administration Guide](https://www.pilosa.com/docs/latest/administration/) to learn more about making the most of your Pilosa cluster and [Configuration Documentation](https://www.pilosa.com/docs/latest/configuration/) to see the available options to configure Pilosa.
Refer to the [Docker documentation](https://docs.docker.com) to see your options about running Docker containers. The [Networking with overlay networks](https://docs.docker.com/network/network-tutorial-overlay/) is a detailed overview of the Docket swarm mode and overlay networks.
### Using Integer Field Values
#### Introduction
Pilosa can store integer values associated to the columns in an index, and those values are used to support `Row`, `Min`, `Max`, and `Sum` queries. In this tutorial we will show how to set up integer fields, populate those fields with data, and query the fields. The example index we're going to create will represent fictional patients at a medical facility and various bits of information about those patients.
First, create an index called `patients`:
``` request
curl localhost:10101/index/patients \
-X POST
```
``` response
{"success":true}
```
In addition to storing rows of bits, a field can also store integer values. The next steps creates three fields (`age`, `weight`, `tcells`) in the `patients` index.
``` request
curl localhost:10101/index/patients/field/age \
-X POST \
-d '{"options":{"type": "int", "min": 0, "max": 120}}'
```
``` response
{"success":true}
```
``` request
curl localhost:10101/index/patients/field/weight \
-X POST \
-d '{"options":{"type": "int", "min": 0, "max": 500}}'
```
``` response
{"success":true}
```
``` request
curl localhost:10101/index/patients/field/tcells \
-X POST \
-d '{"options":{"type": "int", "min": 0, "max": 2000}}'
```
``` response
{"success":true}
```
Next, let's populate our fields with data. There are two ways to get data into fields: use the `Set()` PQL function to set fields individually, or use the `pilosa import` command to import many values at once. First, let's set some field data using PQL.
The following queries set the age, weight, and t-cell count for the patient with ID `1` in our system:
``` request
curl localhost:10101/index/patients/query \
-X POST \
-d 'Set(1, age=34)'
```
``` response
{"results":[true]}
```
``` request
curl localhost:10101/index/patients/query \
-X POST \
-d 'Set(1, weight=128)'
```
``` response
{"results":[true]}
```
``` request
curl localhost:10101/index/patients/query \
-X POST \
-d 'Set(1, tcells=1145)'
```
``` response
{"results":[true]}
```
In the case where we need to load a lot of data at once, we can use the `pilosa import` command. This method lets us import data into Pilosa from a CSV file.
Assuming we have a file called `ages.csv` that is structured like this:
```
1,34
2,57
3,19
4,40
5,32
6,71
7,28
8,33
9,63
```
where the first column of the CSV represents the patient `ID` and the second column represents the patient's `age`, then we can import the data into our `age` field by running this command:
```
pilosa import -i patients --field age ages.csv
```
Now that we have some data in our index, let's run a few queries to demonstrate how to use that data.
In order to find all patients over the age of 40, then simply run a `Row` query against the `age` field.
``` request
curl localhost:10101/index/patients/query \
-X POST \
-d 'Row(age > 40)'
```
``` response
{"results":[{"attrs":{},"columns":[2,6,9]}]}
```
You can find a list of supported range operators in the [Row (BSI) Query](../query-language/#row-bsi) documentation.
To find the average age of all patients, run a `Sum` query:
``` request
curl localhost:10101/index/patients/query \
-X POST \
-d 'Sum(field="age")'
```
``` response
{"results":[{"value":377,"count":9}]}
```
The results you get from the `Sum` query contain the sum of all values as well as the `count` of columns with a value. To get the average you can just divide `value` by `count`.
You can also provide a filter to the `Sum()` function to find the average age of all patients over 40.
``` request
curl localhost:10101/index/patients/query \
-X POST \
-d 'Sum(Row(age > 40), field="age")'
```
``` response
{"results":[{"value":191,"count":3}]}
```
Notice in this case that the count is only `3` because of the `age > 40` filter applied to the query.
To find the minimum age of all patients, run a `Min` query:
``` request
curl localhost:10101/index/patients/query \
-X POST \
-d 'Min(field="age")'
```
``` response
{"results":[{"value":19,"count":1}]}
```
The results you get from the `Min` query contain the minimum `value` of all values as well as the `count` of columns with that value.
You can also provide a filter to the `Min()` function to find the minimum age of all patients over 40.
``` request
curl localhost:10101/index/patients/query \
-X POST \
-d 'Min(Row(age > 40), field="age")'
```
``` response
{"results":[{"value":57,"count":1}]}
```
To find the maximum age of all patients, run a `Max` query:
``` request
curl localhost:10101/index/patients/query \
-X POST \
-d 'Max(field="age")'
```
``` response
{"results":[{"value":71,"count":1}]}
```
The results you get from the `Max` query contain the maximum `value` of all values as well as the `count` of columns with that value.
You can also provide a filter to the `Max()` function to find the maximum age of all patients under 40.
``` request
curl localhost:10101/index/patients/query \
-X POST \
-d 'Max(Row(age < 40), field="age")'
```
``` response
{"results":[{"value":34,"count":1}]}
```
### Storing Row and Column Attributes
#### Introduction
Pilosa can store arbitrary values associated to any row or column. In Pilosa, these are referred to as `attributes`, and they can be of type `string`, `integer`, `boolean`, or `float`. In this tutorial we will store some attribute data and then run some queries that return that data.
First, create an index called `books` to use for this tutorial:
``` request
curl localhost:10101/index/books \
-X POST
```
``` response
{"success":true}
```
Next, create a field in the `books` index called `members` which will represent library members who have read books.
``` request
curl localhost:10101/index/books/field/members \
-X POST \
-d '{}'
```
``` response
{"success":true}
```
Now, let's add some books to our index.
``` request
curl localhost:10101/index/books/query \
-X POST \
-d 'SetColumnAttrs(1, name="To Kill a Mockingbird", year=1960)
SetColumnAttrs(2, name="No Name in the Street", year=1972)
SetColumnAttrs(3, name="The Tipping Point", year=2000)
SetColumnAttrs(4, name="Out Stealing Horses", year=2003)
SetColumnAttrs(5, name="The Forever War", year=2008)'
```
``` response
{"results":[null,null,null,null,null]}
```
And add some members.
``` request
curl localhost:10101/index/books/query \
-X POST \
-d 'SetRowAttrs(members, 10001, fullName="John Smith")
SetRowAttrs(members, 10002, fullName="Sue Perkins")
SetRowAttrs(members, 10003, fullName="Jennifer Hawks")
SetRowAttrs(members, 10004, fullName="Pedro Vazquez")
SetRowAttrs(members, 10005, fullName="Pat Washington")'
```
``` response
{"results":[null,null,null,null,null]}
```
At this point we can query one of the `member` records by querying that row.
``` request
curl localhost:10101/index/books/query \
-X POST \
-d 'Row(members=10002)'
```
``` response
{"results":[{"attrs":{"fullName":"Sue Perkins"},"columns":[]}]}
```
Now let's add some data to the matrix such that each pair represents a member who has read that book.
``` request
curl localhost:10101/index/books/query \
-X POST \
-d 'Set(3, members=10001)
Set(5, members=10001)
Set(1, members=10002)
Set(2, members=10002)
Set(4, members=10002)
Set(3, members=10003)
Set(4, members=10004)
Set(5, members=10004)
Set(1, members=10005)
Set(2, members=10005)
Set(3, members=10005)
Set(4, members=10005)
Set(5, members=10005)'
```
``` response
{"results":[true,true,true,true,true,true,true,true,true,true,true,true,true]}
```
Now pull the record for `Sue Perkins` again.
``` request
curl localhost:10101/index/books/query \
-X POST \
-d 'Row(members=10002)'
```
``` response
{"results":[{"attrs":{"fullName":"Sue Perkins"},"columns":[1,2,4]}]}
```
Notice that the result set now contains a list of integers in the `columns` attribute. These integers match the column IDs of the books that Sue has read.
In order to retrieve the attribute information that we stored for each book, we need to add a URL parameter `columnAttrs=true` to the query.
``` request
curl localhost:10101/index/books/query?columnAttrs=true \
-X POST \
-d 'Row(members=10002)'
```
``` response
{
"results":[{"attrs":{"fullName":"Sue Perkins"},"columns":[1,2,4]}],
"columnAttrs":[
{"id":1,"attrs":{"name":"To Kill a Mockingbird","year":1960}},
{"id":2,"attrs":{"name":"No Name in the Street","year":1972}},
{"id":4,"attrs":{"name":"Out Stealing Horses","year":2003}}
]
}
```
The `book` attributes are included in the result set at the `columnAttrs` attribute.
Finally, if we want to find out which books were read by both `Sue` and `Pedro`, we just perform an `Intersect` query on those two members:
``` request
curl localhost:10101/index/books/query?columnAttrs=true \
-X POST \
-d 'Intersect(Row(members=10002), Row(members=10004))'
```
``` response
{
"results":[{"attrs":{},"columns":[4]}],
"columnAttrs":[
{"id":4,"attrs":{"name":"Out Stealing Horses","year":2003}}
]
}
```
Notice that we don't get row attributes on a complex query, but we still get the column attributes—in this case book information.