mirror of
https://github.com/featurebasedb/featurebase.git
synced 2026-09-09 06:31:02 +00:00
Decrease heading depths now that H3 tags are supported by website CSS
This commit is contained in:
Cody Soyland
parent
b102b8cbc8
commit
c9d5852817
12 changed files with 118 additions and 119 deletions
|
|
@ -4,35 +4,35 @@ title = "Administration Guide"
|
|||
|
||||
## Administration Guide
|
||||
|
||||
#### Installing in production
|
||||
### Installing in production
|
||||
|
||||
##### Hardware
|
||||
#### Hardware
|
||||
|
||||
Pilosa is a standalone, compiled Go application, so there is no need to worry about running and configuring a Java VM. Pilosa can run on very small machines and works well with even a medium sized dataset on a personal laptop. If you are reading this section, you are likely ready to deploy a cluster of Pilosa servers handling very large datasets or high velocity data. These are guidelines for running a cluster; specific needs may differ.
|
||||
|
||||
##### Memory
|
||||
#### Memory
|
||||
|
||||
Pilosa holds all row/column bitmap data in main memory. While this data is compressed more than a typical database, available memory is a primary concern. In a production environment, we recommend choosing hardware with a large amount of memory >= 64GB. Prefer a small number of hosts with lots of memory per host over a larger number with less memory each. Larger clusters tend to be less efficient overall due to increased inter-node communication.
|
||||
|
||||
##### CPUs
|
||||
#### CPUs
|
||||
|
||||
Pilosa is a concurrent application written in Go and can take full advantage of multicore machines. The main unit of parallelism is the slice, so a single query will only use a number of cores up to the number of slices stored on that host. Multiple queries can still take advantage of multiple cores as well though, so tuning in this area is dependent on the expected workload.
|
||||
|
||||
##### Disk
|
||||
#### Disk
|
||||
|
||||
Even though the main dataset is in memory Pilosa does back up to disk frequently. We recommend SSDs--especially if you have a write heavy application.
|
||||
|
||||
##### Network
|
||||
#### Network
|
||||
|
||||
Pilosa is designed to be a distributed application, with data replication shared across the cluster. As such every write and read needs to communicate with several nodes. Therefore fast internode communication is essential. If using a service like AWS we recommend that all node exist in the same region and availability zone. The inherent latency of spreading a Pilosa cluster across physical regions it not usually worth the redundancy protection. Since Pilosa is designed to be an Indexing service there already should be a system of record, or ability to rebuild a Cluster quickly from backups.
|
||||
|
||||
##### Overview
|
||||
#### Overview
|
||||
|
||||
While Pilosa does have some high system requirements it is not a best practice to set up a cluster with the fewest, largest machines available. You want an evenly distributed load across several nodes in a cluster to easily recover from a single node failure, and have the resource capacity to handle a missing node until it's repaired or replaced. Nor is it advisable to have many small machines. The internode network traffic will become a bottleneck. You can always add nodes later, but that does require some down time.
|
||||
|
||||
#### Importing and Exporting Data
|
||||
### Importing and Exporting Data
|
||||
|
||||
##### Importing
|
||||
#### Importing
|
||||
|
||||
The import API expects a csv of RowID,ColumnID's.
|
||||
|
||||
|
|
@ -41,7 +41,7 @@ When importing large datasets remember it is much faster to pre sort the data by
|
|||
pilosa import -d project -f stargazer project-stargazer.csv
|
||||
```
|
||||
|
||||
##### Exporting
|
||||
#### Exporting
|
||||
|
||||
Exporting Data to csv can be performed on a live instance of Pilosa. You need to specify the Index, Frame, and View(default is standard). The API also expects the slice number, but the `pilosa export` sub command will export all slices within a Frame. The data will be in csv format RowID,ColumnID and sorted by column ID.
|
||||
```
|
||||
|
|
@ -49,7 +49,7 @@ curl "http://localhost:10101/export?index=repository&frame=stargazer&slice=0&vie
|
|||
--header "Accept: text/csv"
|
||||
```
|
||||
|
||||
#### Versioning
|
||||
### Versioning
|
||||
|
||||
Pilosa follows [Semantic Versioning](http://semver.org/).
|
||||
|
||||
|
|
@ -59,15 +59,15 @@ MAJOR.MINOR.PATCH:
|
|||
* MINOR version when you add functionality in a backwards-compatible manner, and
|
||||
* PATCH version when you make backwards-compatible bug fixes.
|
||||
|
||||
##### PQL versioning
|
||||
#### PQL versioning
|
||||
|
||||
The Pilosa server should support PQL versioning using HTTP headers. On each request, the client should send a Content-Type header and an Accept header. The server should respond with a Content-Type header that matches the client Accept header. The server should also optionally respond with a Warning header if a PQL version is in a deprecation period, or an HTTP 400 error if a PQL version is no longer supported.
|
||||
|
||||
##### Upgrading
|
||||
#### Upgrading
|
||||
|
||||
When upgrading, upgrade clients first, followed by server for all Minor and Patch level changes.
|
||||
|
||||
#### Backup/restore
|
||||
### Backup/restore
|
||||
|
||||
Pilosa continuously writes out the in-memory bitmap data to disk. This data is organized by Index->Frame->Views->Fragment->numbered slice files. These data files can be routinely backed up to restore nodes in a cluster.
|
||||
|
||||
|
|
@ -77,14 +77,14 @@ For larger datasets and to make this process faster you could copy the relevant
|
|||
|
||||
Note: This will only work when the replication factor is >= 2
|
||||
|
||||
##### Using Index Sync
|
||||
#### Using Index Sync
|
||||
|
||||
- Shutdown the cluster.
|
||||
- Modify config file to replace existing node address with new node.
|
||||
- Restart all nodes in the cluster.
|
||||
- Wait for auto Index sync to replicate data from existing nodes to new node.
|
||||
|
||||
##### Copying data files manually
|
||||
#### Copying data files manually
|
||||
|
||||
- To accomplish this goal you will 1st need:
|
||||
- List of all Indexes on your cluster
|
||||
|
|
|
|||
|
|
@ -5,9 +5,9 @@ title = "API Reference"
|
|||
|
||||
## API Reference
|
||||
|
||||
#### `/index`
|
||||
### `/index`
|
||||
|
||||
##### `GET`
|
||||
#### `GET`
|
||||
|
||||
Returns the schema of all indexes in JSON.
|
||||
|
||||
|
|
@ -21,9 +21,9 @@ Response:
|
|||
{"indexes":[{"name":"user","frames":[{"name":"collab"}]}]}
|
||||
```
|
||||
|
||||
#### `/index/<index-name>`
|
||||
### `/index/<index-name>`
|
||||
|
||||
##### `GET`
|
||||
#### `GET`
|
||||
|
||||
Returns the schema of the specified index in JSON.
|
||||
|
||||
|
|
@ -37,7 +37,7 @@ Response:
|
|||
{"index":{"name":"user"}, "frames":[{"name":"collab"}]}]}
|
||||
```
|
||||
|
||||
##### `POST`
|
||||
#### `POST`
|
||||
|
||||
Creates an index with the given name.
|
||||
|
||||
|
|
@ -57,7 +57,7 @@ Response:
|
|||
{}
|
||||
```
|
||||
|
||||
##### `DELETE`
|
||||
#### `DELETE`
|
||||
|
||||
Removes the given index.
|
||||
|
||||
|
|
@ -71,9 +71,9 @@ Response:
|
|||
{}
|
||||
```
|
||||
|
||||
#### `/index/<index-name>/query`
|
||||
### `/index/<index-name>/query`
|
||||
|
||||
##### `POST`
|
||||
#### `POST`
|
||||
|
||||
Sends a query to the Pilosa server with the given index. The request body is UTF-8 encoded text and response body is in JSON by default.
|
||||
|
||||
|
|
@ -107,9 +107,9 @@ Response:
|
|||
}
|
||||
```
|
||||
|
||||
#### `/index/<index-name>/time-quantum`
|
||||
### `/index/<index-name>/time-quantum`
|
||||
|
||||
##### `PATCH`
|
||||
#### `PATCH`
|
||||
|
||||
Changes the time quantum for the given index. This endpoint should be called at most once right after creating a database.
|
||||
|
||||
|
|
@ -139,9 +139,9 @@ Response:
|
|||
{}
|
||||
```
|
||||
|
||||
#### `/index/<index-name>/frame/<frame-name>`
|
||||
### `/index/<index-name>/frame/<frame-name>`
|
||||
|
||||
##### `POST`
|
||||
#### `POST`
|
||||
|
||||
Creates a frame in the given index with the given name.
|
||||
|
||||
|
|
@ -165,7 +165,7 @@ Response:
|
|||
{}
|
||||
```
|
||||
|
||||
##### `DELETE`
|
||||
#### `DELETE`
|
||||
|
||||
Removes the given frame.
|
||||
|
||||
|
|
@ -179,9 +179,9 @@ Response:
|
|||
{}
|
||||
```
|
||||
|
||||
#### `/index/<index-name>/frame/<frame-name>/time-quantum`
|
||||
### `/index/<index-name>/frame/<frame-name>/time-quantum`
|
||||
|
||||
##### `PATCH`
|
||||
#### `PATCH`
|
||||
|
||||
Changes the time quantum for the given frame. This endpoint should be called at most once right after creating a frame.
|
||||
|
||||
|
|
@ -211,9 +211,9 @@ Response:
|
|||
{}
|
||||
```
|
||||
|
||||
#### `/hosts`
|
||||
### `/hosts`
|
||||
|
||||
##### `GET`
|
||||
#### `GET`
|
||||
|
||||
Returns the hosts in the cluster.
|
||||
|
||||
|
|
@ -227,9 +227,9 @@ Response:
|
|||
[{"host":":10101","internalHost":""}]
|
||||
```
|
||||
|
||||
#### `/version`
|
||||
### `/version`
|
||||
|
||||
##### `GET`
|
||||
#### `GET`
|
||||
|
||||
Returns the version of the Pilosa server.
|
||||
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ title = "Client Libraries"
|
|||
## Client Libraries
|
||||
|
||||
|
||||
#### Go
|
||||
### Go
|
||||
|
||||
You can find the Go client library for Pilosa at our [Go Pilosa Repository](https://github.com/pilosa/go-client-pilosa). Check out its [README](https://github.com/pilosa/go-client-pilosa/blob/master/README.md) for more information and installation instructions.
|
||||
|
||||
|
|
@ -84,7 +84,7 @@ func main() {
|
|||
}
|
||||
```
|
||||
|
||||
#### Python
|
||||
### Python
|
||||
|
||||
You can find the Python client library for Pilosa at our [Python Pilosa Repository](https://github.com/pilosa/python-pilosa). Check out its [README](https://github.com/pilosa/python-pilosa/blob/master/README.rst) for more information and installation instructions.
|
||||
|
||||
|
|
@ -139,7 +139,7 @@ except PilosaError as ex:
|
|||
|
||||
```
|
||||
|
||||
#### Java
|
||||
### Java
|
||||
|
||||
You can find the Java client library for Pilosa at our [Java Pilosa Repository](https://github.com/pilosa/java-pilosa). Check out its [README](https://github.com/pilosa/java-pilosa/blob/master/README.md) for more information and installation instructions.
|
||||
|
||||
|
|
|
|||
|
|
@ -10,15 +10,15 @@ All options are available in all three configuration types with the exception of
|
|||
|
||||
The syntax for each option is slightly different between each of the configuration types, but follows a simple formula. See the following three sections for an explanation of each configuration type.
|
||||
|
||||
#### Command line flags
|
||||
### Command line flags
|
||||
|
||||
Pilosa uses GNU/POSIX style flags. Most flags you specify as `--flagname=value` although some have a short form that is a single character and can be specified with a single dash like `-f value`. Running `pilosa server --help` will give an overview of the available flags as well as their short forms (if applicable).
|
||||
|
||||
#### Environment variables
|
||||
### Environment variables
|
||||
|
||||
Every command line flag has a corresponding environment variable. The environment variable is the flag name in all caps, prefxed by `PILOSA_`, and with any dashes replaced by underscores. For example: `--flag-name` becomes `PILOSA_FLAG_NAME`.
|
||||
|
||||
#### Config file
|
||||
### Config file
|
||||
|
||||
The config file is in the [toml format](https://github.com/toml-lang/toml) and has exactly the same options available as the flags and environment variables. Any flag which contains a dot (".") denotes nesting within the config file, so the two flags `--cluster.poll-interval=2m0s` and `--cluster.replicas=1` look like this in the config file:
|
||||
```toml
|
||||
|
|
@ -33,9 +33,9 @@ Any flag that has a value that is a comma separated list on the command line bec
|
|||
hosts = ["one.pilosa.com:10101", "two.pilosa.com:10101"]
|
||||
```
|
||||
|
||||
#### All Options
|
||||
### All Options
|
||||
|
||||
##### Anti Entropy Interval
|
||||
#### Anti Entropy Interval
|
||||
|
||||
* Description: Interval at which the cluster will run its anti-entropy routine which makes sure that all replicas of each fragment are in sync.
|
||||
* Flag: `--anti-entropy.interval="10m0s"`
|
||||
|
|
@ -47,7 +47,7 @@ Any flag that has a value that is a comma separated list on the command line bec
|
|||
interval = "10m0s"
|
||||
```
|
||||
|
||||
##### Bind
|
||||
#### Bind
|
||||
|
||||
* Description: host:port on which the Pilosa server will listen for requests. Host defaults to localhost and port to 10101.
|
||||
* Flag: `--bind="localhost:10101"`
|
||||
|
|
@ -58,7 +58,7 @@ Any flag that has a value that is a comma separated list on the command line bec
|
|||
bind = localhost:10101
|
||||
```
|
||||
|
||||
##### Cluster Hosts
|
||||
#### Cluster Hosts
|
||||
|
||||
* Description: List of hosts in the cluster. Multiple hosts should be comma separated in the flag and env forms.
|
||||
* Flag: `--cluster.hosts="localhost:10101"`
|
||||
|
|
@ -70,7 +70,7 @@ Any flag that has a value that is a comma separated list on the command line bec
|
|||
hosts = ["localhost:10101"]
|
||||
```
|
||||
|
||||
##### Cluster Internal Hosts
|
||||
#### Cluster Internal Hosts
|
||||
|
||||
* Description: List of hosts in the cluster used for internal communication. Multiple hosts should be comma separated in the flag and env forms.
|
||||
* Flag: `--cluster.internal-hosts="localhost:11101"`
|
||||
|
|
@ -82,7 +82,7 @@ Any flag that has a value that is a comma separated list on the command line bec
|
|||
internal-hosts = ["localhost:11101"]
|
||||
```
|
||||
|
||||
##### Cluster Internal Port
|
||||
#### Cluster Internal Port
|
||||
|
||||
* Description: Port to which Pilosa should bind for internal communication.
|
||||
* Flag: `--cluster.internal-port=11101`
|
||||
|
|
@ -94,7 +94,7 @@ Any flag that has a value that is a comma separated list on the command line bec
|
|||
internal-port = 11101
|
||||
```
|
||||
|
||||
##### Cluster Poll Interval
|
||||
#### Cluster Poll Interval
|
||||
|
||||
* Description: Polling interval for cluster.
|
||||
* Flag: `cluster.poll-interval="1m0s"`
|
||||
|
|
@ -106,7 +106,7 @@ Any flag that has a value that is a comma separated list on the command line bec
|
|||
poll-interval = "1m0s"
|
||||
```
|
||||
|
||||
##### Cluster Replicas
|
||||
#### Cluster Replicas
|
||||
|
||||
* Description: Number of hosts each piece of data should be stored on.
|
||||
* Flag: `cluster.replicas=1`
|
||||
|
|
@ -118,7 +118,7 @@ Any flag that has a value that is a comma separated list on the command line bec
|
|||
replicas = 1
|
||||
```
|
||||
|
||||
##### Cluster Type
|
||||
#### Cluster Type
|
||||
|
||||
* Description: Determine how the cluster handles membership and state sharing. Choose from [static, http, gossip].
|
||||
* static - Messaging between nodes is disabled. This is primarily used for testing.
|
||||
|
|
@ -133,7 +133,7 @@ Any flag that has a value that is a comma separated list on the command line bec
|
|||
type = "gossip"
|
||||
```
|
||||
|
||||
##### Data Dir
|
||||
#### Data Dir
|
||||
|
||||
* Description: Directory to store Pilosa data files.
|
||||
* Flag: `--data-dir="~/.pilosa"`
|
||||
|
|
@ -144,7 +144,7 @@ Any flag that has a value that is a comma separated list on the command line bec
|
|||
data-dir = "~/.pilosa"
|
||||
```
|
||||
|
||||
##### Profile CPU
|
||||
#### Profile CPU
|
||||
|
||||
* Description: If this is set to a path, collect a cpu profile and store it there.
|
||||
* Flag: `--profile.cpu="/path/to/somewhere"`
|
||||
|
|
@ -156,7 +156,7 @@ Any flag that has a value that is a comma separated list on the command line bec
|
|||
cpu = "/path/to/somewhere"
|
||||
```
|
||||
|
||||
##### Profile CPU Time
|
||||
#### Profile CPU Time
|
||||
|
||||
* Description: Amount of time to collect cpu profiling data if `profile.cpu` is set.
|
||||
* Flag: `--profile.cpu-time="30s"`
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ title = "Data Model"
|
|||
|
||||
## Data Model
|
||||
|
||||
#### Overview
|
||||
### Overview
|
||||
|
||||
The central component of Pilosa's data model is a boolean matrix. Each cell in the matrix is a single bit - if the bit is set, it indicates that a relationship exists between that particular row and column.
|
||||
|
||||
|
|
@ -14,59 +14,59 @@ Pilosa lays out data first in rows, so queries which get all the set bits in one
|
|||
|
||||
|
||||
|
||||
#### Index
|
||||
### Index
|
||||
|
||||
The purpose of the Index is to represent a data namespace. You cannot perform cross-index queries. Column-level attributes are global to the Index.
|
||||
|
||||
#### Column
|
||||
### Column
|
||||
|
||||
Column ids are sequential increasing integers and are common to all Frames within an Index.
|
||||
|
||||
#### Row
|
||||
### Row
|
||||
|
||||
Row ids are sequential increasing integers namespaced to each Frame within an Index.
|
||||
|
||||
#### Frame
|
||||
### Frame
|
||||
|
||||
Frames are used to segment and define different functional characteristics within your entire index. You can think of a Frame as a table-like data partition within your Index.
|
||||
|
||||
Row attributes are namespaced at the Frame level.
|
||||
|
||||
##### Ranked
|
||||
#### Ranked
|
||||
|
||||
Ranked Frames maintain a sorted cache of column counts by Row ID (yielding the top rows by columns with a bit set in each). This cache facilitates the TopN query. The cache size defaults to 50,000 and can be set at Frame creation.
|
||||
|
||||
|
||||
|
||||
##### LRU
|
||||
#### LRU
|
||||
|
||||
The LRU cache maintains the most recently accessed Rows.
|
||||
|
||||
|
||||
|
||||
#### Time Quantum
|
||||
### Time Quantum
|
||||
|
||||
Setting a time quantum on a frame creates extra indices which allow Range queries down to the interval specified. For example - if the time quantum is set to `YMD`, Range queries down to the granularity of a day are supported.
|
||||
|
||||
#### Attribute
|
||||
### Attribute
|
||||
|
||||
Attributes are arbitrary key/value pairs that can be associated to both rows or columns. This metadata is stored in a separate BoltDB data structure.
|
||||
|
||||
#### Slice
|
||||
### Slice
|
||||
|
||||
Indexes are sharded into groups of columns called Slices - each Slice contains a fixed number of columns which is the SliceWidth.
|
||||
|
||||
Columns are sharded on a preset width, and each shard is referred to as a Slice. Slices are operated on in parallel, and they are evenly distributed across a cluster via a consistent hash algorithm.
|
||||
|
||||
#### View
|
||||
### View
|
||||
|
||||
Views represent the various data layouts within a Frame. The primary View is called Standard, and it contains the typical Row and Column data. The Inverse View contains the same data with the axes inverted.Time-based Views are automatically generated for each time quantum. Views are internally managed by Pilosa, and never exposed directly via the API. This simplifies the functional interface from the physical data representation.
|
||||
|
||||
##### Standard
|
||||
#### Standard
|
||||
|
||||
The standard View contains the same Row/Column format as the input data.
|
||||
|
||||
##### Inverse
|
||||
#### Inverse
|
||||
|
||||
The Inverse View contains the same data with the Row and Column swapped.
|
||||
|
||||
|
|
@ -79,7 +79,7 @@ SetBit(frame="A", rowID=19, columnID=5)
|
|||
|
||||
|
||||
|
||||
##### Time Quantums
|
||||
#### Time Quantums
|
||||
|
||||
If a Frame has a time quantum, then Views are generated for each of the defined time segments. For example, for a frame with a time quantum of `YMD`, the following `SetBit()` queries will result in the data described in the illustration below:
|
||||
|
||||
|
|
|
|||
14
docs/faq.md
14
docs/faq.md
|
|
@ -4,21 +4,21 @@ title = "FAQ"
|
|||
|
||||
## FAQ
|
||||
|
||||
#### What is Pilosa?
|
||||
### What is Pilosa?
|
||||
|
||||
Pilosa is an in-memory, distributed index that is layered over persistent storage. It supports fast ad-hoc queries and segmentation. Pilosa does not require the underlying data to be moved, rather it can be populated in conjunction with data writes, or it can be backfilled asynchronously from any other data store or event processing system. This allows Pilosa to support sub-second queries against very large underlying data sets.
|
||||
|
||||
#### Is Pilosa a database?
|
||||
### Is Pilosa a database?
|
||||
|
||||
Pilosa is not a database in the traditional sense. While Pilosa does store data (both in-memory as well as persisted to disk), it wouldn't typically be used as a primary data store. Instead, one would likely use Pilosa as an index of the data stored in a traditional database or in a data warehouse.
|
||||
|
||||
#### Where does Pilosa fit in my stack?
|
||||
### Where does Pilosa fit in my stack?
|
||||
|
||||
Pilosa sits on top of a data store or multiple data stores.
|
||||
How is Pilosa different than Elasticsearch since they are both indexes?
|
||||
Elasticsearch is a search engine based on Lucene, and is therefore very good at indexing and searching large volumes of unstructured text. As it matures, Elasticsearch has continued to move into the analytics space, but its core data object is still the "document". Pilosa is specifically designed to index structured data and improve query speed. By representing data as the relationship between objects, and then storing those relationships in bitmaps, Pilosa can very efficiently search and compare many millions of data points while still maintaining a small memory footprint.
|
||||
|
||||
#### How do I get my data into Pilosa?
|
||||
### How do I get my data into Pilosa?
|
||||
|
||||
There are typically two methods for getting data into Pilosa: importing large batches of data from an existing data set, and continuously updating Pilosa as data is added or updated.
|
||||
|
||||
|
|
@ -26,15 +26,15 @@ In the first case, one would use the `pilosa import` command to bulk load struct
|
|||
|
||||
For the case where data is continually mutating, one would apply a parallel data writer at the point at which data is written to the persistent data store. This new writer would simultaneously write to Pilosa. An example use case would be one where Kafka was employed as the message broker in your data pipeline, you could introduce an additional Kafka consumer to read from the message log and write mutated data to Pilosa.
|
||||
|
||||
#### What languages can I use with it?
|
||||
### What languages can I use with it?
|
||||
|
||||
There is currently client support for Go, Python, and Java. If you want to use Pilosa with a different language, you can access Pilosa via the Pilosa API.
|
||||
|
||||
#### Do you query Pilosa using SQL?
|
||||
### Do you query Pilosa using SQL?
|
||||
|
||||
One can access Pilosa directly via the terminal using the Pilosa Query Language (PQL), but a typical implementation would use one of the Pilosa client libraries to integrate with an existing codebase. There is currently client support for Go, Python, and Java.
|
||||
|
||||
|
||||
#### Replication on each node?
|
||||
### Replication on each node?
|
||||
|
||||
Pilosa supports a replication factor greater than or equal to one. When replication is configured to be greater than one, then all mutations will be replicated to additional nodes in the cluster. For example, in a five-node cluster consisting of nodes A-B-C-D-E and with replication factor of three, then a write to node B will result in data being written to nodes B, C, and D. If the replication factor is greater than the number of nodes in the cluster, the data will be replicated to every node in the cluster only once.
|
||||
|
|
|
|||
|
|
@ -7,11 +7,11 @@ title = "Getting Started"
|
|||
Pilosa supports an HTTP interface which uses JSON by default.
|
||||
Any HTTP tool can be used to interact with the Pilosa server. The examples in this documentation will use [curl](https://curl.haxx.se/) which is available by default on many UNIX-like systems including Linux and MacOS. Windows users can download curl [here](https://curl.haxx.se/download.html).
|
||||
|
||||
<div class="admonition">
|
||||
<div class="note">
|
||||
<p>Note that Pilosa server requires a high limit for open files. Check the documentation of your system to see how to increase it in case you hit that limit.</p>
|
||||
</div>
|
||||
|
||||
#### Starting Pilosa
|
||||
### Starting Pilosa
|
||||
|
||||
Follow the steps in the [Install]({{< ref "installation.md" >}}) document to install Pilosa.
|
||||
Execute the following in a terminal to run Pilosa with the default configuration (Pilosa will be available at `localhost:10101`):
|
||||
|
|
@ -30,13 +30,13 @@ curl localhost:10101/status
|
|||
|
||||
Which should output: `{"status":{"Nodes":[{"Host":":10101","State":"UP"}]}}`
|
||||
|
||||
#### Sample Project
|
||||
### Sample Project
|
||||
|
||||
In order to better understand Pilosa's capabilities, we will create a sample project called "Star Trace" containing information about the top 1,000 most recently updated Github repositories which have "go" in their name. The Star Trace index will include data points such as programming language, tags, and stargazers—people who have starred a project.
|
||||
|
||||
Although Pilosa doesn't keep the data in a tabular format, we still use the terms "columns" and "rows" when describing the data model. We put the primary objects in columns, and the properties of those objects in rows. For example, the Star Trace project will contain an index called "repository" which contains columns representing Github repositories, and rows representing properties like programming languages and tags. We can better organize the rows by grouping them into sets called Frames. So the "repository" index might have a "languages" frame as well as a "tags" frame. You can learn more about indexes and frames in the [Data Model](../data-model) section of the documentation.
|
||||
|
||||
##### Create the Schema
|
||||
#### Create the Schema
|
||||
|
||||
Note:
|
||||
The queries in this section which are used to set up the indexes in Pilosa just the empty object on success: `{}` - if you would like to verify that a query worked as you expected, you can request the schema as follows:
|
||||
|
|
@ -74,7 +74,7 @@ curl localhost:10101/index/repository/frame/language \
|
|||
-d '{"options": {"rowLabel": "language_id",
|
||||
"inverseEnabled": true}}'
|
||||
```
|
||||
##### Import Some Data
|
||||
#### Import Some Data
|
||||
|
||||
The sample data for the "Star Trace" project is at [Pilosa Getting Started repository](https://github.com/pilosa/getting-started). Download the `stargazer.csv` and `language.csv` files in that repo.
|
||||
|
||||
|
|
@ -100,9 +100,9 @@ docker exec -it pilosa /pilosa import -i repository -f language /language.csv
|
|||
|
||||
Note that, both the user IDs and the repository IDs were remapped to sequential integers in the data files, they don't correspond to actual Github IDs anymore. You can check out `language.txt` to see the mapping for languages.
|
||||
|
||||
##### Make Some Queries
|
||||
#### Make Some Queries
|
||||
|
||||
<div class="admonition">
|
||||
<div class="note">
|
||||
<p>Note the Pilosa server comes with a <a href="../webui/">WebUI</a> for constructing queries in a browser. In local development, it is available at <a href="http://localhost:10101">localhost:10101</a>.</p>
|
||||
</div>
|
||||
|
||||
|
|
@ -148,6 +148,6 @@ curl localhost:10101/index/repository/query \
|
|||
-d 'SetBit(frame="stargazer", repo_id=77777, stargazer_id=99999)'
|
||||
```
|
||||
|
||||
#### What's Next?
|
||||
### What's Next?
|
||||
|
||||
You can jump to [Data Model](../data-model/) for an in-depth look at Pilosa's data model, or [Query Language](../query-language/) for more details about **PQL**, the query language of Pilosa. Check out the [Tutorials](../tutorials/) for example implementations of real world use cases for Pilosa. Ready to get going in your favorite language? Have a peek at our small but expanding set of official [Client Libraries](../client-libraries/).
|
||||
|
|
|
|||
|
|
@ -2,8 +2,7 @@
|
|||
title = "Glossary"
|
||||
+++
|
||||
|
||||
# Glossary
|
||||
|
||||
## Glossary
|
||||
|
||||
<strong id="index">Index:</strong> Indexes are the top level container in Pilosa - similar to a database in an RDBMS. Queries cannot operate across multiple indexes.
|
||||
|
||||
|
|
|
|||
|
|
@ -7,11 +7,11 @@ title = "Installation"
|
|||
|
||||
Pilosa is currently available for [MacOS](#installing-on-macos) and [Linux](#installing-on-linux).
|
||||
|
||||
#### Installing on MacOS
|
||||
### Installing on MacOS
|
||||
|
||||
There are three ways to install Pilosa on MacOS: download the binary (recommended), build from source, or use Docker.
|
||||
|
||||
##### Download the Binary
|
||||
#### Download the Binary
|
||||
|
||||
1. Download the latest release:
|
||||
```
|
||||
|
|
@ -71,7 +71,7 @@ There are three ways to install Pilosa on MacOS: download the binary (recommende
|
|||
|
||||
You're good to go!
|
||||
|
||||
##### Build from Source
|
||||
#### Build from Source
|
||||
|
||||
1. Install the prerequisites:
|
||||
|
||||
|
|
@ -131,7 +131,7 @@ There are three ways to install Pilosa on MacOS: download the binary (recommende
|
|||
|
||||
You're good to go!
|
||||
|
||||
##### Use Docker
|
||||
#### Use Docker
|
||||
|
||||
1. Install Docker for Mac.
|
||||
|
||||
|
|
@ -152,16 +152,16 @@ If you don't see the server listed, start the Docker application.
|
|||
docker run --rm pilosa/pilosa:latest help
|
||||
```
|
||||
|
||||
##### What's next?
|
||||
#### What's next?
|
||||
|
||||
Head over to the [Getting Started](../getting-started/) guide to create your first Pilosa index.
|
||||
|
||||
|
||||
#### Installing on Linux
|
||||
### Installing on Linux
|
||||
|
||||
There are three ways to install Pilosa on Linux: download the binary (recommended), build from source, or use Docker.
|
||||
|
||||
##### Download the Binary
|
||||
#### Download the Binary
|
||||
|
||||
1. To install the latest version of Pilosa, download the latest release:
|
||||
```
|
||||
|
|
@ -221,7 +221,7 @@ There are three ways to install Pilosa on Linux: download the binary (recommende
|
|||
|
||||
You're good to go!
|
||||
|
||||
##### Build from Source
|
||||
#### Build from Source
|
||||
|
||||
1. Install the prerequisites:
|
||||
|
||||
|
|
@ -282,7 +282,7 @@ There are three ways to install Pilosa on Linux: download the binary (recommende
|
|||
You're good to go!
|
||||
|
||||
|
||||
##### Use Docker
|
||||
#### Use Docker
|
||||
|
||||
1. Install Docker.
|
||||
|
||||
|
|
@ -303,17 +303,17 @@ There are three ways to install Pilosa on Linux: download the binary (recommende
|
|||
docker run --rm pilosa/pilosa:latest help
|
||||
```
|
||||
|
||||
##### What's next?
|
||||
#### What's next?
|
||||
|
||||
Head over to the [Getting Started](../getting-started/) guide to create your first Pilosa index.
|
||||
|
||||
|
||||
<!--
|
||||
#### Windows
|
||||
### Windows
|
||||
|
||||
Windows is currently not supported as a target deployment platform for Pilosa, but developing and running Pilosa is made possible by Windows Subsystem for Linux and Docker. See the [Docker](#docker) documentation for using Docker for Windows and Docker Toolbox. You can find documentation about installing Windows Subsystem for Linux at https://msdn.microsoft.com/en-us/commandline/wsl/install_guide. From there, use the instructions in the [Linux Install](#installing-on-linux) section in the this document.
|
||||
|
||||
#### Docker
|
||||
### Docker
|
||||
|
||||
1. Install Docker for your platform. On Linux, Docker is available via your package manager. On MacOS, you can use Docker for Mac or Docker Toolbox. On Windows, you can use Docker for Windows or Docker Toolbox.
|
||||
|
||||
|
|
|
|||
|
|
@ -8,9 +8,9 @@ The [Pilosa Dev Kit](https://github.com/pilosa/pdk) contains Go libraries to hel
|
|||
|
||||
The PDK also contains some fully worked examples which make use of its tools. These are available in the `usecase` subdirectory and can be run as subcommands of the `pdk` binary.
|
||||
|
||||
#### Library
|
||||
### Library
|
||||
|
||||
##### Mapping
|
||||
#### Mapping
|
||||
|
||||
Importing data into Pilosa is dependent on mapping it to integer IDs. PDK provides some predefined functions for inline mapping to simplify this process, supported by a framework for linking these mappings with the associated fields in a source CSV file. If no custom mapping is necessary, the entire import process can be described by an import definition file. The file is composed of four main parts:
|
||||
|
||||
|
|
@ -51,12 +51,12 @@ Some of the simple mapper functions available with PDK include:
|
|||
* GridMapper: Maps a pair of floats to a single integer, identifying a cell in a rectangular grid. This can be used, for example, to represent (latitude, longitude) location coarsely, as in the taxi data example.
|
||||
* CustomMapper: When none of the predefined mappers will work, or when multiple fields determine a row ID value, an arbitrary mapping function can be used. Define a function in Go, with the necessary behavior, and wrap it in a CustomMapper.
|
||||
|
||||
#### Examples
|
||||
### Examples
|
||||
|
||||
Run `make install` to build and install the `pdk` binary which contains all the examples. Just running `pdk` will bring up a list of all the examples, with a brief description of each. `pdk help <example>` will bring up a more detailed description of that example along with all arguments that it accepts to configure its functionality.
|
||||
|
||||
<!--
|
||||
##### Net
|
||||
#### Net
|
||||
|
||||
A detailed discussion of using Pilosa to index network traffic data is available [here - TODO]link blog post). This will discuss the implementation of `pdk net` as it relates to the use of the PDK library tools.
|
||||
-->
|
||||
|
|
|
|||
|
|
@ -4,9 +4,9 @@ title = "Tutorials"
|
|||
|
||||
## Tutorials
|
||||
|
||||
#### Transportation
|
||||
### Transportation
|
||||
|
||||
##### Introduction
|
||||
#### Introduction
|
||||
|
||||
New York City released an extremely detailed data set of over 1 billion taxi rides taken in the city - this data has become a popular target for analysis by tech bloggers and has been very well studied. For this reason, we thought it would be interesting to import this data to Pilosa in order to compare with other data stores and techniques on the exact same data set.
|
||||
|
||||
|
|
@ -16,7 +16,7 @@ We've written a tool to help import the NYC taxi data into Pilosa - this tool is
|
|||
|
||||
After initial setup, the PDK import tool does everything we need to define a Pilosa schema, map data to bitmaps accordingly, and import it into Pilosa.
|
||||
|
||||
##### Data Model
|
||||
#### Data Model
|
||||
|
||||
The NYC taxi data is comprised of a number of csv files listed here: http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml. These data files have around 20 columns, about half of which are relevant to the benchmark queries we're looking at:
|
||||
|
||||
|
|
@ -54,15 +54,15 @@ frame |mapping
|
|||
duration_minutes |round(drop_timestamp - pickup_timestamp) → row ID
|
||||
speed_mph |round(dist_miles / (drop_timestamp - pickup_timestamp)) → row ID
|
||||
|
||||
##### Mapping
|
||||
#### Mapping
|
||||
|
||||
Each column that we want to use must be mapped to a combination of frames and row IDs according to some rule. There are many ways to approach this mapping, and the taxi dataset gives us a good overview of possibilities.
|
||||
|
||||
###### 0 columns → 1 frame
|
||||
##### 0 columns → 1 frame
|
||||
|
||||
cab_type: contains one row for each type of cab. Each column, representing one ride, has a bit set in exactly one row of this frame. The mapping is a simple enumeration, for example yellow=0, green=1, etc. The values of the bits in this frame are determined by the source of the data. That is, we're importing data from several disparate sources: NYC yellow taxi cabs, NYC green taxi cabs, and Uber cars. For each source, the single row to be set in the cab_type frame is constant.
|
||||
|
||||
###### 1 column → 1 frame
|
||||
##### 1 column → 1 frame
|
||||
|
||||
The following three frames are mapped in a simple direct way from single columns of the original data.
|
||||
|
||||
|
|
@ -124,7 +124,7 @@ Here, we define a list of Mappers, each including a name, which we use to refer
|
|||
|
||||
**passenger_count:** This column contains small integers, so we use one of the simplest possible mappings: the column value is the row ID.
|
||||
|
||||
###### 1 column → multiple frames
|
||||
##### 1 column → multiple frames
|
||||
|
||||
When working with a composite data type like a timestamp, there are plenty of mapping options. In this case, we expect to see interesting periodic trends, so we want to encode the cyclic components of time in a way that allows us to look at them independently during analysis.
|
||||
|
||||
|
|
@ -134,7 +134,7 @@ We might continue this pattern with hours, minutes, and seconds, but we don't ha
|
|||
|
||||
We do all of this for each timestamp of interest, one for pickup time and one for dropoff time. That gives us eight total frames for two timestamps: pickup_year, pickup_month, pickup_day, pickup_time, drop_year, drop_month, drop_day, drop_time.
|
||||
|
||||
###### Multiple columns → 1 frame
|
||||
##### Multiple columns → 1 frame
|
||||
|
||||
The ride data also contains geolocation data: latitude and longitude for both pickup and dropoff. We just want to be able to produce a rough overview heatmap of ride locations, so we use a grid mapping. We divide the area of interest into a 100x100 grid in latitude-longitude space, label each cell in this grid with a single integer, and use that integer as the row ID.
|
||||
|
||||
|
|
@ -142,7 +142,7 @@ We do all of this for each location of interest, one for pickup and one for drop
|
|||
|
||||
Again, there are many mapping options for location data. For example, we might convert to a different coordinate system, apply a projection, or aggregate locations into real-world regions such as neighborhoods. Here, the simple approach is sufficient.
|
||||
|
||||
###### Complex mappings
|
||||
##### Complex mappings
|
||||
|
||||
We also anticipate looking for trends in ride duration and speed, so we want to capture this information during the import process. For the frame `duration_minutes`, we compute a row ID as `round((drop_timestamp - pickup_timestamp).minutes)`. For the frame `speed_mph`, we compute row ID as `round(dist_miles / (drop_timestamp - pickup_timestamp).minutes)`. These mapping calculations are straightforward, but because they require arithmetic operations on multiple columns, they are a bit too complex to capture in the basic mappers available in PDK. Instead, we define custom mappers to do the work:
|
||||
```go
|
||||
|
|
@ -156,11 +156,11 @@ durm := pdk.CustomMapper{
|
|||
}
|
||||
```
|
||||
|
||||
##### Import process
|
||||
#### Import process
|
||||
|
||||
After designing this schema and mapping, we capture it in a JSON definition file that can be read by the PDK import tool. Running `pdk taxi` runs the import based on the information in this file. See [PDK](../pdk) for more details on this process.
|
||||
|
||||
##### Queries
|
||||
#### Queries
|
||||
|
||||
Now we can run some example queries.
|
||||
|
||||
|
|
@ -194,9 +194,9 @@ for pcount, topn in zip(pcounts, resp.json()['results']):
|
|||
|
||||
For more examples and details, see this [ipython notebook](https://github.com/alanbernstein/pilosa-notebooks/blob/master/taxi-use-case.ipynb).
|
||||
|
||||
#### Chemical similarity search
|
||||
### Chemical similarity search
|
||||
|
||||
##### Overview
|
||||
#### Overview
|
||||
|
||||
The notion of chemical similarity (or molecular similarity) plays an important role in predicting the properties of chemical compounds, designing chemicals with a predefined set of properties, and—especially—conducting drug design studies. All of these are accomplished by screening large indexes containing structures of available or potentially available chemicals.
|
||||
|
||||
|
|
@ -211,7 +211,7 @@ A and B are sets of fingerprint bits on in the fingerprints of molecule A and mo
|
|||
|
||||
All source code to calculate tanimoto for molecule fingerprint using Pilosa is available in a Github repository https://github.com/pilosa/chem-usecase
|
||||
|
||||
##### Data model
|
||||
#### Data model
|
||||
|
||||
We use the latest ChEMBL release chembl_22.sdf for test data. Each molecule in the SD file gives us the canonical isomeric SMILES (Simplified molecular-input line-entry system) and chembl_id.
|
||||
|
||||
|
|
@ -248,7 +248,7 @@ Index: mole
|
|||
|
||||
After retrieving chembl_id from the Inverse View, we can use the Tanimoto coefficient to compare chembl_id with the entire data set of molecules. The result of this comparison is the list of `chembl_id`s that have a Tanimoto coefficient greater than the given threshold.
|
||||
|
||||
##### Import process
|
||||
#### Import process
|
||||
|
||||
To import data into Pilosa, we need to get chembl_id and SMILES from SD files, convert SMILES to Morgan fingerprints, and then write chembl_id and fingerprint to Pilosa. The fastest way is to extracted chembl_id and SMILES from SD file to csv file, then use the `pilosa import` command to import the csv file into Pilosa. Since chembl_id in the SD file is always paired with CHEMBL, e.g CHEMBL6329, and because Pilosa doesn't support string keys, we will ignore CHEMBL and instead use chembl_id as an integer key.
|
||||
|
||||
|
|
@ -276,7 +276,7 @@ Run the following commands to import the csv data into the `mole` index:
|
|||
pilosa import -d mole -f fingerprint id_fingerprint.csv
|
||||
```
|
||||
|
||||
##### Queries
|
||||
#### Queries
|
||||
|
||||
Get chembl_id from a given SMILES:
|
||||
```
|
||||
|
|
@ -330,7 +330,7 @@ Return chembl_id = [6223, 269758, 6206, 6228]. This script uses Pilosa’s TopN
|
|||
topn = requests.post("http://127.0.0.1:10101/index/mol/query" , data=query_string)
|
||||
```
|
||||
|
||||
##### Benchmark
|
||||
#### Benchmark
|
||||
|
||||
To run benchmark for specific chembl_id for different similarity threshold at percentage of [50, 70, 75, 80, 85, 90], run following command:
|
||||
```
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ title = "WebUI"
|
|||
The Pilosa server comes packaged with in-browser WebUI. When you run a local Pilosa server on the default host, you can access it at [localhost:10101](http://localhost:10101)
|
||||
This can be used for constructing queries and viewing the cluster status.
|
||||
|
||||
#### Console
|
||||
### Console
|
||||
|
||||
The [Console view](http://localhost:10101/#console) allows you to enter [PQL](../query-language) queries and run them against your locally running server. First you must select an Index with the Select index dropdown.
|
||||
|
||||
|
|
@ -31,6 +31,6 @@ Index and frame creation also supports options like `columnLabel`,`rowLabel` or
|
|||
- `:create frame <framename> rowLabel=row_id inverseEnabled=true cacheSize=10000`
|
||||
|
||||
|
||||
#### Cluster Admin
|
||||
### Cluster Admin
|
||||
|
||||
Use the [Cluster Admin tab](http://localhost:10101/#admin) to view the current status of your cluster. This contains information on each node in the cluster, plus the list of Indexes and Frames.
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue