From c9d585281791ab1d3eb5662dac283fbb00bd8e29 Mon Sep 17 00:00:00 2001 From: Cody Soyland Date: Thu, 18 May 2017 15:47:27 -0500 Subject: [PATCH] Decrease heading depths now that H3 tags are supported by website CSS --- docs/administration.md | 32 ++++++++++++++++---------------- docs/api-reference.md | 38 +++++++++++++++++++------------------- docs/client-libraries.md | 6 +++--- docs/configuration.md | 30 +++++++++++++++--------------- docs/data-model.md | 28 ++++++++++++++-------------- docs/faq.md | 14 +++++++------- docs/getting-started.md | 16 ++++++++-------- docs/glossary.md | 3 +-- docs/installation.md | 24 ++++++++++++------------ docs/pdk.md | 8 ++++---- docs/tutorials.md | 34 +++++++++++++++++----------------- docs/webui.md | 4 ++-- 12 files changed, 118 insertions(+), 119 deletions(-) diff --git a/docs/administration.md b/docs/administration.md index 9859b862c..26c7ed7ba 100644 --- a/docs/administration.md +++ b/docs/administration.md @@ -4,35 +4,35 @@ title = "Administration Guide" ## Administration Guide -#### Installing in production +### Installing in production -##### Hardware +#### Hardware Pilosa is a standalone, compiled Go application, so there is no need to worry about running and configuring a Java VM. Pilosa can run on very small machines and works well with even a medium sized dataset on a personal laptop. If you are reading this section, you are likely ready to deploy a cluster of Pilosa servers handling very large datasets or high velocity data. These are guidelines for running a cluster; specific needs may differ. -##### Memory +#### Memory Pilosa holds all row/column bitmap data in main memory. While this data is compressed more than a typical database, available memory is a primary concern. In a production environment, we recommend choosing hardware with a large amount of memory >= 64GB. Prefer a small number of hosts with lots of memory per host over a larger number with less memory each. Larger clusters tend to be less efficient overall due to increased inter-node communication. -##### CPUs +#### CPUs Pilosa is a concurrent application written in Go and can take full advantage of multicore machines. The main unit of parallelism is the slice, so a single query will only use a number of cores up to the number of slices stored on that host. Multiple queries can still take advantage of multiple cores as well though, so tuning in this area is dependent on the expected workload. -##### Disk +#### Disk Even though the main dataset is in memory Pilosa does back up to disk frequently. We recommend SSDs--especially if you have a write heavy application. -##### Network +#### Network Pilosa is designed to be a distributed application, with data replication shared across the cluster. As such every write and read needs to communicate with several nodes. Therefore fast internode communication is essential. If using a service like AWS we recommend that all node exist in the same region and availability zone. The inherent latency of spreading a Pilosa cluster across physical regions it not usually worth the redundancy protection. Since Pilosa is designed to be an Indexing service there already should be a system of record, or ability to rebuild a Cluster quickly from backups. -##### Overview +#### Overview While Pilosa does have some high system requirements it is not a best practice to set up a cluster with the fewest, largest machines available. You want an evenly distributed load across several nodes in a cluster to easily recover from a single node failure, and have the resource capacity to handle a missing node until it's repaired or replaced. Nor is it advisable to have many small machines. The internode network traffic will become a bottleneck. You can always add nodes later, but that does require some down time. -#### Importing and Exporting Data +### Importing and Exporting Data -##### Importing +#### Importing The import API expects a csv of RowID,ColumnID's. @@ -41,7 +41,7 @@ When importing large datasets remember it is much faster to pre sort the data by pilosa import -d project -f stargazer project-stargazer.csv ``` -##### Exporting +#### Exporting Exporting Data to csv can be performed on a live instance of Pilosa. You need to specify the Index, Frame, and View(default is standard). The API also expects the slice number, but the `pilosa export` sub command will export all slices within a Frame. The data will be in csv format RowID,ColumnID and sorted by column ID. ``` @@ -49,7 +49,7 @@ curl "http://localhost:10101/export?index=repository&frame=stargazer&slice=0&vie --header "Accept: text/csv" ``` -#### Versioning +### Versioning Pilosa follows [Semantic Versioning](http://semver.org/). @@ -59,15 +59,15 @@ MAJOR.MINOR.PATCH: * MINOR version when you add functionality in a backwards-compatible manner, and * PATCH version when you make backwards-compatible bug fixes. -##### PQL versioning +#### PQL versioning The Pilosa server should support PQL versioning using HTTP headers. On each request, the client should send a Content-Type header and an Accept header. The server should respond with a Content-Type header that matches the client Accept header. The server should also optionally respond with a Warning header if a PQL version is in a deprecation period, or an HTTP 400 error if a PQL version is no longer supported. -##### Upgrading +#### Upgrading When upgrading, upgrade clients first, followed by server for all Minor and Patch level changes. -#### Backup/restore +### Backup/restore Pilosa continuously writes out the in-memory bitmap data to disk. This data is organized by Index->Frame->Views->Fragment->numbered slice files. These data files can be routinely backed up to restore nodes in a cluster. @@ -77,14 +77,14 @@ For larger datasets and to make this process faster you could copy the relevant Note: This will only work when the replication factor is >= 2 -##### Using Index Sync +#### Using Index Sync - Shutdown the cluster. - Modify config file to replace existing node address with new node. - Restart all nodes in the cluster. - Wait for auto Index sync to replicate data from existing nodes to new node. -##### Copying data files manually +#### Copying data files manually - To accomplish this goal you will 1st need: - List of all Indexes on your cluster diff --git a/docs/api-reference.md b/docs/api-reference.md index ef89e686a..55bbde24e 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -5,9 +5,9 @@ title = "API Reference" ## API Reference -#### `/index` +### `/index` -##### `GET` +#### `GET` Returns the schema of all indexes in JSON. @@ -21,9 +21,9 @@ Response: {"indexes":[{"name":"user","frames":[{"name":"collab"}]}]} ``` -#### `/index/` +### `/index/` -##### `GET` +#### `GET` Returns the schema of the specified index in JSON. @@ -37,7 +37,7 @@ Response: {"index":{"name":"user"}, "frames":[{"name":"collab"}]}]} ``` -##### `POST` +#### `POST` Creates an index with the given name. @@ -57,7 +57,7 @@ Response: {} ``` -##### `DELETE` +#### `DELETE` Removes the given index. @@ -71,9 +71,9 @@ Response: {} ``` -#### `/index//query` +### `/index//query` -##### `POST` +#### `POST` Sends a query to the Pilosa server with the given index. The request body is UTF-8 encoded text and response body is in JSON by default. @@ -107,9 +107,9 @@ Response: } ``` -#### `/index//time-quantum` +### `/index//time-quantum` -##### `PATCH` +#### `PATCH` Changes the time quantum for the given index. This endpoint should be called at most once right after creating a database. @@ -139,9 +139,9 @@ Response: {} ``` -#### `/index//frame/` +### `/index//frame/` -##### `POST` +#### `POST` Creates a frame in the given index with the given name. @@ -165,7 +165,7 @@ Response: {} ``` -##### `DELETE` +#### `DELETE` Removes the given frame. @@ -179,9 +179,9 @@ Response: {} ``` -#### `/index//frame//time-quantum` +### `/index//frame//time-quantum` -##### `PATCH` +#### `PATCH` Changes the time quantum for the given frame. This endpoint should be called at most once right after creating a frame. @@ -211,9 +211,9 @@ Response: {} ``` -#### `/hosts` +### `/hosts` -##### `GET` +#### `GET` Returns the hosts in the cluster. @@ -227,9 +227,9 @@ Response: [{"host":":10101","internalHost":""}] ``` -#### `/version` +### `/version` -##### `GET` +#### `GET` Returns the version of the Pilosa server. diff --git a/docs/client-libraries.md b/docs/client-libraries.md index ab62e2bdf..e7a439371 100644 --- a/docs/client-libraries.md +++ b/docs/client-libraries.md @@ -5,7 +5,7 @@ title = "Client Libraries" ## Client Libraries -#### Go +### Go You can find the Go client library for Pilosa at our [Go Pilosa Repository](https://github.com/pilosa/go-client-pilosa). Check out its [README](https://github.com/pilosa/go-client-pilosa/blob/master/README.md) for more information and installation instructions. @@ -84,7 +84,7 @@ func main() { } ``` -#### Python +### Python You can find the Python client library for Pilosa at our [Python Pilosa Repository](https://github.com/pilosa/python-pilosa). Check out its [README](https://github.com/pilosa/python-pilosa/blob/master/README.rst) for more information and installation instructions. @@ -139,7 +139,7 @@ except PilosaError as ex: ``` -#### Java +### Java You can find the Java client library for Pilosa at our [Java Pilosa Repository](https://github.com/pilosa/java-pilosa). Check out its [README](https://github.com/pilosa/java-pilosa/blob/master/README.md) for more information and installation instructions. diff --git a/docs/configuration.md b/docs/configuration.md index f79a98253..12153dde1 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -10,15 +10,15 @@ All options are available in all three configuration types with the exception of The syntax for each option is slightly different between each of the configuration types, but follows a simple formula. See the following three sections for an explanation of each configuration type. -#### Command line flags +### Command line flags Pilosa uses GNU/POSIX style flags. Most flags you specify as `--flagname=value` although some have a short form that is a single character and can be specified with a single dash like `-f value`. Running `pilosa server --help` will give an overview of the available flags as well as their short forms (if applicable). -#### Environment variables +### Environment variables Every command line flag has a corresponding environment variable. The environment variable is the flag name in all caps, prefxed by `PILOSA_`, and with any dashes replaced by underscores. For example: `--flag-name` becomes `PILOSA_FLAG_NAME`. -#### Config file +### Config file The config file is in the [toml format](https://github.com/toml-lang/toml) and has exactly the same options available as the flags and environment variables. Any flag which contains a dot (".") denotes nesting within the config file, so the two flags `--cluster.poll-interval=2m0s` and `--cluster.replicas=1` look like this in the config file: ```toml @@ -33,9 +33,9 @@ Any flag that has a value that is a comma separated list on the command line bec hosts = ["one.pilosa.com:10101", "two.pilosa.com:10101"] ``` -#### All Options +### All Options -##### Anti Entropy Interval +#### Anti Entropy Interval * Description: Interval at which the cluster will run its anti-entropy routine which makes sure that all replicas of each fragment are in sync. * Flag: `--anti-entropy.interval="10m0s"` @@ -47,7 +47,7 @@ Any flag that has a value that is a comma separated list on the command line bec interval = "10m0s" ``` -##### Bind +#### Bind * Description: host:port on which the Pilosa server will listen for requests. Host defaults to localhost and port to 10101. * Flag: `--bind="localhost:10101"` @@ -58,7 +58,7 @@ Any flag that has a value that is a comma separated list on the command line bec bind = localhost:10101 ``` -##### Cluster Hosts +#### Cluster Hosts * Description: List of hosts in the cluster. Multiple hosts should be comma separated in the flag and env forms. * Flag: `--cluster.hosts="localhost:10101"` @@ -70,7 +70,7 @@ Any flag that has a value that is a comma separated list on the command line bec hosts = ["localhost:10101"] ``` -##### Cluster Internal Hosts +#### Cluster Internal Hosts * Description: List of hosts in the cluster used for internal communication. Multiple hosts should be comma separated in the flag and env forms. * Flag: `--cluster.internal-hosts="localhost:11101"` @@ -82,7 +82,7 @@ Any flag that has a value that is a comma separated list on the command line bec internal-hosts = ["localhost:11101"] ``` -##### Cluster Internal Port +#### Cluster Internal Port * Description: Port to which Pilosa should bind for internal communication. * Flag: `--cluster.internal-port=11101` @@ -94,7 +94,7 @@ Any flag that has a value that is a comma separated list on the command line bec internal-port = 11101 ``` -##### Cluster Poll Interval +#### Cluster Poll Interval * Description: Polling interval for cluster. * Flag: `cluster.poll-interval="1m0s"` @@ -106,7 +106,7 @@ Any flag that has a value that is a comma separated list on the command line bec poll-interval = "1m0s" ``` -##### Cluster Replicas +#### Cluster Replicas * Description: Number of hosts each piece of data should be stored on. * Flag: `cluster.replicas=1` @@ -118,7 +118,7 @@ Any flag that has a value that is a comma separated list on the command line bec replicas = 1 ``` -##### Cluster Type +#### Cluster Type * Description: Determine how the cluster handles membership and state sharing. Choose from [static, http, gossip]. * static - Messaging between nodes is disabled. This is primarily used for testing. @@ -133,7 +133,7 @@ Any flag that has a value that is a comma separated list on the command line bec type = "gossip" ``` -##### Data Dir +#### Data Dir * Description: Directory to store Pilosa data files. * Flag: `--data-dir="~/.pilosa"` @@ -144,7 +144,7 @@ Any flag that has a value that is a comma separated list on the command line bec data-dir = "~/.pilosa" ``` -##### Profile CPU +#### Profile CPU * Description: If this is set to a path, collect a cpu profile and store it there. * Flag: `--profile.cpu="/path/to/somewhere"` @@ -156,7 +156,7 @@ Any flag that has a value that is a comma separated list on the command line bec cpu = "/path/to/somewhere" ``` -##### Profile CPU Time +#### Profile CPU Time * Description: Amount of time to collect cpu profiling data if `profile.cpu` is set. * Flag: `--profile.cpu-time="30s"` diff --git a/docs/data-model.md b/docs/data-model.md index f11d1da88..b4b98de9a 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -4,7 +4,7 @@ title = "Data Model" ## Data Model -#### Overview +### Overview The central component of Pilosa's data model is a boolean matrix. Each cell in the matrix is a single bit - if the bit is set, it indicates that a relationship exists between that particular row and column. @@ -14,59 +14,59 @@ Pilosa lays out data first in rows, so queries which get all the set bits in one ![data model diagram](/img/docs/data-model.svg) -#### Index +### Index The purpose of the Index is to represent a data namespace. You cannot perform cross-index queries. Column-level attributes are global to the Index. -#### Column +### Column Column ids are sequential increasing integers and are common to all Frames within an Index. -#### Row +### Row Row ids are sequential increasing integers namespaced to each Frame within an Index. -#### Frame +### Frame Frames are used to segment and define different functional characteristics within your entire index. You can think of a Frame as a table-like data partition within your Index. Row attributes are namespaced at the Frame level. -##### Ranked +#### Ranked Ranked Frames maintain a sorted cache of column counts by Row ID (yielding the top rows by columns with a bit set in each). This cache facilitates the TopN query. The cache size defaults to 50,000 and can be set at Frame creation. ![ranked frame diagram](/img/docs/frame-ranked.svg) -##### LRU +#### LRU The LRU cache maintains the most recently accessed Rows. ![lru frame diagram](/img/docs/frame-lru.svg) -#### Time Quantum +### Time Quantum Setting a time quantum on a frame creates extra indices which allow Range queries down to the interval specified. For example - if the time quantum is set to `YMD`, Range queries down to the granularity of a day are supported. -#### Attribute +### Attribute Attributes are arbitrary key/value pairs that can be associated to both rows or columns. This metadata is stored in a separate BoltDB data structure. -#### Slice +### Slice Indexes are sharded into groups of columns called Slices - each Slice contains a fixed number of columns which is the SliceWidth. Columns are sharded on a preset width, and each shard is referred to as a Slice. Slices are operated on in parallel, and they are evenly distributed across a cluster via a consistent hash algorithm. -#### View +### View Views represent the various data layouts within a Frame. The primary View is called Standard, and it contains the typical Row and Column data. The Inverse View contains the same data with the axes inverted.Time-based Views are automatically generated for each time quantum. Views are internally managed by Pilosa, and never exposed directly via the API. This simplifies the functional interface from the physical data representation. -##### Standard +#### Standard The standard View contains the same Row/Column format as the input data. -##### Inverse +#### Inverse The Inverse View contains the same data with the Row and Column swapped. @@ -79,7 +79,7 @@ SetBit(frame="A", rowID=19, columnID=5) ![inverse frame diagram](/img/docs/frame-inverse.svg) -##### Time Quantums +#### Time Quantums If a Frame has a time quantum, then Views are generated for each of the defined time segments. For example, for a frame with a time quantum of `YMD`, the following `SetBit()` queries will result in the data described in the illustration below: diff --git a/docs/faq.md b/docs/faq.md index 89c99f890..cea161aa3 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -4,21 +4,21 @@ title = "FAQ" ## FAQ -#### What is Pilosa? +### What is Pilosa? Pilosa is an in-memory, distributed index that is layered over persistent storage. It supports fast ad-hoc queries and segmentation. Pilosa does not require the underlying data to be moved, rather it can be populated in conjunction with data writes, or it can be backfilled asynchronously from any other data store or event processing system. This allows Pilosa to support sub-second queries against very large underlying data sets. -#### Is Pilosa a database? +### Is Pilosa a database? Pilosa is not a database in the traditional sense. While Pilosa does store data (both in-memory as well as persisted to disk), it wouldn't typically be used as a primary data store. Instead, one would likely use Pilosa as an index of the data stored in a traditional database or in a data warehouse. -#### Where does Pilosa fit in my stack? +### Where does Pilosa fit in my stack? Pilosa sits on top of a data store or multiple data stores. How is Pilosa different than Elasticsearch since they are both indexes? Elasticsearch is a search engine based on Lucene, and is therefore very good at indexing and searching large volumes of unstructured text. As it matures, Elasticsearch has continued to move into the analytics space, but its core data object is still the "document". Pilosa is specifically designed to index structured data and improve query speed. By representing data as the relationship between objects, and then storing those relationships in bitmaps, Pilosa can very efficiently search and compare many millions of data points while still maintaining a small memory footprint. -#### How do I get my data into Pilosa? +### How do I get my data into Pilosa? There are typically two methods for getting data into Pilosa: importing large batches of data from an existing data set, and continuously updating Pilosa as data is added or updated. @@ -26,15 +26,15 @@ In the first case, one would use the `pilosa import` command to bulk load struct For the case where data is continually mutating, one would apply a parallel data writer at the point at which data is written to the persistent data store. This new writer would simultaneously write to Pilosa. An example use case would be one where Kafka was employed as the message broker in your data pipeline, you could introduce an additional Kafka consumer to read from the message log and write mutated data to Pilosa. -#### What languages can I use with it? +### What languages can I use with it? There is currently client support for Go, Python, and Java. If you want to use Pilosa with a different language, you can access Pilosa via the Pilosa API. -#### Do you query Pilosa using SQL? +### Do you query Pilosa using SQL? One can access Pilosa directly via the terminal using the Pilosa Query Language (PQL), but a typical implementation would use one of the Pilosa client libraries to integrate with an existing codebase. There is currently client support for Go, Python, and Java. -#### Replication on each node? +### Replication on each node? Pilosa supports a replication factor greater than or equal to one. When replication is configured to be greater than one, then all mutations will be replicated to additional nodes in the cluster. For example, in a five-node cluster consisting of nodes A-B-C-D-E and with replication factor of three, then a write to node B will result in data being written to nodes B, C, and D. If the replication factor is greater than the number of nodes in the cluster, the data will be replicated to every node in the cluster only once. diff --git a/docs/getting-started.md b/docs/getting-started.md index 108c12737..96fd4e9e7 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -7,11 +7,11 @@ title = "Getting Started" Pilosa supports an HTTP interface which uses JSON by default. Any HTTP tool can be used to interact with the Pilosa server. The examples in this documentation will use [curl](https://curl.haxx.se/) which is available by default on many UNIX-like systems including Linux and MacOS. Windows users can download curl [here](https://curl.haxx.se/download.html). -
+

Note that Pilosa server requires a high limit for open files. Check the documentation of your system to see how to increase it in case you hit that limit.

-#### Starting Pilosa +### Starting Pilosa Follow the steps in the [Install]({{< ref "installation.md" >}}) document to install Pilosa. Execute the following in a terminal to run Pilosa with the default configuration (Pilosa will be available at `localhost:10101`): @@ -30,13 +30,13 @@ curl localhost:10101/status Which should output: `{"status":{"Nodes":[{"Host":":10101","State":"UP"}]}}` -#### Sample Project +### Sample Project In order to better understand Pilosa's capabilities, we will create a sample project called "Star Trace" containing information about the top 1,000 most recently updated Github repositories which have "go" in their name. The Star Trace index will include data points such as programming language, tags, and stargazers—people who have starred a project. Although Pilosa doesn't keep the data in a tabular format, we still use the terms "columns" and "rows" when describing the data model. We put the primary objects in columns, and the properties of those objects in rows. For example, the Star Trace project will contain an index called "repository" which contains columns representing Github repositories, and rows representing properties like programming languages and tags. We can better organize the rows by grouping them into sets called Frames. So the "repository" index might have a "languages" frame as well as a "tags" frame. You can learn more about indexes and frames in the [Data Model](../data-model) section of the documentation. -##### Create the Schema +#### Create the Schema Note: The queries in this section which are used to set up the indexes in Pilosa just the empty object on success: `{}` - if you would like to verify that a query worked as you expected, you can request the schema as follows: @@ -74,7 +74,7 @@ curl localhost:10101/index/repository/frame/language \ -d '{"options": {"rowLabel": "language_id", "inverseEnabled": true}}' ``` -##### Import Some Data +#### Import Some Data The sample data for the "Star Trace" project is at [Pilosa Getting Started repository](https://github.com/pilosa/getting-started). Download the `stargazer.csv` and `language.csv` files in that repo. @@ -100,9 +100,9 @@ docker exec -it pilosa /pilosa import -i repository -f language /language.csv Note that, both the user IDs and the repository IDs were remapped to sequential integers in the data files, they don't correspond to actual Github IDs anymore. You can check out `language.txt` to see the mapping for languages. -##### Make Some Queries +#### Make Some Queries -
+

Note the Pilosa server comes with a WebUI for constructing queries in a browser. In local development, it is available at localhost:10101.

@@ -148,6 +148,6 @@ curl localhost:10101/index/repository/query \ -d 'SetBit(frame="stargazer", repo_id=77777, stargazer_id=99999)' ``` -#### What's Next? +### What's Next? You can jump to [Data Model](../data-model/) for an in-depth look at Pilosa's data model, or [Query Language](../query-language/) for more details about **PQL**, the query language of Pilosa. Check out the [Tutorials](../tutorials/) for example implementations of real world use cases for Pilosa. Ready to get going in your favorite language? Have a peek at our small but expanding set of official [Client Libraries](../client-libraries/). diff --git a/docs/glossary.md b/docs/glossary.md index 650ac8ddf..e4bef02e3 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -2,8 +2,7 @@ title = "Glossary" +++ -# Glossary - +## Glossary Index: Indexes are the top level container in Pilosa - similar to a database in an RDBMS. Queries cannot operate across multiple indexes. diff --git a/docs/installation.md b/docs/installation.md index b65fb5338..c97ed4303 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -7,11 +7,11 @@ title = "Installation" Pilosa is currently available for [MacOS](#installing-on-macos) and [Linux](#installing-on-linux). -#### Installing on MacOS +### Installing on MacOS There are three ways to install Pilosa on MacOS: download the binary (recommended), build from source, or use Docker. -##### Download the Binary +#### Download the Binary 1. Download the latest release: ``` @@ -71,7 +71,7 @@ There are three ways to install Pilosa on MacOS: download the binary (recommende You're good to go! -##### Build from Source +#### Build from Source 1. Install the prerequisites: @@ -131,7 +131,7 @@ There are three ways to install Pilosa on MacOS: download the binary (recommende You're good to go! -##### Use Docker +#### Use Docker 1. Install Docker for Mac. @@ -152,16 +152,16 @@ If you don't see the server listed, start the Docker application. docker run --rm pilosa/pilosa:latest help ``` -##### What's next? +#### What's next? Head over to the [Getting Started](../getting-started/) guide to create your first Pilosa index. -#### Installing on Linux +### Installing on Linux There are three ways to install Pilosa on Linux: download the binary (recommended), build from source, or use Docker. -##### Download the Binary +#### Download the Binary 1. To install the latest version of Pilosa, download the latest release: ``` @@ -221,7 +221,7 @@ There are three ways to install Pilosa on Linux: download the binary (recommende You're good to go! -##### Build from Source +#### Build from Source 1. Install the prerequisites: @@ -282,7 +282,7 @@ There are three ways to install Pilosa on Linux: download the binary (recommende You're good to go! -##### Use Docker +#### Use Docker 1. Install Docker. @@ -303,17 +303,17 @@ There are three ways to install Pilosa on Linux: download the binary (recommende docker run --rm pilosa/pilosa:latest help ``` -##### What's next? +#### What's next? Head over to the [Getting Started](../getting-started/) guide to create your first Pilosa index. diff --git a/docs/tutorials.md b/docs/tutorials.md index d5ec25433..5b36bd62b 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -4,9 +4,9 @@ title = "Tutorials" ## Tutorials -#### Transportation +### Transportation -##### Introduction +#### Introduction New York City released an extremely detailed data set of over 1 billion taxi rides taken in the city - this data has become a popular target for analysis by tech bloggers and has been very well studied. For this reason, we thought it would be interesting to import this data to Pilosa in order to compare with other data stores and techniques on the exact same data set. @@ -16,7 +16,7 @@ We've written a tool to help import the NYC taxi data into Pilosa - this tool is After initial setup, the PDK import tool does everything we need to define a Pilosa schema, map data to bitmaps accordingly, and import it into Pilosa. -##### Data Model +#### Data Model The NYC taxi data is comprised of a number of csv files listed here: http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml. These data files have around 20 columns, about half of which are relevant to the benchmark queries we're looking at: @@ -54,15 +54,15 @@ frame |mapping duration_minutes |round(drop_timestamp - pickup_timestamp) → row ID speed_mph |round(dist_miles / (drop_timestamp - pickup_timestamp)) → row ID -##### Mapping +#### Mapping Each column that we want to use must be mapped to a combination of frames and row IDs according to some rule. There are many ways to approach this mapping, and the taxi dataset gives us a good overview of possibilities. -###### 0 columns → 1 frame +##### 0 columns → 1 frame cab_type: contains one row for each type of cab. Each column, representing one ride, has a bit set in exactly one row of this frame. The mapping is a simple enumeration, for example yellow=0, green=1, etc. The values of the bits in this frame are determined by the source of the data. That is, we're importing data from several disparate sources: NYC yellow taxi cabs, NYC green taxi cabs, and Uber cars. For each source, the single row to be set in the cab_type frame is constant. -###### 1 column → 1 frame +##### 1 column → 1 frame The following three frames are mapped in a simple direct way from single columns of the original data. @@ -124,7 +124,7 @@ Here, we define a list of Mappers, each including a name, which we use to refer **passenger_count:** This column contains small integers, so we use one of the simplest possible mappings: the column value is the row ID. -###### 1 column → multiple frames +##### 1 column → multiple frames When working with a composite data type like a timestamp, there are plenty of mapping options. In this case, we expect to see interesting periodic trends, so we want to encode the cyclic components of time in a way that allows us to look at them independently during analysis. @@ -134,7 +134,7 @@ We might continue this pattern with hours, minutes, and seconds, but we don't ha We do all of this for each timestamp of interest, one for pickup time and one for dropoff time. That gives us eight total frames for two timestamps: pickup_year, pickup_month, pickup_day, pickup_time, drop_year, drop_month, drop_day, drop_time. -###### Multiple columns → 1 frame +##### Multiple columns → 1 frame The ride data also contains geolocation data: latitude and longitude for both pickup and dropoff. We just want to be able to produce a rough overview heatmap of ride locations, so we use a grid mapping. We divide the area of interest into a 100x100 grid in latitude-longitude space, label each cell in this grid with a single integer, and use that integer as the row ID. @@ -142,7 +142,7 @@ We do all of this for each location of interest, one for pickup and one for drop Again, there are many mapping options for location data. For example, we might convert to a different coordinate system, apply a projection, or aggregate locations into real-world regions such as neighborhoods. Here, the simple approach is sufficient. -###### Complex mappings +##### Complex mappings We also anticipate looking for trends in ride duration and speed, so we want to capture this information during the import process. For the frame `duration_minutes`, we compute a row ID as `round((drop_timestamp - pickup_timestamp).minutes)`. For the frame `speed_mph`, we compute row ID as `round(dist_miles / (drop_timestamp - pickup_timestamp).minutes)`. These mapping calculations are straightforward, but because they require arithmetic operations on multiple columns, they are a bit too complex to capture in the basic mappers available in PDK. Instead, we define custom mappers to do the work: ```go @@ -156,11 +156,11 @@ durm := pdk.CustomMapper{ } ``` -##### Import process +#### Import process After designing this schema and mapping, we capture it in a JSON definition file that can be read by the PDK import tool. Running `pdk taxi` runs the import based on the information in this file. See [PDK](../pdk) for more details on this process. -##### Queries +#### Queries Now we can run some example queries. @@ -194,9 +194,9 @@ for pcount, topn in zip(pcounts, resp.json()['results']): For more examples and details, see this [ipython notebook](https://github.com/alanbernstein/pilosa-notebooks/blob/master/taxi-use-case.ipynb). -#### Chemical similarity search +### Chemical similarity search -##### Overview +#### Overview The notion of chemical similarity (or molecular similarity) plays an important role in predicting the properties of chemical compounds, designing chemicals with a predefined set of properties, and—especially—conducting drug design studies. All of these are accomplished by screening large indexes containing structures of available or potentially available chemicals. @@ -211,7 +211,7 @@ A and B are sets of fingerprint bits on in the fingerprints of molecule A and mo All source code to calculate tanimoto for molecule fingerprint using Pilosa is available in a Github repository https://github.com/pilosa/chem-usecase -##### Data model +#### Data model We use the latest ChEMBL release chembl_22.sdf for test data. Each molecule in the SD file gives us the canonical isomeric SMILES (Simplified molecular-input line-entry system) and chembl_id. @@ -248,7 +248,7 @@ Index: mole After retrieving chembl_id from the Inverse View, we can use the Tanimoto coefficient to compare chembl_id with the entire data set of molecules. The result of this comparison is the list of `chembl_id`s that have a Tanimoto coefficient greater than the given threshold. -##### Import process +#### Import process To import data into Pilosa, we need to get chembl_id and SMILES from SD files, convert SMILES to Morgan fingerprints, and then write chembl_id and fingerprint to Pilosa. The fastest way is to extracted chembl_id and SMILES from SD file to csv file, then use the `pilosa import` command to import the csv file into Pilosa. Since chembl_id in the SD file is always paired with CHEMBL, e.g CHEMBL6329, and because Pilosa doesn't support string keys, we will ignore CHEMBL and instead use chembl_id as an integer key. @@ -276,7 +276,7 @@ Run the following commands to import the csv data into the `mole` index: pilosa import -d mole -f fingerprint id_fingerprint.csv ``` -##### Queries +#### Queries Get chembl_id from a given SMILES: ``` @@ -330,7 +330,7 @@ Return chembl_id = [6223, 269758, 6206, 6228]. This script uses Pilosa’s TopN topn = requests.post("http://127.0.0.1:10101/index/mol/query" , data=query_string) ``` -##### Benchmark +#### Benchmark To run benchmark for specific chembl_id for different similarity threshold at percentage of [50, 70, 75, 80, 85, 90], run following command: ``` diff --git a/docs/webui.md b/docs/webui.md index 191ab964e..8084cbb80 100644 --- a/docs/webui.md +++ b/docs/webui.md @@ -7,7 +7,7 @@ title = "WebUI" The Pilosa server comes packaged with in-browser WebUI. When you run a local Pilosa server on the default host, you can access it at [localhost:10101](http://localhost:10101) This can be used for constructing queries and viewing the cluster status. -#### Console +### Console The [Console view](http://localhost:10101/#console) allows you to enter [PQL](../query-language) queries and run them against your locally running server. First you must select an Index with the Select index dropdown. @@ -31,6 +31,6 @@ Index and frame creation also supports options like `columnLabel`,`rowLabel` or - `:create frame rowLabel=row_id inverseEnabled=true cacheSize=10000` -#### Cluster Admin +### Cluster Admin Use the [Cluster Admin tab](http://localhost:10101/#admin) to view the current status of your cluster. This contains information on each node in the cluster, plus the list of Indexes and Frames.