scriptoriumadmin/featurebase
1
0
Fork 0
mirror of https://github.com/featurebasedb/featurebase.git synced 2026-08-28 10:54:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Minor grammar and formatting fixes

Browse source
This commit is contained in:
Alan Bernstein 2018-05-14 17:54:02 -05:00
parent b670afaa8d
commit 6e1ddf7400
5 changed files with 36 additions and 34 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

46
docs/administration.md
View file

@ -20,7 +20,7 @@ Pilosa is a standalone, compiled Go application, so there is no need to worry ab
#### Memory
Pilosa holds all row/column bitmap data in main memory. While this data is compressed more than a typical database, available memory is a primary concern. In a production environment, we recommend choosing hardware with a large amount of memory >= 64GB. Prefer a small number of hosts with lots of memory per host over a larger number with less memory each. Larger clusters tend to be less efficient overall due to increased inter-node communication.
Pilosa holds all row/column bitmap data in main memory. While this data is compressed more than a typical database, available memory is a primary concern. In a production environment, we recommend choosing hardware with a large amount of memory >= 64GB. Prefer a small number of hosts with lots of memory per host over a larger number with less memory each. Larger clusters tend to be less efficient overall due to increased inter-node communication.
#### CPUs
@ -28,15 +28,15 @@ Pilosa is a concurrent application written in Go and can take full advantage of
#### Disk
Even though the main dataset is in memory Pilosa does back up to disk frequently. We recommend SSDs--especially if you have a write heavy application.
Even though the main dataset is in memory Pilosa does back up to disk frequently. We recommend SSDs--especially if you have a write heavy application.
#### Network
Pilosa is designed to be a distributed application, with data replication shared across the cluster. As such every write and read needs to communicate with several nodes. Therefore fast internode communication is essential. If using a service like AWS we recommend that all node exist in the same region and availability zone. The inherent latency of spreading a Pilosa cluster across physical regions it not usually worth the redundancy protection. Since Pilosa is designed to be an Indexing service there already should be a system of record, or ability to rebuild a Cluster quickly from backups.
Pilosa is designed to be a distributed application, with data replication shared across the cluster. As such every write and read needs to communicate with several nodes. Therefore fast internode communication is essential. If using a service like AWS we recommend that all node exist in the same region and availability zone. The inherent latency of spreading a Pilosa cluster across physical regions it not usually worth the redundancy protection. Since Pilosa is designed to be an indexing service there already should be a system of record, or ability to rebuild a cluster quickly from backups.
#### Overview
While Pilosa does have some high system requirements it is not a best practice to set up a cluster with the fewest, largest machines available. You want an evenly distributed load across several nodes in a cluster to easily recover from a single node failure, and have the resource capacity to handle a missing node until it's repaired or replaced. Nor is it advisable to have many small machines. The internode network traffic will become a bottleneck. You can always add nodes later, but that does require some down time.
While Pilosa does have some high system requirements it is not a best practice to set up a cluster with the fewest, largest machines available. You want an evenly distributed load across several nodes in a cluster to easily recover from a single node failure, and have the resource capacity to handle a missing node until it's repaired or replaced. Nor is it advisable to have many small machines. The internode network traffic will become a bottleneck. You can always add nodes later, but that does require some down time.
### Open File Limits
@ -48,7 +48,7 @@ On Mac OS X, `ulimit` does not behave predictably. [This blog post](https://blog
#### Importing
The import API expects a csv of rowID,columnID's.
The import API expects a csv of the format `Row,Column`.
When importing large datasets remember it is much faster to pre sort the data by row ID and then by column ID in ascending order. You can use the `--sort` flag to do that. Also, avoid querying Pilosa until the import is complete, otherwise you will experience inconsistent results.
@ -58,7 +58,7 @@ pilosa import --sort -i project -f stargazer project-stargazer.csv
##### Importing Field Values
If you are using [BSI Range-Encoding](../data-model/#bsi-range-encoding) field values, you can import field values for a single frame and single field using `--field`. The CSV file should be in the format `ColumnID,Value`.
If you are using [BSI Range-Encoding](../data-model/#bsi-range-encoding) field values, you can import field values for a single frame and single field using `--field`. The CSV file should be in the format `Column,Value`.
```
pilosa import -i project -f stargazer --field star_count project-stargazer-counts.csv
@ -195,9 +195,9 @@ curl localhost:10101/cluster/resize/set-coordinator \
### Backup/restore
Pilosa continuously writes out the in-memory bitmap data to disk. This data is organized by Index->Frame->Views->Fragment->numbered slice files. These data files can be routinely backed up to restore nodes in a cluster.
Pilosa continuously writes out the in-memory bitmap data to disk. This data is organized by Index->Frame->Views->Fragment->numbered slice files. These data files can be routinely backed up to restore nodes in a cluster.
Depending on the size of your data you have two options. For a small dataset you can rely on the periodic anti-entropy sync process to replicate existing data back to this node.
Depending on the size of your data you have two options. For a small dataset you can rely on the periodic anti-entropy sync process to replicate existing data back to this node.
For larger datasets and to make this process faster you could copy the relevant data files from the other nodes to the new one before startup.
@ -230,19 +230,19 @@ Each Pilosa cluster is configured by default to share anonymous usage details wi
- **Version:** Version string of the build.
- **Host:** Host URI.
- **Cluster:** List of nodes in the Cluster.
- **NumNodes:** Number of nodes in the Cluster.
- **NumCPU:** Number of Cores per Node
- **Cluster:** List of nodes in the cluster.
- **NumNodes:** Number of nodes in the cluster.
- **NumCPU:** Number of cores per node
- **BSIEnabled:** Bit Slice Index Frames in use.
- **TimeQuantumEnabled:** Time Quantum Frames in use.
- **NumIndexes:** Number of Indexes in the Cluster.
- **NumFrames:** Number of Frames in the Cluster.
- **NumSlices:** Number of Slices in the Cluster.
- **NumViews:** Number of Views in the Cluster.
- **NumIndexes:** Number of indexes in the Cluster.
- **NumFrames:** Number of frames in the Cluster.
- **NumSlices:** Number of slices in the Cluster.
- **NumViews:** Number of views in the Cluster.
- **OpenFiles:** Open file handle count.
- **GoRoutines:** Go routine count.
You can opt-out of the Pilosa diagnostics reporting by setting either the command line configuration option `--metric.diagnostics=false`, use the `PILOSA_METRIC_DIAGNOSTICS` environment variable, or the TOML configuration file `[metric]` `diagnostics` option.
You can opt-out of the Pilosa diagnostics reporting by setting the command line configuration option `--metric.diagnostics=false`, the `PILOSA_METRIC_DIAGNOSTICS` environment variable, or the TOML configuration file `[metric]` `diagnostics` option.
### Metrics
@ -265,14 +265,14 @@ StatsD Tags adhere to the DataDog format (key:value), and we tag the following:
#### Events
We currently track the following events
- **Index:** The creation of a new Index.
- **Frame:** The creation of a new Frame.
- **MaxSlice:** The Creation of a new Slice.
- **Index:** The creation of a new index.
- **Frame:** The creation of a new frame.
- **MaxSlice:** The creation of a new Slice.
- **SetBit:** Count of set bits.
- **ClearBit:** Count of cleared bits.
- **ImportBit:** During a bulk data import this represents the count of bits created.
- **SetRowAttrs:** Count of Attributes set per row.
- **SetColumnAttrs:** Count of Attributes set per column.
- **SetRowAttrs:** Count of attributes set per row.
- **SetColumnAttrs:** Count of attributes set per column.
- **Bitmap:** Count of Bitmap queries.
- **TopN:** Count of TopN queries.
- **Union:** Count of Union queries.
@ -282,6 +282,6 @@ We currently track the following events
- **Range:** Count of Range queries.
- **Snapshot:** Event count when the snapshot process is triggered.
- **BlockRepair:** Count of data blocks that were out of sync and repaired.
- **Garbage Collection:** Event count when Garbage Collection occurs.
- **Goroutines:** Number of running Goroutines.
- **GarbageCollection:** Event count when garbage collection occurs.
- **Goroutines:** Number of running goroutines.
- **OpenFiles:** Number of open file handles associated with running Pilosa process ID.

10
docs/configuration.md
View file

@ -11,7 +11,7 @@ nav = [
## Configuration
Pilosa can be configured through command line flags, environment variables, and/or a configuration file; configured options take precedence in that order. So if an option is specified in a command line flag, it will take precedence over the same option specified in the environment, which would take precedence over that same option specified in the configuration file.
Pilosa can be configured through command line flags, environment variables, and/or a configuration file; configured options take precedence in that order. So if an option is specified in a command line flag, it will take precedence over the same option specified in the environment, which will take precedence over that same option specified in the configuration file.
All options are available in all three configuration types with the exception of the `--config` option which specifies the location of the config file, and therefore will not be used if it is present in the config file.
@ -118,18 +118,18 @@ The config file is in the [toml format](https://github.com/toml-lang/toml) and h
#### Gossip Seeds
* Description: This specifies which internal host(s) should be used to initialize membership in the cluster. Typically this can be the address of any available host in the cluster. For example, when starting a three-node cluster made up of `node0`, `node1`, and `node2`, the `gossip.seeds` for all three nodes can be configured to be the address of `node0`. Multiple seeds should be comma-separated in the flag and env forms.
* Flag: `--gossip.seeds="localhost:11101"`
* Env: `PILOSA_GOSSIP_SEEDS="localhost:11101"`
* Flag: `--gossip.seeds="localhost:11101,localhost:11110"`
* Env: `PILOSA_GOSSIP_SEEDS="localhost:11101,localhost:11110"`
* Config:
```toml
[gossip]
seeds = ["localhost:11101"]
seeds = ["localhost:11101", "localhost:11110"]
```
#### Gossip Key
* Description: Path to the file which contains the key to encrypt gossip communication. The contents of the file should be either 16, 24, or 32 bytes to select AES-128, AES-192, or AES-256 encryption. You can read from `/dev/random` device on UNIX-like systems to create the key file; e.g., `head -c 32 /dev/random > gossip.key32` creates a key file to use AES-256.
* Description: Path to the file which contains the key to encrypt gossip communication. The contents of the file should be either 16, 24, or 32 bytes to select AES-128, AES-192, or AES-256 encryption. You can read from `/dev/random` device on UNIX-like systems to create the key file; e.g., `head -c 32 /dev/random > gossip.key32` creates a key file to use AES-256.
* Flag: `--gossip.key="/var/secret/gossip.key32"`
* Env: `PILOSA_GOSSIP_KEY="/var/secret/gossip.key32"`
* Config:

6
docs/faq.md
View file

@ -16,8 +16,10 @@ Pilosa is not a database in the traditional sense. While Pilosa does store data
### Where does Pilosa fit in my stack?
Pilosa sits on top of a data store or multiple data stores.
How is Pilosa different than Elasticsearch since they are both indexes?
Pilosa sits on top of a data store or multiple data stores.
### How is Pilosa different than Elasticsearch since they are both indexes?
Elasticsearch is a search engine based on Lucene, and is therefore very good at indexing and searching large volumes of unstructured text. As it matures, Elasticsearch has continued to move into the analytics space, but its core data object is still the "document". Pilosa is specifically designed to index structured data and improve query speed. By representing data as the relationship between objects, and then storing those relationships in bitmaps, Pilosa can very efficiently search and compare many millions of data points while still maintaining a small memory footprint.
### How do I get my data into Pilosa?

4
docs/pdk.md
View file

@ -56,11 +56,11 @@ are created in the default frame. Frames are a dash-separated concatenation of
all key values in the path - you can see this with frames like location-city.
Most the options to `pdk kafka` are self-explanatory (kafka hosts, pilosa hosts,
Most of the options to `pdk kafka` are self-explanatory (kafka hosts, pilosa hosts,
kafka topics, kafka group, etc.), but there are a few options that give some
control over the way data is indexed, and ingestion performance.
* `--batch-size`: The batch size control how many set bits or values are batched up to be imported *per frame*. So for fields that have one value per record, you have to wait for `batch-size` records to come through before you'll see the data indexed in Pilosa. Fields like `favorite_foods` which can have multiple values could be indexed sooner.
* `--batch-size`: The batch size controls how many set bits or values are batched up to be imported *per frame*. So for fields that have one value per record, you have to wait for `batch-size` records to come through before you'll see the data indexed in Pilosa. Fields like `favorite_foods` which can have multiple values could be indexed sooner.
* `--framer.collapse`: This is a list of strings which will be removed from the frame names created by dash-concatentating all names in the JSON path to a value. E.G. if "location" were listed in `framer.collapse`, then there would be frames named "city" and "state" rather than "location-city" and "location-state".
* `--framer.ignore`: This allows you to skip indexing on any path containing these strings. If you have a field like email address or some other unique ID, you might not want to index it.
* `--subject-path`: If nothing is passed for this option, then each record will be assigned a unique sequential column ID. If `subject-path` is specified, then the value at this path in the record will be mapped to a column ID. If the same value appears in another record, the same column ID will be used.

4
docs/webui.md
View file

@ -15,7 +15,7 @@ This can be used for constructing queries and viewing the cluster status.
### Console
The [Console view](http://localhost:10101/#console) allows you to enter [PQL](../query-language/) queries and run them against your locally running server. First you must select an Index with the Select index dropdown.
The [Console view](http://localhost:10101/#console) allows you to enter [PQL](../query-language/) queries and run them against your locally running server. First you must select an Index with the Select index dropdown.
Each query's result will be displayed in the Output section along with the query time.
@ -39,4 +39,4 @@ Frame creation also supports options like `timeQuantum`. When creating a new fra
### Cluster Admin
Use the [Cluster Admin tab](http://localhost:10101/#admin) to view the current status of your cluster. This contains information on each node in the cluster, plus the list of Indexes and Frames.
Use the [Cluster Admin tab](http://localhost:10101/#admin) to view the current status of your cluster. This contains information on each node in the cluster, plus the list of Indexes and Frames.