diff --git a/docs/api-reference.md b/docs/api-reference.md index c3a0144df..8975655d9 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -17,7 +17,7 @@ Returns the schema of all indexes in JSON. curl -XGET localhost:10101/index ``` ``` response -{"indexes":[{"name":"user","fields":[{"name":"collab"}]}]} +{"indexes":[{"name":"user","fields":[{"name":"event","options":{"type":"time","timeQuantum":"YMD","keys":false}}]}]} ``` ### List index schema @@ -30,7 +30,7 @@ Returns the schema of the specified index in JSON. curl -XGET localhost:10101/index/user ``` ``` response -{"name":"user", "fields":[{"name":"collab"}]} +{"name":"user","fields":[{"name":"event","options":{"type":"time","timeQuantum":"YMD","keys":false}}]} ``` ### Create index @@ -100,19 +100,27 @@ By default, all bits and attributes (*for `Row` queries only*) are returned. In Creates a field in the given index with the given name. -The request payload is in JSON, and may contain the `options` field. The `options` field is a JSON object which may contain the following fields: +The request payload is in JSON, and may contain the `options` field. The `options` field is a JSON object which must contain a `type` along with the corresponding configuration options. -* `timeQuantum` (string): [Time Quantum](../data-model/#time-quantum) for this field. -* `cacheType` (string): [ranked](../data-model/#ranked) or [LRU](../data-model/#lru) caching on this field. Default is `lru`. -* `cacheSize` (int): Number of rows to keep in the cache. Default 50,000. -* `fields` (array): List of range-encoded [fields](../data-model/#bsi-range-encoding). +* `set` + * `cacheType` (string): [ranked](../data-model/#ranked) or [LRU](../data-model/#lru) caching on this field. Default is `lru`. + * `cacheSize` (int): Number of rows to keep in the cache. Default 50,000. +* `int` + * `min` (int): Minimum integer value allowed for the field. + * `max` (int): Maximum integer value allowed for the field. +* `time` + * `timeQuantum` (string): [Time Quantum](../data-model/#time-quantum) for this field. -Each individual `field` contains the following: +The following example creates an `int` field called "quantity" capable of storing values from -1000 to 2000: -* `name` (string): Field name. -* `type` (string): Field type, "set", "int" or "time". -* `min` (int): Minimum value allowed for this field. -* `max` (int): Maximum value allowed for this field. +``` request +curl localhost:10101/index/user/field/quantity \ + -X POST \ + -d '{"options": {"type": "int", "min": -1000, "max":2000}}' +``` +``` response +{"success":true} +``` Integer fields are stored as n-bit range-encoded values. Pilosa supports 63-bit, signed integers with values between `min` and `max`. diff --git a/docs/data-model.md b/docs/data-model.md index e453d9245..4609032fc 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -18,11 +18,11 @@ nav = [ ### Overview -The central component of Pilosa's data model is a boolean matrix. Each cell in the matrix is a single bit - if the bit is set, it indicates that a relationship exists between that particular row and column. +The central component of Pilosa's data model is a boolean matrix. Each cell in the matrix is a single bit; if the bit is set, it indicates that a relationship exists between that particular row and column. -Rows and columns can represent anything (they could even represent the same set of things - a [bigraph](https://en.wikipedia.org/wiki/Bigraph)). Pilosa can associate arbitrary key/value pairs (referred to as attributes) to rows and columns, but queries and storage are optimized around the core matrix. +Rows and columns can represent anything (they could even represent the same set of things as in a [bigraph](https://en.wikipedia.org/wiki/Bigraph)). Pilosa can associate arbitrary key/value pairs (referred to as attributes) to rows and columns, but queries and storage are optimized around the core matrix. -Pilosa lays out data first in rows, so queries which get all the set bits in one or many rows, or compute a combining operation on multiple rows such as Intersect or Union are the fastest. Pilosa categorizes rows into different *fields* and quickly retrieves the top rows in a field sorted by the number of columns set in each row. +Pilosa lays out data first in rows, so queries which get all the set bits in one or many rows, or compute a combining operation—such as Intersect or Union—on multiple rows, are the fastest. Pilosa categorizes rows into different *fields* and quickly retrieves the top rows in a field sorted by the number of columns set in each row. Please note that Pilosa is most performant when row and column IDs are sequential starting from 0. You can deviate from this to some degree, but setting a bit with column ID 263 on a single-node cluster, for example, will not work well due to memory limitations. @@ -35,11 +35,11 @@ The purpose of the Index is to represent a data namespace. You cannot perform cr ### Column -Column ids are sequential increasing integers and are common to all Fields within an Index. A single column often corresponds to a record in a relational table, although other configurations are possible, and sometimes preferable. +Column ids are sequential, increasing integers and they are common to all Fields within an Index. A single column often corresponds to a record in a relational table, although other configurations are possible, and sometimes preferable. ### Row -Row ids are sequential increasing integers namespaced to each Field within an Index. +Row ids are sequential, increasing integers namespaced to each Field within an Index. ### Field @@ -49,8 +49,6 @@ Fields are used to segment rows within an index, for example to define different The Pilosa index is a flexible structure; it can represent any sort of high-cardinality binary matrix. We have explored a number of modeling patterns in Pilosa use cases; one accessible example is a direct analogy to the relational model, summarized here. -TODO diagram showing a few rows of a relational table and corresponding pilosa index - Entities: Relational | Pilosa @@ -103,7 +101,7 @@ The LRU cache maintains the most recently accessed Rows. ### Time Quantum -Setting a time quantum on a field creates extra views which allow Range queries down to the time interval specified. For example - if the time quantum is set to `YMD`, Range queries down to the granularity of a day are supported. +Setting a time quantum on a field creates extra views which allow Range queries down to the time interval specified. For example, if the time quantum is set to `YMD`, Range queries down to the granularity of a day are supported. ### Attribute @@ -117,27 +115,36 @@ Indexes are segmented into groups of columns called shards (previously known as Query operations run in parallel, and they are evenly distributed across a cluster via a consistent hash algorithm. -### View +### Field Type -Views represent the various data layouts within a Field. The primary View is called Standard, and it contains the typical Row and Column data. Time-based Views are automatically generated for each time quantum. Views are internally managed by Pilosa, and never exposed directly via the API. +Upon creation, fields are configured to be of a certain type. Pilosa supports three field types: `set`, `int`, and `time`. -#### Standard +#### Set -The standard View contains the same Row/Column format as the input data. - -#### Time Quantums - -If a Field has a time quantum, then Views are generated for each of the defined time segments. For example, for a field with a time quantum of `YMD`, the following `Set()` queries will result in the data described in the diagram below: +Set is the default field type in Pilosa. Set fields represent a standard, binary matrix of rows and columns where each row key represents a possible field value. The following example creates a `set` field called "info" with a ranked cache containing up to 100,000 records. +``` request +curl localhost:10101/index/repository/field/info \ + -X POST \ + -d '{"options": {"type": "set", "cacheType": "ranked", "cacheSize":100000}}' ``` -Set(3, A=8, 2017-05-18T00:00) -Set(3, A=8, 2017-05-19T00:00) +``` response +{"success":true} ``` -![time quantum field diagram](/img/docs/field-time-quantum.svg) -*Time quantum fueld diagram* +#### Int +Fields of type `int` are used to store integer values. Integer fields share the same columns as the other fields in the index, but values for the field must be integers that fall between the `min` and `max` values specified when creating the field. The following example creates an `int` field called "quantity" capable of storing values from -1000 to 2000: -#### BSI Range-Encoding +``` request +curl localhost:10101/index/repository/field/quantity \ + -X POST \ + -d '{"options": {"type": "int", "min": -1000, "max":2000}}' +``` +``` response +{"success":true} +``` + +##### BSI Range-Encoding Bit-Sliced Indexing (BSI) is the storage method Pilosa uses to represent multi-bit integers in a bitmap index. Integers are stored as n-bit, range-encoded bit-sliced indexes of base-2, along with an additional row indicating "not null". This means that a 16-bit integer will require 17 rows: one for each 0-bit of the 16 bit-slice components (the 1-bit does not need to be stored because with range-encoding the highest bit position is always 1) and one for the non-null row. Pilosa can evaluate `Range`, `Min`, `Max`, and `Sum` queries on these BSI integers. The result of a `Sum` query includes a count, which can be used to compute an average with no other overhead. @@ -158,3 +165,26 @@ Set(3, B=6) *BSI field diagram* Check out this [blog post](/blog/range-encoded-bitmaps/) for some more details about BSI in Pilosa. + +#### Time + +Time fields are similar to `set` fields, but in addition to row and column information, they also store a per-bit time value down to a defined granularity. The following example creates a `time` field called "event" which stores timestamp information down to a day granularity. + +``` request +curl localhost:10101/index/repository/field/event \ + -X POST \ + -d '{"options": {"type": "time", "timeQuantum": "YMD"}}' +``` +``` response +{"success":true} +``` + +With `time` fields, data views are generated for each of the defined time segments. For example, for a field with a time quantum of `YMD`, the following `Set()` queries will result in the data described in the diagram below: + +``` +Set(3, A=8, 2017-05-18T00:00) +Set(3, A=8, 2017-05-19T00:00) +``` + +![time quantum field diagram](/img/docs/field-time-quantum.svg) +*Time quantum fueld diagram* diff --git a/docs/getting-started.md b/docs/getting-started.md index 6f935f735..563176c62 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -19,7 +19,7 @@ Any HTTP tool can be used to interact with the Pilosa server. The examples in th ### Starting Pilosa -Follow the steps in the [Install](../installation/) document to install Pilosa. +Follow the steps in the [Installation](../installation/) document to install Pilosa. Execute the following in a terminal to run Pilosa with the default configuration (Pilosa will be available at [localhost:10101](http://localhost:10101)): ``` pilosa server diff --git a/docs/glossary.md b/docs/glossary.md index 81323674d..550b9017b 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -14,7 +14,7 @@ nav = [] [Bitmap](../data-model/#overview): The on-disk and in-memory representation of a [row](#row). Implemented with [Roaring](#roaring-bitmap). -[BSI](../data-model/#bsi-range-encoding) Bit-sliced indexing is the method Pilosa uses to represent multi-bit integers. Integer values are stored in [fields](#field), and can be used for [Range](#range-bsi), [Min](#min), [Max](#max), and [Sum](#sum) queries. +[BSI](../data-model/#bsi-range-encoding) Bit-sliced indexing is the method Pilosa uses to represent multi-bit integers. Integer values are stored in `int` [fields](#field), and can be used for [Range](#range-bsi), [Min](#min), [Max](#max), and [Sum](#sum) queries. Cluster: A cluster consists of one or more [nodes](#node) which share a cluster configuration. The cluster also defines how data is [replicated](#replica) and how internode communication is coordinated. Pilosa does not have a leader node, all data is evenly distributed, and any node can respond to queries. diff --git a/docs/introduction.md b/docs/introduction.md index 305c9680e..fdbd8d360 100644 --- a/docs/introduction.md +++ b/docs/introduction.md @@ -8,7 +8,7 @@ nav = [] ## Introduction -Pilosa is an open source, distributed bitmap index. +Pilosa is an open source, distributed index. [//]: # (TODO insert a graphic here?) diff --git a/docs/tutorials.md b/docs/tutorials.md index e55a2094a..040d85a1c 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -25,7 +25,7 @@ Some of our tutorials work better as standalone repos, since you can git c #### Introduction -Pilosa supports encrypting the communication between and to nodes in a cluster using TLS. In this tutorial, we will be setting up a three node Pilosa cluster running on the same computer. The same steps can be used for a multi-computer cluster but that requires setting up firewalls and other platform-specific configuration which is out of the scope of this tutorial. +Pilosa supports encrypting all communication with nodes in a cluster using TLS. In this tutorial, we will be setting up a three node Pilosa cluster running on the same computer. The same steps can be used for a multi-computer cluster but that requires setting up firewalls and other platform-specific configuration which is beyond the scope of this tutorial. This tutorial assumes that you are using a UNIX-like system, such as Linux or MacOS. [Windows Subsystem for Linux (WSL)](https://msdn.microsoft.com/en-us/commandline/wsl/about) works equally well on Windows 10 systems. @@ -34,11 +34,41 @@ This tutorial assumes that you are using a UNIX-like system, such as Linux or Ma If you haven't already done so, install Pilosa server on your computer. For Linux and WSL (Windows Subsystem for Linux) use the [Installing on Linux](../installation/#installing-on-linux) instructions. For MacOS use the [Installing on MacOS](../installation/#installing-on-macos). We do not support precompiled releases for other platforms, but you can always compile it yourself from source. See [Build from Source](../installation/#build-from-source). After installing Pilosa, you may have to add it to your `$PATH`. Check that you can run Pilosa from the command line: -``` +``` request pilosa --help ``` +``` response +Pilosa is a fast index to turbocharge your database. -Let's create a directory for the tutorial to put all of our files and switch to that directory: +This binary contains Pilosa itself, as well as common +tools for administering pilosa, importing/exporting data, +backing up, and more. Complete documentation is available +at https://www.pilosa.com/docs/. + +Version: v1.0.0 +Build Time: 2018-05-14T22:14:01+0000 + +Usage: + pilosa [command] + +Available Commands: + check Do a consistency check on a pilosa data file. + config Print the current configuration. + export Export data from pilosa. + generate-config Print the default configuration. + help Help about any command + import Bulk load data into pilosa. + inspect Get stats on a pilosa data file. + server Run Pilosa. + +Flags: + -c, --config string Configuration file to read from. + -h, --help help for pilosa + +Use "pilosa [command] --help" for more information about a command. +``` + +First, create a directory in which to put all of the files for this tutorial. Then switch to that directory: ``` mkdir $HOME/pilosa-tls-tutorial && cd $_ ``` @@ -47,9 +77,9 @@ mkdir $HOME/pilosa-tls-tutorial && cd $_ Securing a Pilosa cluster consists of securing the communication between nodes using TLS and Gossip encryption. [Pilosa Enterprise](https://www.pilosa.com/enterprise/) additionally supports authentication and other security features, but those are not covered in this tutorial. -The first step is acquiring an SSL certificate. You can buy a commercial certificate or retrieve a Let's Encrypt certificate but we will be using a self signed certificate for practical reasons. Using self-signed certificates is not recommended in production, since it makes man in the middle attacks easy. +The first step is acquiring an SSL certificate. You can buy a commercial certificate or retrieve a [Let's Encrypt](https://letsencrypt.org/) certificate, but we will be using a self signed certificate for practical reasons. Using self-signed certificates is not recommended in production since it makes man-in-the-middle attacks easy. -The following command creates a 2048bit self-signed wildcard certificate for `*.pilosa.local` which expires 10 years later. +The following command creates a 2048-bit, self-signed wildcard certificate for `*.pilosa.local` which expires 10 years later. ``` openssl req -x509 -newkey rsa:2048 -keyout pilosa.local.key -out pilosa.local.crt -days 3650 -nodes -subj "/C=US/ST=Texas/L=Austin/O=Pilosa/OU=Com/CN=*.pilosa.local" @@ -60,16 +90,16 @@ The command above creates two files in the current directory: * `pilosa.local.crt` is the SSL certificate. * `pilosa.local.key` is the private key file which must be kept as secret. -Having created the SSL certificate, we can now create the gossip encryption key. Gossip encryption key file must be exactly 16, 24, or 32 bytes to select one of AES-128, AES-192, or AES-256 encryption. Reading random bytes from cryptographically secure `/dev/random` serves our purpose very well: +Having created the SSL certificate, we can now create the gossip encryption key. The gossip encryption key file must be exactly 16, 24, or 32 bytes to select one of AES-128, AES-192, or AES-256 encryption. Reading random bytes from cryptographically secure `/dev/random` serves our purpose very well: ``` head -c 32 /dev/random > pilosa.local.gossip32 ``` -We now should have `pilosa.local.gossip32` in the current directory with 32 random bytes. +We now have a file called `pilosa.local.gossip32` in the current directory which contains 32 random bytes. #### Creating the Configuration Files -Pilosa supports passing configuration items using the command line, environment variables or a configuration file. We will use the last option in this tutorial and create three configuration files for our three nodes. +Pilosa supports passing configuration items using command line options, environment variables, or a configuration file. For this tutorial, we will use three configuration files; one configuration file for each of our three nodes. One of the nodes in the cluster must be chosen as the *coordinator*. We choose the first node as the coordinator in this tutorial. The coordinator is only important during cluster resizing operations, and otherwise acts like any other node in the cluster. In the future, the coordinator will be chosen transparently by distributed consensus, and this option will be deprecated. @@ -139,7 +169,7 @@ Here is some explanation of the configuration items: * `bind` is the address to which the server listens for incoming requests. The address is composed of three parts: scheme, host, and port. The default scheme is `http` so we explicitly specify `https` to use the HTTPS protocol for communication between nodes. * `[cluster]` section contains the settings for a cluster. We set `coordinator = true` for only the first node to choose that as the coordinator node. See [Cluster Configuration](../configuration/#cluster-coordinator) for other settings. * `[tls]` section contains the TLS settings, including the path to the SSL certificate and the corresponding key. Set `skip-verify` to `true` in order to disable host name verification and other security measures. Do not set `skip-verify` to `true` on production servers. -* `[gossip]` section contains settings for the Gossip protocol. `seeds` contain the seed nodes which other nodes gather cluster topology. There must be at least one gossip seed. The `port` setting is the gossip listen address for the node. It should be different for each node, if the cluster is running on the same computer, otherwise you can set it to the same value. Finally, the `key` points to the gossip encryption key we created before. +* `[gossip]` section contains settings for the gossip protocol. `seeds` contains the list of nodes from which to seed cluster membership. There must be at least one gossip seed. The `port` setting is the gossip listen address for the node. If all nodes of the cluster are running on the same computer, the gossip listen address should be different for each node. Otherwise, it can be set to the same value. Finally, the `key` points to the gossip encryption key we created earlier. #### Final Touches Before Running the Cluster @@ -159,7 +189,7 @@ If any of the commands above return `ping: unknown host`, make sure your `/etc/h #### Running the Cluster -Let's open three terminal windows and run each node in its window. This will enable us to better observe what's happening on which node. +Let's open three terminal windows and run each node in its own window. This will enable us to better observe what's happening on each node. Switch to the first terminal window, change to the project directory and start the first node: ``` @@ -180,59 +210,72 @@ pilosa server -c node3.config.toml ``` Let's ensure that all three Pilosa servers are running and they are connected: -``` +``` request curl -k --ipv4 https://01.pilosa.local:10501/status ``` - -The `-k` flag is used to tell curl that it shouldn't bother with checking the certificate the server provides and `--ipv4` workarounds an issue on MacOS where the curl requests take a long time if the address resolves to `127.0.0.1`. You can leave it out on Linux and WSL. - -All nodes should be in the `NORMAL` state: ``` response {"state":"NORMAL","nodes":[{"id":"98ebd177-c082-4c54-8d48-7e7c75857b52","uri":{"scheme":"https","host":"02.pilosa.local","port":10502},"isCoordinator":false},{"id":"a33dc0d6-c35f-4559-984a-e582bf032a21","uri":{"scheme":"https","host":"03.pilosa.local","port":10503},"isCoordinator":false},{"id":"e24ac014-ee2f-4cb0-b565-74df6c551f0a","uri":{"scheme":"https","host":"01.pilosa.local","port":10501},"isCoordinator":true}]} ``` +The `-k` flag is used to tell curl that it shouldn't bother checking the certificate the server provides, and the `--ipv4` flag avoids an issue on MacOS where the curl request takes a long time if the address resolves to `127.0.0.1`. You can leave it out on Linux and WSL. + +If everything is set up correctly, the cluster state should be `NORMAL`. + #### Running Queries -Having confirmed that our cluster is running OK, let's run a few queries. But before that, we need to create an index and a frame: +Having confirmed that our cluster is running normally, let's perform a few queries. First, we need to create an index and a field: ``` request -curl -k --ipv4 https://01.pilosa.local:10501/index/sample-index -d '' +curl https://01.pilosa.local:10501/index/sample-index \ + -k --ipv4 \ + -X POST ``` ``` response -{} +{"success":true} ``` -This will create index `sample-index` with default options. Let's create the frame now: +This will create index `sample-index` with default options. Let's create the field now: ``` request -curl -k --ipv4 https://01.pilosa.local:10501/index/sample-index/frame/sample-frame -d '' +curl https://01.pilosa.local:10501/index/sample-index/field/sample-field \ + -k --ipv4 \ + -X POST ``` ``` response -{} +{"success":true} ``` -We just created frame `sample-frame` with default options. +We just created field `sample-field` with default options. -Let's run a `SetBit` query: +Let's run a `Set` query: ``` request -curl -k --ipv4 https://01.pilosa.local:10501/index/sample-index/query -d 'SetBit(frame="sample-frame", row=1, col=100)' +curl https://01.pilosa.local:10501/index/sample-index/query \ + -k --ipv4 \ + -X POST \ + -d 'Set(100, sample-field=1)' ``` ``` response {"results":[true]} ``` -Confirm that the bit was indeed set: +Confirm that the value was indeed set: ``` request -curl -k --ipv4 https://01.pilosa.local:10501/index/sample-index/query -d 'Bitmap(frame="sample-frame", row=1)' +curl https://01.pilosa.local:10501/index/sample-index/query \ + -k --ipv4 \ + -X POST \ + -d 'Row(sample-field=1)' ``` ``` response -{"results":[{"attrs":{},"bits":[100]}]} +{"results":[{"attrs":{},"columns":[100]}]} ``` The same response should be returned when querying other nodes in the cluster: ``` request -curl -k --ipv4 https://02.pilosa.local:10502/index/sample-index/query -d 'Bitmap(frame="sample-frame", row=1)' +curl https://02.pilosa.local:10501/index/sample-index/query \ + -k --ipv4 \ + -X POST \ + -d 'Row(sample-field=1)' ``` ``` response -{"results":[{"attrs":{},"bits":[100]}]} +{"results":[{"attrs":{},"columns":[100]}]} ``` #### What's Next? @@ -265,7 +308,7 @@ docker run -it --rm --name pilosa1 -p 10101:10101 --network=pilosanet pilosa/pil Let's run the second Pilosa node and attach it to the virtual network as well. Note that we set the address of the gossip seed to the address of the first node: ``` -docker run -it --rm --name pilosa2 --network=pilosanet pilosa/pilosa:latest server --bind pilosa2 --gossip.seeds=pilosa1:14000 +docker run -it --rm --name pilosa2 -p 10102:10101 --network=pilosanet pilosa/pilosa:latest server --bind pilosa2 --gossip.seeds=pilosa1:14000 ``` Let's test that the nodes in the cluster connected with each other: @@ -273,7 +316,7 @@ Let's test that the nodes in the cluster connected with each other: curl localhost:10101/status ``` ``` response -{"state":"NORMAL","nodes":[{"id":"2e8332d0-1fee-44dd-a359-e0d6ecbcefc1","uri":{"scheme":"http","host":"pilosa1","port":10101},"isCoordinator":true},{"id":"8c0dbcdc-9503-4265-8ad2-ba85a4bb10fa","uri":{"scheme":"http","host":"pilosa2","port":10101},"isCoordinator":false}]} +{"state":"NORMAL","nodes":[{"id":"2e8332d0-1fee-44dd-a359-e0d6ecbcefc1","uri":{"scheme":"http","host":"pilosa1","port":10101},"isCoordinator":true},{"id":"8c0dbcdc-9503-4265-8ad2-ba85a4bb10fa","uri":{"scheme":"http","host":"pilosa2","port":10101},"isCoordinator":false}],"localID":"2e8332d0-1fee-44dd-a359-e0d6ecbcefc1"} ``` And similarly for the second node: @@ -281,7 +324,7 @@ And similarly for the second node: curl localhost:10102/status ``` ``` response -{"state":"NORMAL","nodes":[{"id":"2e8332d0-1fee-44dd-a359-e0d6ecbcefc1","uri":{"scheme":"http","host":"pilosa1","port":10101},"isCoordinator":true},{"id":"8c0dbcdc-9503-4265-8ad2-ba85a4bb10fa","uri":{"scheme":"http","host":"pilosa2","port":10101},"isCoordinator":false}]} +{"state":"NORMAL","nodes":[{"id":"2e8332d0-1fee-44dd-a359-e0d6ecbcefc1","uri":{"scheme":"http","host":"pilosa1","port":10101},"isCoordinator":true},{"id":"8c0dbcdc-9503-4265-8ad2-ba85a4bb10fa","uri":{"scheme":"http","host":"pilosa2","port":10101},"isCoordinator":false}],"localID":"2e8332d0-1fee-44dd-a359-e0d6ecbcefc1"} ``` The corresponding [Docker Compose](https://docs.docker.com/compose/) file is below: @@ -304,6 +347,8 @@ services: - "pilosa1:10101" pilosa2: image: pilosa/pilosa:latest + ports: + - "10102:10101" environment: - PILOSA_GOSSIP_SEEDS=pilosa1:14000 networks: @@ -319,13 +364,13 @@ networks: #### Running a Docker Swarm -It is very easy to run a Pilosa Cluster on different servers using [Docker Swarm mode](https://docs.docker.com/engine/swarm/). All we have to do is creating an overlay network instead of the bridge network. +It is very easy to run a Pilosa Cluster on different servers using [Docker Swarm mode](https://docs.docker.com/engine/swarm/). All we have to do is create an overlay network instead of a bridge network. -The instructions in this section require Docker 17.06 and better. Although it is possible to run a Docker swarm on MacOS or Windows, it is easiest to run it on Linux. So we assume you are trying these instructions on Linux, probably on the cloud. +The instructions in this section require Docker 17.06 or newer. Although it is possible to run a Docker swarm on MacOS or Windows, it is easiest to run it on Linux. The following instructions assume you are running on Linux. We are going to use two servers: the manager node runs in the first server and a worker node in the second server. -Docker nodes require some ports to be accesible from outside. Before carrying on, make sure the following ports are open on all nodes: TCP/2377, TCP/7946, UDP/7946, UDP/4789. +Docker nodes require some ports to be accesible from the outside. Before proceeding, make sure the following ports are open on all nodes: TCP/2377, TCP/7946, UDP/7946, UDP/4789. Let's initialize the swarm first. Run the following on the manager: ``` @@ -383,10 +428,10 @@ These were the same commands we used in the previous section except the port map docker run -it --rm --network=pilosanet --name shell alpine wget -q -O- pilosa1:10101/status ``` ``` response -{"state":"NORMAL","nodes":[{"id":"3e3b0abd-1945-441a-a01f-5a28272972f5","uri":{"scheme":"http","host":"pilosa1","port":10101},"isCoordinator":true},{"id":"71ed27cc-9443-4f41-88fb-1c22f92bf695","uri":{"scheme":"http","host":"pilosa2","port":10101},"isCoordinator":false}]} +{"state":"NORMAL","nodes":[{"id":"3e3b0abd-1945-441a-a01f-5a28272972f5","uri":{"scheme":"http","host":"pilosa1","port":10101},"isCoordinator":true},{"id":"71ed27cc-9443-4f41-88fb-1c22f92bf695","uri":{"scheme":"http","host":"pilosa2","port":10101},"isCoordinator":false}],"localID":"3e3b0abd-1945-441a-a01f-5a28272972f5"} ``` -You can add as many as worker nodes to both the swarm and the Pilosa cluster using the steps above. +You can add additional worker nodes to both the swarm and the Pilosa cluster using the steps above. #### What's Next? @@ -410,7 +455,7 @@ curl localhost:10101/index/patients \ {"success":true} ``` -In addition to storing rows of bits, a frame can also contain fields that store integer values. The next steps creates three fields (`age`, `weight`, `tcells`) in the `measurements` frame. +In addition to storing rows of bits, a field can also store integer values. The next steps creates three fields (`age`, `weight`, `tcells`) in the `measurements` field. ``` request curl localhost:10101/index/patients/field/age \ -X POST \