From 10939f41d6a639b6e4d27d9ef141b010fd924e7b Mon Sep 17 00:00:00 2001 From: Yuce Tekol Date: Mon, 21 May 2018 19:44:48 +0300 Subject: [PATCH 1/2] Added Docker tutorial --- docs/tutorials.md | 155 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 155 insertions(+) diff --git a/docs/tutorials.md b/docs/tutorials.md index 432c7466e..442c22507 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -238,6 +238,161 @@ curl -k --ipv4 https://02.pilosa.local:10502/index/sample-index/query -d 'Bitmap Check out our [Administration Guide](https://www.pilosa.com/docs/latest/administration/) to learn more about making the most of your Pilosa cluster and [Configuration Documentation](https://www.pilosa.com/docs/latest/configuration/) to see the available options to configure Pilosa. +### Setting Up a Docker Cluster + +In this tutorial, we will be setting up a 2-node Pilosa cluster using Docker containers. + +#### Running a Docker Cluster on a Single Server + +The instructions below require Docker 1.13 or better. + +Let's first be sure that the Pilosa image is up to date: +``` +docker pull pilosa/pilosa:latest +``` + +Then, create a virtual network to attach our containers. We are going to name our network `pilosanet`: + +``` +docker network create pilosanet +``` + +Let's run the first Pilosa node and attach it to that virtual network. We set the first node as the cluster coordinator and use its address as the gossip seed. And also set the server address to `pilosa1`: +``` +docker run -it --rm --name pilosa1 -p 10101:10101 --network=pilosanet pilosa/pilosa:latest server --bind pilosa1 --cluster.coordinator=true --gossip.seeds=pilosa1:14000 +``` + +Let's run the second Pilosa node and attach it to the virtual network as well. Note that we set the address of the gossip seed to the address of the first node: +``` +docker run -it --rm --name pilosa2 --network=pilosanet pilosa/pilosa:latest server --bind pilosa2 --gossip.seeds=pilosa1:14000 +``` + +Let's test that the nodes in the cluster connected with each other: +``` request +curl localhost:10101/status +``` +``` response +{"state":"NORMAL","nodes":[{"id":"2e8332d0-1fee-44dd-a359-e0d6ecbcefc1","uri":{"scheme":"http","host":"pilosa1","port":10101},"isCoordinator":true},{"id":"8c0dbcdc-9503-4265-8ad2-ba85a4bb10fa","uri":{"scheme":"http","host":"pilosa2","port":10101},"isCoordinator":false}]} +``` + +And similarly for the second node: +``` request +curl localhost:10102/status +``` +``` response +{"state":"NORMAL","nodes":[{"id":"2e8332d0-1fee-44dd-a359-e0d6ecbcefc1","uri":{"scheme":"http","host":"pilosa1","port":10101},"isCoordinator":true},{"id":"8c0dbcdc-9503-4265-8ad2-ba85a4bb10fa","uri":{"scheme":"http","host":"pilosa2","port":10101},"isCoordinator":false}]} +``` +The corresponding [Docker Compose](https://docs.docker.com/compose/) file is below: + +```yaml +version: '2' +services: + pilosa1: + image: pilosa/pilosa:latest + ports: + - "10101:10101" + environment: + - PILOSA_CLUSTER_COORDINATOR=true + - PILOSA_GOSSIP_SEEDS=pilosa1:14000 + networks: + - pilosanet + entrypoint: + - /pilosa + - server + - --bind + - "pilosa1:10101" + pilosa2: + image: pilosa/pilosa:latest + environment: + - PILOSA_GOSSIP_SEEDS=pilosa1:14000 + networks: + - pilosanet + entrypoint: + - /pilosa + - server + - --bind + - "pilosa2:10101" +networks: + pilosanet: +``` + +#### Running a Docker Swarm + +It is very easy to run a Pilosa Cluster on different servers using [Docker Swarm mode](https://docs.docker.com/engine/swarm/). All we have to do is creating an overlay network instead of the bridge network. + +The instructions in this section require Docker 17.06 and better. Although it is possible to run a Docker swarm on MacOS or Windows, it is easiest to run it on Linux. So we assume you are trying these instructions on Linux, probably on the cloud. + +We are going to use two servers: the master node runs in the first server and a slave node in the second server. + +Docker nodes require some ports to be accesible from outside. Before carrying on, make sure the following ports are open on all nodes: TCP/2377, TCP/7946, UDP/7946, UDP/4789. + +Let's initialize the swarm first. Run the following on the master: +``` +docker swarm init --advertise-addr=IP-ADDRESS +``` + +Virtual machines running on the cloud usually have at least two network interfaces: the external interface and the internal interface. Use the IP of the external interface. + +The output of the command above should be similar to: +``` +To add a manager to this swarm, run the following command: + + docker swarm join --token SOME-TOKEN MASTER-IP-ADDRESS:2377 +``` + +Let's make the slave node join the master. Copy/paste the command above in a shell on the slave, replacing the token and IP address with the correct values. You may neeed to add `--advertise-addr=SLAVE-EXTERNAL-IP-ADDRESS` parameter if the slave has more than one network interface: +``` +docker swarm join --token SOME-TOKEN MASTER-IP-ADDRESS:2377 +``` + +Run the following on the master to check that the slave joined to the swarm: +``` +docker node ls +``` + +Which should output: + +ID|HOSTNAME|STATUS|AVAILABILITY|MANAGER STATUS|ENGINE VERSION +---|--------|------|------------|--------------|------------- +MASTER-ID *|swarm1|Ready|Active|Leader|18.05.0-ce| +SLAVE-ID|swarm2|Ready|Active||18.05.0-ce| + +If you have created the `pilosanet` network before, delete it before carrying on, otherwise skip to the next step: +``` +docker network rm pilosanet +``` + +Let's create the `pilosanet` network, but with `overlay` type this time. We should also make this network attachable in order to be able to attach containers to it. Run the following on the master: +``` +docker network create -d overlay pilosanet --attachable +``` + +We can now create the Pilosa containers. Let's start the coordinator node first. Run the following on one of the servers: +``` +docker run -it --rm --name pilosa1 --network=pilosanet pilosa/pilosa:latest server --bind pilosa1 --cluster.coordinator=true --gossip.seeds=pilosa1:14000 +``` + +And the following on the other server: +``` +docker run -it --rm --name pilosa2 --network=pilosanet pilosa/pilosa:latest server --bind pilosa2 --gossip.seeds=pilosa1:14000 +``` + +These were the same commands we used in the previous section except the port mapping! Let's run another container on the same virtual network to read the status from the coordinator: +``` request +docker run -it --rm --network=pilosanet --name shell alpine wget -q -O- pilosa1:10101/status +``` +``` response +{"state":"NORMAL","nodes":[{"id":"3e3b0abd-1945-441a-a01f-5a28272972f5","uri":{"scheme":"http","host":"pilosa1","port":10101},"isCoordinator":true},{"id":"71ed27cc-9443-4f41-88fb-1c22f92bf695","uri":{"scheme":"http","host":"pilosa2","port":10101},"isCoordinator":false}]} +``` + +You can add as many as slave nodes to both the swarm and the Pilosa cluster using the steps above. + +#### What's Next? + +Check out our [Administration Guide](https://www.pilosa.com/docs/latest/administration/) to learn more about making the most of your Pilosa cluster and [Configuration Documentation](https://www.pilosa.com/docs/latest/configuration/) to see the available options to configure Pilosa. + +Refer to the [Docker documentation](https://docs.docker.com) to see your options about running Docker containers. The [Networking with overlay networks](https://docs.docker.com/network/network-tutorial-overlay/) is a detailed overview of the Docket swarm mode and overlay networks. + ### Using Integer Field Values From d1153755e98cc64fddc9bb1869a0dfc108c398fc Mon Sep 17 00:00:00 2001 From: Yuce Tekol Date: Mon, 21 May 2018 20:31:40 +0300 Subject: [PATCH 2/2] updates --- docs/tutorials.md | 71 ++++++++++++++++++++++++----------------------- 1 file changed, 36 insertions(+), 35 deletions(-) diff --git a/docs/tutorials.md b/docs/tutorials.md index 442c22507..06cdab286 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -3,6 +3,7 @@ title = "Tutorials" weight = 4 nav = [ "Setting Up a Secure Cluster", + "Setting Up a Docker Cluster", "Using Integer Field Values", "Storing Row and Column Attributes", ] @@ -287,31 +288,31 @@ The corresponding [Docker Compose](https://docs.docker.com/compose/) file is bel ```yaml version: '2' services: - pilosa1: - image: pilosa/pilosa:latest - ports: - - "10101:10101" - environment: - - PILOSA_CLUSTER_COORDINATOR=true - - PILOSA_GOSSIP_SEEDS=pilosa1:14000 - networks: - - pilosanet - entrypoint: - - /pilosa - - server - - --bind - - "pilosa1:10101" - pilosa2: - image: pilosa/pilosa:latest - environment: - - PILOSA_GOSSIP_SEEDS=pilosa1:14000 - networks: - - pilosanet - entrypoint: - - /pilosa - - server - - --bind - - "pilosa2:10101" + pilosa1: + image: pilosa/pilosa:latest + ports: + - "10101:10101" + environment: + - PILOSA_CLUSTER_COORDINATOR=true + - PILOSA_GOSSIP_SEEDS=pilosa1:14000 + networks: + - pilosanet + entrypoint: + - /pilosa + - server + - --bind + - "pilosa1:10101" + pilosa2: + image: pilosa/pilosa:latest + environment: + - PILOSA_GOSSIP_SEEDS=pilosa1:14000 + networks: + - pilosanet + entrypoint: + - /pilosa + - server + - --bind + - "pilosa2:10101" networks: pilosanet: ``` @@ -322,11 +323,11 @@ It is very easy to run a Pilosa Cluster on different servers using [Docker Swarm The instructions in this section require Docker 17.06 and better. Although it is possible to run a Docker swarm on MacOS or Windows, it is easiest to run it on Linux. So we assume you are trying these instructions on Linux, probably on the cloud. -We are going to use two servers: the master node runs in the first server and a slave node in the second server. +We are going to use two servers: the manager node runs in the first server and a worker node in the second server. Docker nodes require some ports to be accesible from outside. Before carrying on, make sure the following ports are open on all nodes: TCP/2377, TCP/7946, UDP/7946, UDP/4789. -Let's initialize the swarm first. Run the following on the master: +Let's initialize the swarm first. Run the following on the manager: ``` docker swarm init --advertise-addr=IP-ADDRESS ``` @@ -337,15 +338,15 @@ The output of the command above should be similar to: ``` To add a manager to this swarm, run the following command: - docker swarm join --token SOME-TOKEN MASTER-IP-ADDRESS:2377 + docker swarm join --token SOME-TOKEN MANAGER-IP-ADDRESS:2377 ``` -Let's make the slave node join the master. Copy/paste the command above in a shell on the slave, replacing the token and IP address with the correct values. You may neeed to add `--advertise-addr=SLAVE-EXTERNAL-IP-ADDRESS` parameter if the slave has more than one network interface: +Let's make the worker node join the manager. Copy/paste the command above in a shell on the worker, replacing the token and IP address with the correct values. You may neeed to add `--advertise-addr=WORKER-EXTERNAL-IP-ADDRESS` parameter if the worker has more than one network interface: ``` -docker swarm join --token SOME-TOKEN MASTER-IP-ADDRESS:2377 +docker swarm join --token SOME-TOKEN MANAGER-IP-ADDRESS:2377 ``` -Run the following on the master to check that the slave joined to the swarm: +Run the following on the manager to check that the worker joined to the swarm: ``` docker node ls ``` @@ -354,15 +355,15 @@ Which should output: ID|HOSTNAME|STATUS|AVAILABILITY|MANAGER STATUS|ENGINE VERSION ---|--------|------|------------|--------------|------------- -MASTER-ID *|swarm1|Ready|Active|Leader|18.05.0-ce| -SLAVE-ID|swarm2|Ready|Active||18.05.0-ce| +MANAGER-ID *|swarm1|Ready|Active|Leader|18.05.0-ce| +WORKER-ID|swarm2|Ready|Active||18.05.0-ce| If you have created the `pilosanet` network before, delete it before carrying on, otherwise skip to the next step: ``` docker network rm pilosanet ``` -Let's create the `pilosanet` network, but with `overlay` type this time. We should also make this network attachable in order to be able to attach containers to it. Run the following on the master: +Let's create the `pilosanet` network, but with `overlay` type this time. We should also make this network attachable in order to be able to attach containers to it. Run the following on the manager: ``` docker network create -d overlay pilosanet --attachable ``` @@ -385,7 +386,7 @@ docker run -it --rm --network=pilosanet --name shell alpine wget -q -O- pilosa1: {"state":"NORMAL","nodes":[{"id":"3e3b0abd-1945-441a-a01f-5a28272972f5","uri":{"scheme":"http","host":"pilosa1","port":10101},"isCoordinator":true},{"id":"71ed27cc-9443-4f41-88fb-1c22f92bf695","uri":{"scheme":"http","host":"pilosa2","port":10101},"isCoordinator":false}]} ``` -You can add as many as slave nodes to both the swarm and the Pilosa cluster using the steps above. +You can add as many as worker nodes to both the swarm and the Pilosa cluster using the steps above. #### What's Next?