From 89ece0e147cd12ca24a28ddfd99ad1e5d16ad6bf Mon Sep 17 00:00:00 2001 From: Yuce Tekol Date: Fri, 18 May 2018 16:41:45 +0300 Subject: [PATCH 1/4] Added docker swarm tutorial --- docs/tutorials.md | 81 ++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 77 insertions(+), 4 deletions(-) diff --git a/docs/tutorials.md b/docs/tutorials.md index d728b0d20..7f3c700e2 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -240,7 +240,11 @@ Check out our [Administration Guide](https://www.pilosa.com/docs/latest/administ ### Setting Up a Docker Cluster -In this tutorial, we will be setting up a 2-node Pilosa cluster using Docker containers. The instructions below require Docker 1.13 or better. +In this tutorial, we will be setting up a 2-node Pilosa cluster using Docker containers. + +#### Running a Docker Cluster on a Single Server + +The instructions below require Docker 1.13 or better. Let's first be sure that the Pilosa image is up to date: ``` @@ -255,12 +259,12 @@ docker network create pilosanet Let's run the first Pilosa node and attach it to that virtual network. We set the first node as the cluster coordinator and use its address as the gossip seed. And also set the server address to `pilosa1`: ``` -docker run -it --rm --name pilosa1 -p 10101:10101 --network="pilosanet" pilosa/pilosa:latest server --bind pilosa1 --cluster.coordinator=true --gossip.seeds=pilosa1:14000 +docker run -it --rm --name pilosa1 -p 10101:10101 --network=pilosanet pilosa/pilosa:latest server --bind pilosa1 --cluster.coordinator=true --gossip.seeds=pilosa1:14000 ``` Let's run the second Pilosa node and attach it to the virtual network as well. Note that we set the address of the gossip seed to the address of the first node: ``` -docker run -it --rm --name pilosa2 -p 10102:10101 --network="pilosanet" pilosa/pilosa:latest server --bind pilosa2 --gossip.seeds=pilosa1:14000 +docker run -it --rm --name pilosa2 --network=pilosanet pilosa/pilosa:latest server --bind pilosa2 --gossip.seeds=pilosa1:14000 ``` Let's test that the nodes in the cluster connected with each other: @@ -312,11 +316,80 @@ networks: pilosanet: ``` +#### Running a Docker Swarm + +It is very easy to run a Pilosa Cluster on different servers using [Docker Swarm mode](https://docs.docker.com/engine/swarm/). All we have to do is creating an overlay network instead of the bridge network. + +The instructions in this section require Docker 17.06 and better. Although it is possible to run a Docker swarm on MacOS or Windows, it is easiest to run it on Linux. So we assume you are trying these instructions on Linux, probably on the cloud. + +We are going to use two servers: the master node runs in the first server and a slave node in the second server. + +Let's initialize the swarm first. Run the following on the master: +``` +docker swarm init --advertise-addr=IP-ADDRESS +``` + +Virtual machines running on the cloud usually have at least two network interfaces: the external interface and the internal interface. Use the IP of the external interface. + +The output of the command above should be similar to: +``` +To add a manager to this swarm, run the following command: + + docker swarm join --token SOME-TOKEN MASTER-IP-ADDRESS:2377 +``` + +Let's make the slave node join the master. Copy/paste the command above in a shell on the slave, replacing the token and IP address with the correct values. You may neeed to add `--advertise-addr=SLAVE-EXTERNAL-IP-ADDRESS` parameter if the slave has more than one network interface: +``` +docker swarm join --token SOME-TOKEN MASTER-IP-ADDRESS:2377 +``` + +Run the following on the master to check that the slave joined to the swarm: +``` +docker node ls +``` + +Which should output: + +ID|HOSTNAME|STATUS|AVAILABILITY|MANAGER STATUS|ENGINE VERSION +---|--------|------|------------|--------------|------------- +MASTER-ID *|swarm1|Ready|Active|Leader|18.05.0-ce| +SLAVE-ID|swarm2|Ready|Active||18.05.0-ce| + +If you have created the `pilosanet` network before, delete it before carrying on, otherwise skip to the next step: +``` +docker network rm pilosanet +``` + +Let's create the `pilosanet` network, but with `overlay` type this time. We should also make this network attachable in order to be able to attach containers to it. Run the following on the master: +``` +docker network create -d overlay pilosanet --attachable +``` + +We can now create the Pilosa containers. Let's start the coordinator node first. Run the following on one of the servers: +``` +docker run -it --rm --name pilosa1 --network=pilosanet pilosa/pilosa:latest server --bind pilosa1 --cluster.coordinator=true --gossip.seeds=pilosa1:14000 +``` + +And the following on the other server: +``` +docker run -it --rm --name pilosa2 --network=pilosanet pilosa/pilosa:latest server --bind pilosa2 --gossip.seeds=pilosa1:14000 +``` + +These were the same commands we used in the previous section except the port mapping! Let's run another container on the same virtual network to read the status from the coordinator: +``` request +docker run -it --rm --network=pilosanet --name shell alpine wget -q -O- pilosa1:10101/status +``` +``` response +{"state":"NORMAL","nodes":[{"id":"2f831340-4026-4cfd-8b1a-03d817de691a","uri":{"scheme":"http","host":"pilosa1","port":10101},"isCoordinator":true},{"id":"5549d6ee-d5f3-40a1-a53e-feebf6012c44","uri":{"scheme":"http","host":"pilosa2","port":10101},"isCoordinator":false},{"id":"9daf87b7-d1a8-4ce0-867e-11dbc7fc973e","uri":{"scheme":"http","host":"pilosa2","port":10101},"isCoordinator":false},{"id":"f85ab179-c72b-4846-bf16-9cec6f0086cd","uri":{"scheme":"http","host":"pilosa2","port":10101},"isCoordinator":false}]} +``` + +You can add as many as slave nodes to both the swarm and the Pilosa cluster using the steps above. + #### What's Next? Check out our [Administration Guide](https://www.pilosa.com/docs/latest/administration/) to learn more about making the most of your Pilosa cluster and [Configuration Documentation](https://www.pilosa.com/docs/latest/configuration/) to see the available options to configure Pilosa. -Refer to the [Docker documentation](https://docs.docker.com) to see your options about running Docker containers. +Refer to the [Docker documentation](https://docs.docker.com) to see your options about running Docker containers. The [Networking with overlay networks](https://docs.docker.com/network/network-tutorial-overlay/) is a detailed overview of the Docket swarm mode and overlay networks. ### Using Integer Field Values From 8f79815f9b28c41a058bee79e94e96c3a4a34f1e Mon Sep 17 00:00:00 2001 From: Yuce Tekol Date: Fri, 18 May 2018 16:44:57 +0300 Subject: [PATCH 2/4] Fixed the response in the docker swarm tut. --- docs/tutorials.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials.md b/docs/tutorials.md index 7f3c700e2..221a02a4b 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -380,7 +380,7 @@ These were the same commands we used in the previous section except the port map docker run -it --rm --network=pilosanet --name shell alpine wget -q -O- pilosa1:10101/status ``` ``` response -{"state":"NORMAL","nodes":[{"id":"2f831340-4026-4cfd-8b1a-03d817de691a","uri":{"scheme":"http","host":"pilosa1","port":10101},"isCoordinator":true},{"id":"5549d6ee-d5f3-40a1-a53e-feebf6012c44","uri":{"scheme":"http","host":"pilosa2","port":10101},"isCoordinator":false},{"id":"9daf87b7-d1a8-4ce0-867e-11dbc7fc973e","uri":{"scheme":"http","host":"pilosa2","port":10101},"isCoordinator":false},{"id":"f85ab179-c72b-4846-bf16-9cec6f0086cd","uri":{"scheme":"http","host":"pilosa2","port":10101},"isCoordinator":false}]} +{"state":"NORMAL","nodes":[{"id":"3e3b0abd-1945-441a-a01f-5a28272972f5","uri":{"scheme":"http","host":"pilosa1","port":10101},"isCoordinator":true},{"id":"71ed27cc-9443-4f41-88fb-1c22f92bf695","uri":{"scheme":"http","host":"pilosa2","port":10101},"isCoordinator":false}]} ``` You can add as many as slave nodes to both the swarm and the Pilosa cluster using the steps above. From 91df211be768c5fa715aad8e9990bc0de17f7124 Mon Sep 17 00:00:00 2001 From: Yuce Tekol Date: Mon, 21 May 2018 19:25:19 +0300 Subject: [PATCH 3/4] Updated docker swarm tutorial with required open ports --- docs/tutorials.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/tutorials.md b/docs/tutorials.md index 221a02a4b..442c22507 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -324,6 +324,8 @@ The instructions in this section require Docker 17.06 and better. Although it is We are going to use two servers: the master node runs in the first server and a slave node in the second server. +Docker nodes require some ports to be accesible from outside. Before carrying on, make sure the following ports are open on all nodes: TCP/2377, TCP/7946, UDP/7946, UDP/4789. + Let's initialize the swarm first. Run the following on the master: ``` docker swarm init --advertise-addr=IP-ADDRESS From 2a60f09c8c631c79561b25983e5a03147b4ee411 Mon Sep 17 00:00:00 2001 From: Yuce Tekol Date: Mon, 21 May 2018 20:32:27 +0300 Subject: [PATCH 4/4] updates --- docs/tutorials.md | 71 ++++++++++++++++++++++++----------------------- 1 file changed, 36 insertions(+), 35 deletions(-) diff --git a/docs/tutorials.md b/docs/tutorials.md index 442c22507..06cdab286 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -3,6 +3,7 @@ title = "Tutorials" weight = 4 nav = [ "Setting Up a Secure Cluster", + "Setting Up a Docker Cluster", "Using Integer Field Values", "Storing Row and Column Attributes", ] @@ -287,31 +288,31 @@ The corresponding [Docker Compose](https://docs.docker.com/compose/) file is bel ```yaml version: '2' services: - pilosa1: - image: pilosa/pilosa:latest - ports: - - "10101:10101" - environment: - - PILOSA_CLUSTER_COORDINATOR=true - - PILOSA_GOSSIP_SEEDS=pilosa1:14000 - networks: - - pilosanet - entrypoint: - - /pilosa - - server - - --bind - - "pilosa1:10101" - pilosa2: - image: pilosa/pilosa:latest - environment: - - PILOSA_GOSSIP_SEEDS=pilosa1:14000 - networks: - - pilosanet - entrypoint: - - /pilosa - - server - - --bind - - "pilosa2:10101" + pilosa1: + image: pilosa/pilosa:latest + ports: + - "10101:10101" + environment: + - PILOSA_CLUSTER_COORDINATOR=true + - PILOSA_GOSSIP_SEEDS=pilosa1:14000 + networks: + - pilosanet + entrypoint: + - /pilosa + - server + - --bind + - "pilosa1:10101" + pilosa2: + image: pilosa/pilosa:latest + environment: + - PILOSA_GOSSIP_SEEDS=pilosa1:14000 + networks: + - pilosanet + entrypoint: + - /pilosa + - server + - --bind + - "pilosa2:10101" networks: pilosanet: ``` @@ -322,11 +323,11 @@ It is very easy to run a Pilosa Cluster on different servers using [Docker Swarm The instructions in this section require Docker 17.06 and better. Although it is possible to run a Docker swarm on MacOS or Windows, it is easiest to run it on Linux. So we assume you are trying these instructions on Linux, probably on the cloud. -We are going to use two servers: the master node runs in the first server and a slave node in the second server. +We are going to use two servers: the manager node runs in the first server and a worker node in the second server. Docker nodes require some ports to be accesible from outside. Before carrying on, make sure the following ports are open on all nodes: TCP/2377, TCP/7946, UDP/7946, UDP/4789. -Let's initialize the swarm first. Run the following on the master: +Let's initialize the swarm first. Run the following on the manager: ``` docker swarm init --advertise-addr=IP-ADDRESS ``` @@ -337,15 +338,15 @@ The output of the command above should be similar to: ``` To add a manager to this swarm, run the following command: - docker swarm join --token SOME-TOKEN MASTER-IP-ADDRESS:2377 + docker swarm join --token SOME-TOKEN MANAGER-IP-ADDRESS:2377 ``` -Let's make the slave node join the master. Copy/paste the command above in a shell on the slave, replacing the token and IP address with the correct values. You may neeed to add `--advertise-addr=SLAVE-EXTERNAL-IP-ADDRESS` parameter if the slave has more than one network interface: +Let's make the worker node join the manager. Copy/paste the command above in a shell on the worker, replacing the token and IP address with the correct values. You may neeed to add `--advertise-addr=WORKER-EXTERNAL-IP-ADDRESS` parameter if the worker has more than one network interface: ``` -docker swarm join --token SOME-TOKEN MASTER-IP-ADDRESS:2377 +docker swarm join --token SOME-TOKEN MANAGER-IP-ADDRESS:2377 ``` -Run the following on the master to check that the slave joined to the swarm: +Run the following on the manager to check that the worker joined to the swarm: ``` docker node ls ``` @@ -354,15 +355,15 @@ Which should output: ID|HOSTNAME|STATUS|AVAILABILITY|MANAGER STATUS|ENGINE VERSION ---|--------|------|------------|--------------|------------- -MASTER-ID *|swarm1|Ready|Active|Leader|18.05.0-ce| -SLAVE-ID|swarm2|Ready|Active||18.05.0-ce| +MANAGER-ID *|swarm1|Ready|Active|Leader|18.05.0-ce| +WORKER-ID|swarm2|Ready|Active||18.05.0-ce| If you have created the `pilosanet` network before, delete it before carrying on, otherwise skip to the next step: ``` docker network rm pilosanet ``` -Let's create the `pilosanet` network, but with `overlay` type this time. We should also make this network attachable in order to be able to attach containers to it. Run the following on the master: +Let's create the `pilosanet` network, but with `overlay` type this time. We should also make this network attachable in order to be able to attach containers to it. Run the following on the manager: ``` docker network create -d overlay pilosanet --attachable ``` @@ -385,7 +386,7 @@ docker run -it --rm --network=pilosanet --name shell alpine wget -q -O- pilosa1: {"state":"NORMAL","nodes":[{"id":"3e3b0abd-1945-441a-a01f-5a28272972f5","uri":{"scheme":"http","host":"pilosa1","port":10101},"isCoordinator":true},{"id":"71ed27cc-9443-4f41-88fb-1c22f92bf695","uri":{"scheme":"http","host":"pilosa2","port":10101},"isCoordinator":false}]} ``` -You can add as many as slave nodes to both the swarm and the Pilosa cluster using the steps above. +You can add as many as worker nodes to both the swarm and the Pilosa cluster using the steps above. #### What's Next?