Updated: July 10, 2015
Time to explore the wonders of Docker some more. We've had two tutorials so far, one focused on a very thorough introduction, where we learned all about the technology, how to run services and expose ports, how to commit and build images with Dockerfiles, and a few other tricks. Then we used supervisord as a substitute to init scripts and systemd.
Today, we will learn about networking. How we can connect to our containers, how we can access the host from within spawned instances, and most importantly, how to connect from one container to another, without knowing anything about the topology in advance. This ought to be interesting. Much like the previous two guides, we'll go step by step, and explain everything in detail. After me.
Let's first create several containers. We already have our images from the last testing, so there's no need to start from scratch. We will run three containers, each one with its own SSH and Apache service. Ignore the security side of it, because we're trying to focus on the networking piece.
One thing we will do here that we haven't done the last time is, we will name our containers, for easier identification. This is done using the --name option. For example:
docker run -d -ti -p 22 -p 80 --name net3 image-4:latest
Let's also see what IP addresses our three instances have:
[root@localhost ~]# docker inspect net1 | grep -i ipaddr
[root@localhost ~]# docker inspect net2 | grep -i ipaddr
[root@localhost ~]# docker inspect net3 | grep -i ipaddr
This is our initial configuration for the test. We have three containers, all running on the same host, with their services and exposed service ports. Now, we can start doing some rather interesting stuff.
This is the easy part, and truth to be told, we've already done this in the previous exercise. We have connected to our containers using SSH, and we've tried testing ports using telnet. Yes, netcat (nc) also works fine, turn your security klaxon off please. There are two ways to connect. One, you can access containers directly, using their IP address, as we've done before.
The authenticity of host '172.17.0.5 (172.17.0.5)' can't be established. ECDSA key fingerprint is 00:4b:de:91:60:e5:22:cc:f7:89:01:19:3e:61:cb:ea.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.17.0.5' (ECDSA) to the list of known hosts.
Two, you can also access the host-mapped ports that correspond to the service ports inside the containers. If we just look at the PORTS information in the docker ps output:
0.0.0.0:49162->22/tcp, 0.0.0.0:49161->80/tcp net1
0.0.0.0:49163->22/tcp, 0.0.0.0:49164->80/tcp net2
0.0.0.0:49165->22/tcp, 0.0.0.0:49166->80/tcp net3
In the example above, for the container named net1, port 22 is mapped to port 49162 on the host, which means the docker0 interface, which by default is assigned the 172.17.42.1/16 network. Indeed, we see:
telnet 172.17.42.1 49163
Connected to 172.17.42.1.
Escape character is '^]'.
Now, why would you want to use the second method, as it is more confusing and difficult to remember? The answer is, if you do not use dynamic mapping, but instead you specify known, given host ports, then you can have a very clear topography of what your container network. Furthermore, if you want to allow your containers to be accessible from outside the host, you will need IP forwarding and NAT. If you use the docker0 interface for communication, you will need fewer, simpler iptables rules.
You can verify this by running the iptables -L command, and checking the DOCKER chain. New rules will be added automatically whenever you run a container with the -p option.
Chain DOCKER (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere 172.17.0.5 tcp dpt:http
ACCEPT tcp -- anywhere 172.17.0.5 tcp dpt:ssh
ACCEPT tcp -- anywhere 172.17.0.6 tcp dpt:ssh
ACCEPT tcp -- anywhere 172.17.0.6 tcp dpt:http
ACCEPT tcp -- anywhere 172.17.0.7 tcp dpt:ssh
ACCEPT tcp -- anywhere 172.17.0.7 tcp dpt:http
ACCEPT tcp -- anywhere 172.17.0.8 tcp dpt:ssh
ACCEPT tcp -- anywhere 172.17.0.8 tcp dpt:http
However, if you do not have any security restrictions, then the dynamic method can be quite useful, because you do not need to know the mapping of ports when you start your containers. You can derive information in real time, parsing the docker ps output. However, for services, it is often useful to set static IP addresses and known ports.
Because this is slightly confusing, let's recap briefly. Any docker host will have a docker0 or a similarly named bridge running, usually with a /16 subnet, although you can change the name the subnet or even the network. Behind it, you can have a whole range of containers, each one with its own private IP address and one or more exposed ports.
If you want to connect to these containers and their relevant ports, then you have several options. You can connect directly, but then you need IP forwarding, NAT and firewall rules for each one, and you need to know the IP addresses.
If you want to connect using the host interface (docker0) and the host port method, then you do not need to know how your containers run. You only need to know the mapping of host ports to container ports. In other words, if you want your containers to be accessible between multiple host nodes, you only need to make the host interface (docker0) visible to other hosts. This makes IP forwarding, NAT and firewall rules simple. For instance:
The above example lets you connect to SSH of three different containers. If you use a one-to-one mapping, i.e. same host port and container port (e.g. 22:22), then you are effectively limited to having only a single container with a single service listening on it. This is like your home router. You will have to use different ports for your containers to be able to use the one-to-one mapping. But this is confusing, because you expect your services to run on predictable ports.
With the example above, we see that we can have three containers use SSH. Internally, from the container perspective, it's always port 22. From the outside, the port numbers both identify the container as well as the service. And this becomes the only piece you need to know, without bothering with what happens behind the scenes.
This means you can dynamically assign host ports when creating containers, using some kind of script logic or such, and then use that information to identify your service ports. Thus, the host to container networking becomes a simple and elegant affair. And you can also connect from host to another, again, without knowing the container topography in advance.
This is a somewhat more interesting piece. Your containers may serve a long-lived purpose, and therefore, they might have to remain up and running for a bit of time. In turn, this means you might have to maintain the containers, which means upgrades, patching, hosts files updates, and whatnot. You might also be using a configuration management tool like Chef, Puppet or Cfengine, so there's that piece, too.
The first big question you need to ask is, can containers contact the outside world? Well, yes. In the introduction guide, we downloaded CentOS updates directly from the official repositories, so we know this piece works just fine. How about pinging docker0:
The second question, can we access docker0 interface, and THEN connect to its host ports. Let's assume for a moment that we have somehow been given the necessary information, inside the container.
telnet 172.17.42.1 49162
telnet: connect to address 172.17.42.1: No route to host
This will not work, because we have not configured IP forwarding on our host. Moreover, we will need firewall rules to allow the necessary traffic. Here, it becomes a little complicated, because if you're not familiar with iptables, you will somewhat struggle.
echo 1 > /proc/sys/net/ipv4/ip_forward
And the iptables rules - a somewhat lax set, mind (on the host):
iptables -t filter -A FORWARD -d 172.17.0.0/16 \
-o docker0 -j ACCEPT
iptables -t filter -A FORWARD -s 172.17.0.0/16 \
-i docker0 -j ACCEPT
iptables -t filter -A FORWARD -i docker0 -o docker0 -j ACCEPT
iptables -t nat -A POSTROUTING -s 172.17.0.0/16 \
! -d 172.17.0.0/16 -p tcp -j MASQUERADE --to-ports 1016-65535 \
iptables -t nat -A POSTROUTING -s 172.17.0.0/16
! -d 172.17.0.0/16 -p udp -j MASQUERADE --to-ports 1016-65535 \
iptables -t nat -A POSTROUTING -s 172.17.0.0/16
! -d 172.17.0.0/16 -j MASQUERADE
The simplest alternative is just to turn the firewall off - or allow all. This means running the necessary command, which could be systemctl stop firewalld, iptables -F or equivalent. In some scenarios, this may not be possible, as you need the extra security. However, this also ties into the first usecase we discussed. Now, imagine you had to create manual rules for every single container and its associated services.
With the right networking rules in place, you will be able to connect to the host ports, which effectively means talking to other containers, although there's nothing in the specific IP address or port that requires the target to be another container. Furthermore, you will also be able to connect to other networking devices on the host, or maybe even remote hosts.
Like David Bowie's song, Ashes to Ashes, only not as cool. This seems like the most useful scenario. Again, we have two options. We can try to connect to other containers directly, or we can try using the docker0 dynamically mapped ports. We will cheat. We will use the host and host ports as a gateway into containers. Now, the first method is pretty straightforward. Let's try to ping net2 and net3 from net1.
It works without any problems. But the issue is, we know about the IP addresses from the external world! We have not obtained this information from inside the container. Indeed, for security reasons as well as the fact the base container is a relatively small image, you don't have any great networking tools available by default. You will not even have the ifconfig utility available. Moreover, you cannot manipulate iptables inside containers, either:
iptables v1.4.21: can't initialize iptables table `filter': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.
Therefore, the second method, of accessing docker0 and connecting to host ports might not really be feasible, as we cannot manipulate host firewall rules, not from within the containers. This is why we will discuss a third method.
The proper way of doing this is by using the --link feature. What this means is, we will spawn new instances while linking them to other, existing containers. The link function will insert the hostname and IP address of linked containers into the environment and the /etc/hosts file of the new instance. Maybe it sounds a little confusing, so let's exemplify:
docker run -d -ti -p 80 --name web image-4:latest
docker inspect web | grep -i ipaddr
nc -w1 -v 172.17.0.9 80
Ncat: Version 6.40 ( http://nmap.org/ncat )
Ncat: Connected to 172.17.0.9:80.
We have a container running, and it's supposed to be a Web server. Now, we want clients to be able to access it. We want to use the server name, rather than the IP address, as we may not have any prior knowledge of the topography. Furthermore, names are useful as they allow us to change the network layout. Indeed, let's create a client container and link it to our Web server:
docker run -ti --name client --link web:web image-4:latest /bin/bash
What we have here is the following. We are making a link of source:target. In other words, the server (container) named web will be visible inside our new container named client as web. If you spawn the container and then attach to it, you can check the environment by running the env command in a BASH shell:
Because the same name mapping can be confusing, let's try a different mapping:
docker run -ti --name client --link web:dedoimedo image-4:latest /bin/bash
And then we get:
Optionally, if you're spawning containers that just need to be quick workers, then you may want to consider creating them using the run --rm option. For short-lived tasks, there is no reason to keep container files on the disk, and you can make Docker automatically clean them up. Reading the official documentation:
By default a container's file system persists even after the container exits. This makes debugging a lot easier (since you can inspect the final state) and you retain all your data by default. But if you are running short-term foreground processes, these container file systems can really pile up. If instead you'd like Docker to automatically clean up the container and remove the file system when the container exits, you can add the --rm flag.
docker run --rm -ti --name client --link web:dedoimedo image-4:latest /bin/bash
The second good bit about the link function is that new containers will automatically have their /etc/hosts file populated with the necessary names, so you're in for a treat. It will look something like below, with IPv6 entries removed:
[root@e1e92f6918dd /]# cat /etc/hosts
172.17.0.9 dedoimedo 8e689d8ae3ef web
And since we have the host name resolution and environment settings, it becomes just like any networking anywhere. Really simple, memorable and most importantly, scriptable.
If you don't think you've had enough fun, then:
Docker advanced networking guide
There you go. Another dreadful topic demystified. The thing is, I believe the biggest challenge you will face when working with Docker containers is the configuration of firewall rules rather than anything integral to the networking piece of this technology. But once you get the hang of it, it's not that difficult or complicated.
We have seen how to start containers and expose ports, exercise host to container, container to host and container to container networking, setup forwarding, and use links to make our life much easier and predictable. All in all, it was a busy but hopefully practical guide. Again, if you have any other topics, feel free to suggest them. And we're done.
P.S. If you like this article, then you'd better give some love back to Dedoimedo!