HAProxy and container IP changes in Docker

HAProxy and Docker containers

Docker is a nice tool to handle containers: it allows building and running your apps in a simple and efficient way.

When used in production together with HAProxy, devops teams face a big challenge: how to followup a container IP change when restarting a container?

This blog article aims at giving a first answer to this question.

The version of docker used for this article is 1.8.1 (very important, since docker’s default behavior has changed in 1.9.0…)

HAProxy, webapp and docker diagram

The diagram below shows how docker runs on my laptop:

  • A docker0 network interface, with IP
  • all containers run in the subnet
|                                                                               |
|  +-----------------------docker-----------------------+         |
|  |                                                    | docker0:   |
|  |  +---------+  +---------+       +----------+       |                       |
|  |  | HAProxy |  | appsrv1 |       | rsyslogd |       |                       |
|  |  +---------+  +---------+       +----------+       |                       |
|  |                                                    |                       |
|  +----------------------------------------------------+                       |
|                                                                               |

The IP address associated to docker0 will be used to export some services.

In this article, we’ll start up 3 containers:

  1. rsyslogd: where HAProxy will send all its logs
  2. appsrv1: our application server, which may be restarted at any time
  3. haproxy: our load-balancer, which must follow-up appsrv1‘s IP

Building and running the lab


First, we need rsyslogd and haproxy containers. They can be build from the following Dockerfiles:

Then run:

docker build -t blog:haproxy_dns ~/tmp/haproxy/blog/haproxy_docker_dns_link/blog_haproxy_dns/
docker build -t blog:rsyslogd ~/tmp/haproxy/blog/haproxy_docker_dns_link/blog_rsyslogd/

I consider appsrv container as yours: it’s your application.

Starting up our lab

Docker assign IPs to containers in the order they are started up, incrementing last byte for each new container.

To make it simpler, let’s restart docker first, so our container IPs are predictible:

sudo /etc/rc.d/docker restart

Then let’s start up our rsyslogd container:

docker run --detach --name rsyslogd --hostname=rsyslogd \
	--publish= \

And let’s attach a terminal to it:

docker attach rsyslogd

Now, run appsrv container as appsrv1:

docker run --detach --name appsrv1 --hostname=appsrv1 demo:appsrv

And finally, let’s start HAProxy, with a docker link to appsrv1:

docker run --detach --name haproxy --hostname=haproxy \
	--link appsrv1:appsrv1 \

Docker links, /etc/hosts file updated and DNS

When using the ”–link” option, docker creates a new entry in the containers /etc/hosts file with the IP address and name provided by the ”link” directive.
Docker will also update this file when the remote container (here appsrv1) IP address is changed (IE when restarting the container).

If you’re familiar with HAProxy, you already know it doesn’t do file system IOs at run time. Furthermore, HAProxy doesn’t use /etc/hosts file directly. The glibc might use it when HAProxy asks for DNS resolution when parsing the configuration file. (read below for DNS resolution at runtime)

That said, if appsrv1 IP get changed, then /etc/hosts file is updated accordingly, then HAProxy is not aware of the change and the application may fail.
A quick solution would be to reload HAProxy process in its container, to force it taking into account the new IP.

A more reliable solution, is to use HAProxy 1.6 DNS resolution capability to follow-up the IP change. With this purpose in mind, we added 2 tools into our HAProxy container:

  1. dnsmasq: tiny software which can act as a DNS server which takes /etc/hosts file as its database
  2. inotifytools: watch changes on /etc/hosts file and force dnsmasq to reload it when necessary

I guess now you got it:

  • when appsrv1 is restarted, then docker gives it a new IP
  • Docker populates then this IP address into all /etc/hosts file required (those using ”link” directives)
  • Once populated, inotify tool detect the file change and triggers a dnsmasq reload
  • HAProxy periodically (can be configured) probes DNS and will get the new IP address quickly from dnsmasq

Docker container restart and HAProxy followup in action

At this stage, we should have a container attached to rsyslogd and we should be able to see HAProxy logging. Let’s give it a try:


Nov 17 09:29:09 haproxy[10]: [17/Nov/2015:09:29:09.729] f_myapp b_myapp/appsrv1 0/0/0/1/1 200 858 - - ---- 1/1/0/1/0 0/0 "GET / HTTP/1.1"

Now, let’s consider your dev team delivered a new version of your application, so you build it and need to restart its running container:

docker restart appsrv1

and voilà:

==> /var/log/haproxy/events <==
Nov 17 09:29:29 haproxy[10]: b_myapp/appsrv1 changed its IP from to by docker/dnsmasq.
Nov 17 09:29:29 haproxy[10]: b_myapp/appsrv1 changed its IP from to by docker/dnsmasq.

Let’s test the application again:


Nov 17 09:29:31 haproxy[10]: [17/Nov/2015:09:29:31.013] f_myapp b_myapp/appsrv1 0/0/0/0/0 200 858 - - ---- 1/1/0/1/0 0/0 "GET / HTTP/1.1"


There are a few limitations in this mechanism:

  • it’s painful to maintaint ”link” directive when you have a 10s or 100s or more of containers….
  • the host computer, and computers in the host network can’t easily access our containers, because we don’t know their IPs and their hostnames are resolved in HAProxy container only
  • if we want to add more appserver in HAProxy‘s farm we still need to restart HAProxy‘s container (and update configuration accordingly)

To fix some of the issues above, we can dedicate a container to perform DNS resolution within our docker world and deliver responses to any running containers or hosts in the network. We’ll see that in a next blog article


4 thoughts on “HAProxy and container IP changes in Docker”

  1. Thanks for the nice example!

    As you already mentioned under “Limitations” you might consider to use the Docker 1.9 networking feature, which also allows transparent multi-host networking. An example has been described at https://github.com/gesellix/docker-haproxy-network – I’d be happy about feedback for improvements. If you have questions or need someone to give feedback for your next article, feel free to contact me 🙂

    Only one questions regarding the last limitation: is it possible to dynamically add new backends without reloading the configuration? Would that even make sense?


    1. Hi,

      Docker frustrated me when I upgraded from 1.8.1 to 1.9.0 a couple of days before a conference where I use Docker to show HAProxy nice features (slides here: http://www.slideshare.net/haproxytech/haproxy-web-performance-55536394) and all my DNS was broken because of this upgrade.

      In order to perform seamless upgrade, I’m now working on a DNS-to-dockerapi container to server container IPs based on docker API.

      About dynamically create new backends, this is quite complicated at run time, since HAProxy’s memory is fully pre-allocated and no more memory is allocated at run time.
      But to be transparent, we’re aware of such requirement and we’re thinking about it.

        1. I know docker has improved this recently, but I had not time to upgrade my docker yet. I’ll do it in a couple of weeks and give it a try update the article (or write a new one).

Leave a Reply

Your email address will not be published. Required fields are marked *