HAProxy 1.6-dev1 and LUA

HAProxy 1.6-dev1

Yesterday, Willy has released HAProxy 1.6-dev1: ANNOUNCE HAProxy 1.6-dev1.
This version contains many new features and core improvements. Amongst the new features, one is LUA, contributed by Thierry (HAProxy Technologies developer).

NOTE: We invite everyone to download and test HAProxy 1.6-dev1 and to report bugs to the mailing list.

Please note that there may be a code freeze of HAProxy 1.6 in May. Purpose is to release a stable version in September.


LUA is a powerful, fast, lightweight, embeddable scripting language.

More about lua

The main advantage of LUA for HAProxy is to give the ability to anyone to write their own features without having to know writing C code and patching HAProxy…

About this article

Please note that LUA integration in HAProxy is brand new. So the information provided here are true today but may be obsolete later, because of internal changes and updates.

Compiling HAProxy and LUA

Installaing LUA 5.3

For now, HAProxy requires LUA 5.3 to work, here is the procedure to install it:

cd /usr/src
curl -R -O http://www.lua.org/ftp/lua-5.3.0.tar.gz
tar zxf lua-5.3.0.tar.gz
cd lua-5.3.0
make linux
sudo make INSTALL_TOP=/opt/lua53 install

LUA 5.3 library and include files are now installed in /opt/lua53.

Compiling HAProxy for lua

Since HAProxy is in a dev state, we may encounter some bugs. Let’s enable some debugging features:

make DEBUG=-ggdb CFLAGS=-O0 TARGET=linux2628 
USE_LUA=yes LUA_LIB=/opt/lua53/lib/ 
LUA_INC=/opt/lua53/include/ LDFLAGS=-ldl

Using LUA in HAProxy

turning HAProxy as a mirror web server

By mirror web server, I mean a web server which returns the request in the body of the response, without any modification.
It may be useful to use in HAProxy when you want to setup some HTTP header manipulation in HAProxy and you want to see the result. (It is more convenient than tcpdump).

* First, the HAProxy configuration:

 lua-load ./mylua.lua

 mode http
 timeout connect 1s
 timeout client 1s
 timeout server 1s

frontend fe
 bind name fe
 acl debugme req.hdr_cnt(X-debug-me) ge 1
 # add your http manipulation rules here
 http-request lua mirror if debugme
 default_backend be

backend be
 server s

* And the LUA which does the mirroring of the request:

-- a simple mirror web server
-- it generates a response whose body contains the requests headers
function mirror(txn)
local buffer = ""
local response = ""
local mydate = txn.sc:http_date(txn.f:date())

buffer = buffer .. "You sent the following headersrn"
buffer = buffer .. "===============================================rn"
buffer = buffer .. txn.req:dup()
buffer = buffer .. "===============================================rn"

response = response .. "HTTP/1.0 200 OKrn"
response = response .. "Server: haproxy-lua/mirrorrn"
response = response .. "Content-Type: text/htmlrn"
response = response .. "Date: " .. mydate .. "rn"
response = response .. "Content-Length: " .. buffer:len() .. "rn"
response = response .. "Connection: closern"
response = response .. "rn"
response = response .. buffer


Some explanations:
* mydate is the result of an HAProxy converter txn.sc:http_date applied to an HAProxy fetch txn.f:date()
* The call to txn.req:dup returns the content of HAProxy’s buffer request.

Testing: simply run a curl against HAProxy, don’t forget to send a X-debug-me HTTP header with the parameter -H "X-debug-me: yes". HAProxy should return your request in the response body:

HTTP/1.0 200 OK
Server: haproxy-lua/mirror
Content-Type: text/html
Date: Fri, 12 Mar 2015 13:06:44 GMT
Content-Length: 208
Connection: keep-alive

You sent the following headers
GET / HTTP/1.1
User-Agent: curl/7.41.0
Accept: */*



A HTTP monitor which matches multiple conditions in HAProxy

Load-Balancing and health checking

Health checking is the method to check a service availability on a server.
It is one of the most important feature of a load-balancer. How could we balance traffic amongst servers if we don’t know if the service is alive???

HAProxy and HTTP check

HAProxy can probe HTTP applications using httpchk option.
This option can be customized using the http-check expect directive to match different status codes or content.
That said, a single http-check expect rule can match.
So we can’t match a status code and the presence of a string in the page, for example.

Make HAProxy match multiple conditions for HTTP health checking

The solution is to use to the raw tcp-check and write a health check script sequence which match all the conditions.

For example, you want to ensure the server’s response has:

  • HTTP status code is 200
  • absence of keyword Error
backend myapp
 option tcp-check
 tcp-check send GET /my/check/url HTTP/1.1rn
 tcp-check send Host: myhostrn
 tcp-check send Connection: closern
 tcp-check send rn
 tcp-check expect string HTTP/1.1 200 OK
 tcp-check expect ! string Error


Web application name to backend mapping in HAProxy


Let’s take a web application platform where many HTTP Host header points to.
Of course, this platform hosts many backends and HAProxy is used to perform content switching based on the Host header to route HTTP traffic to each backend.

HAProxy map

HAProxy 1.5 introduced a cool feature: converters. One converter type is map.
Long story made short: a map allows to map a data in input to an other one on output.

A map is stored in a flat file which is loaded by HAProxy on startup. It is composed by 2 columns, on the left the input string, on the right the output one:

in out

Basically, if you call the map above and give it the in strings, it will return out.


Now, the interesting part of the article 🙂

As stated in introduction, we want to map hundreds of Host headers to tens of backends.

The old way of mapping: acl and use_backend rules

Before the map, we had to use acls and use_backend rules.

like below:

frontend ft_allapps
 use_backend bk_app1 if { hdr(Host) -i app1.domain1.com app1.domain2.com }
 use_backend bk_app2 if { hdr(Host) -i app2.domain1.com app2.domain2.com }
 default_backend bk_default

Add one statement per use_backend rule.

This works nicely for a few backends and a few domain names. But this type of configuration is hardly scallable…

The new way of mapping: one map and one use_backend rule

Now we can use map to achieve the same purpose.

First, let’s create a map file called domain2backend.map, with the following content: on the left, the domain name, on the right, the backend name:

#domainname  backendname
app1.domain1.com bk_app1
app1.domain2.com bk_app1
app2.domain1.com bk_app2
app2.domain2.com bk_app2

And now, HAProxy configuration:

frontend ft_allapps
 use_backend %[req.hdr(host),lower,map_dom(/etc/hapee-1.5/domain2backend.map,bk_default)]

Here is what HAProxy will do:

  1. req.hdr(host) ==> fetch the Host header from the HTTP request
  2. lower ==> convert the string into lowercase
  3. map_dom(/etc/hapee-1.5/domain2backend.map) ==> look for the lowercase Host header in the map and return the backend name if found. If not found, the name of a default backend is returned
  4. route traffic to the backend name returned by the map

Now, adding a new content switching rule means just add one new line in the map content (and reload HAProxy). No regexes, map data is stored in a tree, so processing time is very low compared to matching many string in many ACLs for many use_backend rules.

simple is beautiful!!!

HAProxy map content auto update

If you are an HAPEE user (and soon available for the ALOHA), you can use the lb-update content to download the content of the map automatically.
Add the following statement in your configuration:

 update id domain2backend.map url delay 60s timeout 5s retries 3 map


HAProxy and sslv3 poodle vulnerability

SSLv3 poodle vulnerability

Yesterday, Google security researchers have disclosed a new vulnerability on SSL protocol.
Fortunately, this vulnerability is only on an old version of the SSL protocol: SSLv3 (15 years old protocol).
An attacker can force a browser to downgrade the protocol version used to cipher traffic to SSLv3 in order to exploit the POODLE vulnerability and access to data in clear.

Some reading about SSLv3 Poodle vulnerability:
* http://googleonlinesecurity.blogspot.fr/2014/10/this-poodle-bites-exploiting-ssl-30.html
* https://www.imperialviolet.org/2014/10/14/poodle.html
* https://www.poodletest.com/

Today’s article is going to explain how to use HAProxy to simply prevent using SSLv3 or to prevent those users to reach your applications and print them a message.

Disable SSLv3 in HAProxy

In SSL offloading mode

In this mode, HAProxy is the SSL endpoint of the connection.
It’s a simple keyword on the frontend bind directive:

  bind ssl crt /pat/to/cert.pem no-sslv3

In SSL forward mode

In this mode, HAProxy forwards the SSL traffic to the server without deciphering it.
We must setup an ACL to match the SSL protocol version, then we can refuse the connection. This must be added in a **frontend** section:

  tcp-request inspect-delay 2s
  acl sslv3 req.ssl_ver 3
  tcp-request content reject if sslv3

Communicate a message to users

Denying sslv3 is a good way, but a better one would to educate as well users who are using this protocol.
The configuration below shows how to redirect a user to a specific page when they want to use your application over an SSLv3 connection. Of course, HAProxy must allow itself SSLv3:

frontend ft_www
  bind ssl crt /pat/to/cert.pem
  acl sslv3 ssl_fc_protocol SSLv3
# first rule after all your 'http-request deny' and
# before all the redirect, rewrite, etc....
  http-request allow if sslv3
# first content switching rule
  use_backend bk_sslv3 if sslv3

backend bk_sslv3
  mode http
  errorfile 503 /etc/haproxy/pages/sslv3.http

And the content of the file /etc/haproxy/pages/sslv3.http:

HTTP/1.0 200 OK
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<title>SSLv3 spotted</title>
<body><h1>SSLv3 spotted</h1></body>
SSLv3 forbidden for your safety:<BR>
If you want to browse this website, you should upgrade your browser.


Mitigating the shellshock vulnerability with HAProxy

Bash Shellshock vulnerability (CVE-2014-6271 and CVE-2014-7169)

Last week, a vulnerability in bash has been discovered. It is possible, under some circumstances, to inject code into a bash shell script.
It could be very dangerous if bash is used to process request sent remotely.
For now, you’re safe if no bash scripts are called by services with remote accesses.

Some reading about bash shellshock vulnerability:
  * https://access.redhat.com/announcements/1210053
  * https://access.redhat.com/articles/1212303

Today’s article is going to explain how to use HAProxy to protect your application from bash shellshock vulnerability if you’re in the case where you have to be protected.


The diagram is pretty simple. Our purpose will to detect any purposely built requests and to prevent them to reach the server:

|                                                 |
| +----------+     +---------+     +------------+ |
| |          |     |         |     |            | |
| | Attacker | +-> | HAProxy | +-> | Vulnerable | |
| |          |     |         |     | server     | |
| +----------+     +---------+     |            | |
|                                  +------------+ |
|                                                 |


Place the configuration sniplet into your HAProxy frontend configuration:

  reqdeny  ^[^:]+:s*(s*)s+{
  reqdeny  ^[^:]+:s+.*?(<<[^<;]+){5,}

Of course, your frontend must be in http mode and HAProxy must have been compiled with USE_PCRE option.

HAProxy will return a 403 if a request matches the shellshock attack.

Note: greeting to Thomas for providing the tip on HAProxy’s mailing list


binary health check with HAProxy 1.5: php-fpm/fastcgi probe example

Application layer health checking

Health checking is the ability to probe a server to ensure the service is up and running.
This is one of the root feature of any load-balancer.

One can probe servers and services at different layer of the OSI model:
  * ARP check (not available in HAProxy)
  * ICMP (ping) check (not available in HAProxy)
  * TCP (handshake) check
  * Application (HTTP, MySql, SMTP, POP, etc…) check

The most representative of the application status the check is, the best.

This means that the best way to check a service is to “speak” the protocol itself.
Unfortunately, it is impossible to write one check per protocol, there are too many protocols and some of them are proprietary and/or binary.
That’s why HAProxy 1.5 now embeds a new health checking method called “tcp-check“. It is very simple and basic “send/expect” probing method where HAProxy can send arbitrary strings and match string or regex comparison on server responses.
Many send and expect can be executed in a row to determine the status of a server.

I’ve already explained how to check redis server and how to balance traffic only to the redis master server of a cluster.

Today’s article introduces a binary protocol widely deployed: fastcgi used by php-fpm.

fastcgi binary ping/pong health check

fastcgi is a binary protocol. It means data on the network are not readable by humans, like HTTP or SMTP.
php-fpm relies on this protocol to treat PHP code. It is common to use HAProxy to load-balance many php-fpm servers for resiliency and scalability.

php-fpm Configuration

Enable a dedicated url to probe and setup the response in your php-fpm configuration:

ping.path = /ping
ping.response = pong

Which means HAProxy has to send a fastcgi request on /ping and expects the server to response with “pong“.

HAProxy health checking for php-fpm

Add the following tcp-check sequence in your php-fpm backend to probe the /ping url and ensure the server answers a “pong“.
The comment at the end of the line describe the php-cgi protocol fields.

option tcp-check
 tcp-check send-binary   01 # version
 tcp-check send-binary   01 # FCGI_BEGIN_REQUEST
 tcp-check send-binary 0001 # request id
 tcp-check send-binary 0008 # content length
 tcp-check send-binary   00 # padding length
 tcp-check send-binary   00 #
 tcp-check send-binary 0001 # FCGI responder
 tcp-check send-binary 0000 # flags
 tcp-check send-binary 0000 #
 tcp-check send-binary 0000 #
 tcp-check send-binary   01 # version
 tcp-check send-binary   04 # FCGI_PARAMS
 tcp-check send-binary 0001 # request id
 tcp-check send-binary 0045 # content length
 tcp-check send-binary   03 # padding length: padding for content % 8 = 0
 tcp-check send-binary   00 #
 tcp-check send-binary 0e03524551554553545f4d4554484f44474554 # REQUEST_METHOD = GET
 tcp-check send-binary 0b055343524950545f4e414d452f70696e67   # SCRIPT_NAME = /ping
 tcp-check send-binary 0f055343524950545f46494c454e414d452f70696e67 # SCRIPT_FILENAME = /ping
 tcp-check send-binary 040455534552524F4F54 # USER = ROOT
 tcp-check send-binary 000000 # padding
 tcp-check send-binary   01 # version
 tcp-check send-binary   04 # FCGI_PARAMS
 tcp-check send-binary 0001 # request id
 tcp-check send-binary 0000 # content length
 tcp-check send-binary   00 # padding length: padding for content % 8 = 0
 tcp-check send-binary   00 #

 tcp-check expect binary 706f6e67 # pong

Note that the whole send string could be written on a single line.

Any protocol???

If some day, you have to do the same type of configuration on a protocol nobody knows, simply capture network traffic of a “hello” sequence using tcpdump.
Then send the tcp payload cpatured using the tcp-check send command and configure the appropriate expect.
And it will work!


HAProxy and HTTP errors 408 in Chrome

Lately, there was some discussions on HAProxy’s mailing list about 408 errors printed in Chrome browsers.

Origin of 408 errors

408 is the status code used by web servers or proxies when the client has not sent a whole HTTP request during a certain period of time. It is an absolutely normal behavior!

That said, modern browsers have now some advanced features like “tcp pre-connect”…
As its name designate it, a browser may open tcp connection to web servers before you want to browse them. Purpose is to speed up browsing for the client by saving the time to establish these connections.
The problem, is that in most of the case, these connections won’t be used.
The other problem a very popular website can maintain thousands (or even tens or hundred of thousands) of connections that might be used.
It has an expensive cost on the server side for a so small positive effect on the client side.

When a web server or a proxy accept an incoming connection, in expects a request to come in the next few seconds. If a full request has not arrived after some time, the server or the proxy can close the connection and report a 408 error to the client, to tell him “you were to slow to send me a request on this connection, so I closed it”.
Unfortunately, some browsers, when they want to start using the connection mentioned above, see that data is already available for reading and print it to the user without even checking the connection state (server closing it) neither the response status code (408 in this case).

HAProxy’s timeout related to 408 errors

In HAProxy’s configuration, there are two parameters that may trigger a 408 error:

  timeout http-request 10s
  timeout client 20s

The timeout http-request is the time you let to a client to send it’s request.
If it is not set, then the timeout client will be used.

If both of them are configured, the shortest value is used.

In a default configuration, when answering with a 408, HAProxy sends a message such as this one:

HTTP/1.0 408 Request Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html><body><h1>408 Request Time-out</h1>
Your browser didn't send a complete request in time.

This message is printed by some browsers.

Workaround in HAProxy

One way to workaround this issue with HAProxy is to tell HAProxy to silently close the connection without sending any data.
You can add the line below in your defaults (or frontend/listen) section:

 errorfile 408 /dev/null

Bear in mind this is just a workaround and not a definitive solution

Related Links