binary health check with HAProxy 1.5: php-fpm/fastcgi probe example

Application layer health checking

Health checking is the ability to probe a server to ensure the service is up and running.
This is one of the root feature of any load-balancer.

One can probe servers and services at different layer of the OSI model:
  * ARP check (not available in HAProxy)
  * ICMP (ping) check (not available in HAProxy)
  * TCP (handshake) check
  * Application (HTTP, MySql, SMTP, POP, etc…) check

The most representative of the application status the check is, the best.

This means that the best way to check a service is to “speak” the protocol itself.
Unfortunately, it is impossible to write one check per protocol, there are too many protocols and some of them are proprietary and/or binary.
That’s why HAProxy 1.5 now embeds a new health checking method called “tcp-check“. It is very simple and basic “send/expect” probing method where HAProxy can send arbitrary strings and match string or regex comparison on server responses.
Many send and expect can be executed in a row to determine the status of a server.

I’ve already explained how to check redis server and how to balance traffic only to the redis master server of a cluster.

Today’s article introduces a binary protocol widely deployed: fastcgi used by php-fpm.

fastcgi binary ping/pong health check

fastcgi is a binary protocol. It means data on the network are not readable by humans, like HTTP or SMTP.
php-fpm relies on this protocol to treat PHP code. It is common to use HAProxy to load-balance many php-fpm servers for resiliency and scalability.

php-fpm Configuration

Enable a dedicated url to probe and setup the response in your php-fpm configuration:

ping.path = /ping
ping.response = pong

Which means HAProxy has to send a fastcgi request on /ping and expects the server to response with “pong“.

HAProxy health checking for php-fpm

Add the following tcp-check sequence in your php-fpm backend to probe the /ping url and ensure the server answers a “pong“.
The comment at the end of the line describe the php-cgi protocol fields.

option tcp-check
 # FCGI_BEGIN_REQUEST
 tcp-check send-binary   01 # version
 tcp-check send-binary   01 # FCGI_BEGIN_REQUEST
 tcp-check send-binary 0001 # request id
 tcp-check send-binary 0008 # content length
 tcp-check send-binary   00 # padding length
 tcp-check send-binary   00 #
 tcp-check send-binary 0001 # FCGI responder
 tcp-check send-binary 0000 # flags
 tcp-check send-binary 0000 #
 tcp-check send-binary 0000 #
 # FCGI_PARAMS
 tcp-check send-binary   01 # version
 tcp-check send-binary   04 # FCGI_PARAMS
 tcp-check send-binary 0001 # request id
 tcp-check send-binary 0045 # content length
 tcp-check send-binary   03 # padding length: padding for content % 8 = 0
 tcp-check send-binary   00 #
 tcp-check send-binary 0e03524551554553545f4d4554484f44474554 # REQUEST_METHOD = GET
 tcp-check send-binary 0b055343524950545f4e414d452f70696e67   # SCRIPT_NAME = /ping
 tcp-check send-binary 0f055343524950545f46494c454e414d452f70696e67 # SCRIPT_FILENAME = /ping
 tcp-check send-binary 040455534552524F4F54 # USER = ROOT
 tcp-check send-binary 000000 # padding
 # FCGI_PARAMS
 tcp-check send-binary   01 # version
 tcp-check send-binary   04 # FCGI_PARAMS
 tcp-check send-binary 0001 # request id
 tcp-check send-binary 0000 # content length
 tcp-check send-binary   00 # padding length: padding for content % 8 = 0
 tcp-check send-binary   00 #

 tcp-check expect binary 706f6e67 # pong

Note that the whole send string could be written on a single line.

Any protocol???


If some day, you have to do the same type of configuration on a protocol nobody knows, simply capture network traffic of a “hello” sequence using tcpdump.
Then send the tcp payload cpatured using the tcp-check send command and configure the appropriate expect.
And it will work!

Links

Posted in Aloha, HAProxy | Tagged , , , , | Leave a comment

HAProxy and HTTP errors 408 in Chrome

Lately, there was some discussions on HAProxy’s mailing list about 408 errors printed in Chrome browsers.

Origin of 408 errors


408 is the status code used by web servers or proxies when the client has not sent a whole HTTP request during a certain period of time. It is an absolutely normal behavior!

That said, modern browsers have now some advanced features like “tcp pre-connect”…
As its name designate it, a browser may open tcp connection to web servers before you want to browse them. Purpose is to speed up browsing for the client by saving the time to establish these connections.
The problem, is that in most of the case, these connections won’t be used.
The other problem a very popular website can maintain thousands (or even tens or hundred of thousands) of connections that might be used.
It has an expensive cost on the server side for a so small positive effect on the client side.

When a web server or a proxy accept an incoming connection, in expects a request to come in the next few seconds. If a full request has not arrived after some time, the server or the proxy can close the connection and report a 408 error to the client, to tell him “you were to slow to send me a request on this connection, so I closed it”.
Unfortunately, some browsers, when they want to start using the connection mentioned above, see that data is already available for reading and print it to the user without even checking the connection state (server closing it) neither the response status code (408 in this case).

HAProxy’s timeout related to 408 errors


In HAProxy’s configuration, there are two parameters that may trigger a 408 error:

  timeout http-request 10s
  timeout client 20s

The timeout http-request is the time you let to a client to send it’s request.
If it is not set, then the timeout client will be used.

If both of them are configured, the shortest value is used.

In a default configuration, when answering with a 408, HAProxy sends a message such as this one:

HTTP/1.0 408 Request Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html><body><h1>408 Request Time-out</h1>
Your browser didn't send a complete request in time.
</body></html>

This message is printed by some browsers.

Workaround in HAProxy


One way to workaround this issue with HAProxy is to tell HAProxy to silently close the connection without sending any data.
You can add the line below in your defaults (or frontend/listen) section:

 errorfile 408 /dev/null

Bear in mind this is just a workaround and not a definitive solution

Related Links

Links

Posted in HAProxy | Tagged , , , | 4 Comments

Howto write apache ProxyPass* rules in HAProxy

Apache mod_proxy

Apache webserver is a widely deployed modular web server.
One of its module is called mod_proxy. It aims to turn the web server into a proxy / reverse proxy server with load-balancing capabilities.

At HAProxy Technologies, we only use HAProxy :). Heh, what else ???
And during some deployments, customers ask us to migrate Apache mod_proxy configuration into HAProxy.
Present article explains how to translate ProxyPass related rules.

ProxyPass, ProxyPassReverse, etc…

Apache mod_proxy defines a few directives which let it forward traffic to a remote server.
They are listed below with a short description.

Note: I know the directives introduced in this article could be used in much complicated ways.

ProxyPass

ProxyPass maps local server URLs to remote servers + URL.
It applies on traffic from client to server.

For example:

ProxyPass /mirror/foo/ http://backend.example.com/

This makes the external URL http://example.com/mirror/foo/bar to be translated and forwarded to a remote server this way: http://backend.example.com/bar

This directive makes apache to update URL and headers to match both external traffic to internal needs.

ProxyPassReverse

ProxyPassReverse Adjusts the URL in HTTP response headers sent from a reverse proxied server. It only updates Location, Content-Location and URL.
It applies to traffic from server to client.

For example:

ProxyPassReverse /mirror/foo/ http://backend.example.com/

This directive makes apache to adapt responses generated by servers following internal urls to match external urls.

ProxyPassReverseCookieDomain

ProxyPassReverseCookieDomain adjusts the Set-Cookie header sent by the server to match external domain name.

It’s usage is pretty simple. For example:

ProxyPassReverseCookieDomain internal-domain public-domain

ProxyPassReverseCookiePath

ProxyPassReverseCookiePath adjusts the Set-Cookie header sent by the server to match external path.

It’s usage is pretty simple. For example:

ProxyPassReverseCookiePath internal-path public-path

Configure ProxyPass and ProxyPassReverse in HAProxy

Bellow, an example HAProxy configuration to make HAProxy work the same way as apache ProxyPass and ProxyPassReverse configuration. It should be added in the backend section while the frontend ensure that only traffic matching this external URL would be redirected to that backend.

frontend ft_global
 acl host_dom.com    req.hdr(Host) dom.com
 acl path_mirror_foo path -m beg   /mirror/foo/
 use_backend bk_myapp if host_dom.com path_mirror_foo

backend bk_myapp
[...]
# external URL                  => internal URL
# http://dom.com/mirror/foo/bar => http://bk.dom.com/bar

 # ProxyPass /mirror/foo/ http://bk.dom.com/bar
 http-request set-header Host bk.dom.com
 reqirep  ^([^\ :]*)\ /mirror/foo/(.*)     \1\ /\2

 # ProxyPassReverse /mirror/foo/ http://bk.dom.com/bar
 # Note: we turn the urls into absolute in the mean time
 acl hdr_location res.hdr(Location) -m found
 rspirep ^Location:\ (https?://bk.dom.com(:[0-9]+)?)?(/.*) Location:\ /mirror/foo\3 if hdr_location

 # ProxyPassReverseCookieDomain bk.dom.com dom.com
 acl hdr_set_cookie_dom res.hdr(Set-cookie) -m sub Domain:\ bk.dom.com
 rspirep ^(Set-Cookie:.*)\ Domain:\ bk.dom.com(.*) \1\ Domain:\ dom.com\2 if hdr_set_cookie_dom

 # ProxyPassReverseCookieDomain / /mirror/foo/
 acl hdr_set_cookie_path res.hdr(Set-cookie) -m sub Path:\ 
 rspirep ^(Set-Cookie:.*)\ Path:\ (.*) \1\ Path:\ /mirror/foo\2 if hdr_set_cookie_path

Notes:
  * http to https redirect rules should be handled by HAProxy itself and not by the application server (to avoid some redirect loops)

Links

Posted in HAProxy | Tagged , | Leave a comment

Asymmetric routing, multiple default gateways on Linux with HAProxy

Why we may need multiple default gateways?

Nowadays, Application Delivery controllers (aka Load-Balancers) become the entry point for all the applications hosted in a company or administration.
That said, many different type of population could access the applications:
  * internal users from the LAN
  * partners through MPLS or VPNs
  * external users from internet

On the other side, applications could be hosted on different VLANs in the architecture:
  * internal LAN
  * external DMZ

The diagram below shows the “big picture” of this type of architecture:
multiple_default_gateways

Routing in the Linux network stack

I’m not going to deeply explain how it works, sorry… It would deserve a complete blog post :)
That said, any device connected on an IP network needs an IP address to be able to talk to other devices in its LAN. It also needs a default gateway to be able to reach devices which are located outside its LAN.
A Linux kernel can use a single default gateway at a time, but thanks to the metric you can configure many default gateways.
When needed, the Linux Kernel will parse the default gateway table and will use the one with the lowest metric. By default, when no metric is configured, the kernel attributes a metric 0.
Each metric must be unique in your Kernel IP stack.

How HAProxy can help in such situation??


Users access applications through a HAProxy bind. The bind can be hosted on any IP address available or not (play with your sysctl for this purpose) on the server.
By default, the traffic comes in HAProxy through this bind and HAProxy let the kernel choose the most appropriate default gateway to forward the answer to the client. As we’ve seen above, the most appropriate default gateway from the kernel point of view is the one with the lowest metric usually 0.

That said, HAProxy is smart enough to tell the kernel which network interface to use to forward the response to the client. Just add the statement interface ethX (where X is the id of the interface you want to use) on HAProxy bind line.
With this parameter, HAProxy can force the kernel to use the default gateway associated to the network interface ethX if it exists, otherwise, the interface with the lowest metric will be used.

Security concern


From a security point of view, some security manager would say that it is absolutely unsecure to plug a device in multiple DMZ or VLANs. They are right. But usually, this type of company’s business is very important and they can affoard one load-balancer per DMZ or LAN.
That said, there is no security breach with the setup introduced here. HAProxy is a reverse-proxy and so you don’t need to allow ip_forward between all interfaces for this solution to work.
I mean that nobody could use the loadbalancer as a default gateway to reach an other subnet bypassing the firewall…
Then only traffic allowed to pass through is the one load-balanced!

Configuration

The configuration below applies to the ALOHA Loadbalancer. Just update the content to match your Linux distribution configuration syntax.
The configuration is also related to the diagram above.

Network configuration


In your ALOHA, go in the Services tab, then edit the Network configuration.
To keep it simple, I’m not going to add any VRRP configuration.

service network eth0
    ########## eth0.
    auto on
    ip   address 10.0.0.2/24
    ip   route   default 10.0.0.1

service network eth1
    ########## eth1.
    auto on
    ip   address 10.0.1.2/24
    ip   route   default 10.0.1.1 metric 1

service network eth2
    ########## eth2.
    auto on
    ip   address 10.0.2.2/24
    ip   route   default 10.0.2.1 metric 2

service network eth3
    ########## eth3.
    auto on
    ip   address 10.0.3.2/24
    ip   route   default 10.0.3.1 metric 3

service network eth4
    ########## eth4.
    auto on
    ip   address 10.0.4.2/24
    ip   route   default 10.0.4.1 metric 4

The routing table from the ALOHA looks like:

default via 10.0.0.1 dev eth0
default via 10.0.1.1 dev eth1  metric 1
default via 10.0.2.1 dev eth2  metric 2
default via 10.0.3.1 dev eth3  metric 3
default via 10.0.4.1 dev eth4  metric 4

HAProxy configuration for Corporate website or ADFS proxies


These services are used by internet users only.

frontend ft_www
 bind 10.0.0.2:80
[...]

no need to specify any interface here, since the traffic comes from internet, HAProxy can let the kernel to use the default gateway which points in that direction (here eth0).

HAProxy configuration for Exchange 2010 or 2013


This service is used by both internal and internet users.

frontend ft_exchange
 bind 10.0.0.3:443
 bind 10.0.2.3:443 interface eth2
[...]

The responses to internet users will go through eth0 while the one for internal LAN users will use the default gateway configured on eth2 10.0.2.1.

HAProxy configuration for Sharepoint 2010 or 2013


This service is used by MPLS/VPN users and internal users.

frontend ft_exchange
 bind 10.0.1.4:443 interface eth1
 bind 10.0.2.4:443 interface eth2
[...]

The responses to MPLS/VPN users will go through eth1 default gateway 10.0.1.1 while the one for internal LAN users will use the default gateway configured on eth2 10.0.2.1.

Links

Posted in Aloha, architecture, HAProxy, layer7 | Tagged , , , , , | 2 Comments

How to protect application cookies while offloading SSL

SSL offloading

SSL offloading or acceleration is often seen as a huge benefit for applications. People usually forget that it may have impacts on the application itself. Some times ago, I wrote a blog article which lists these impacts and propose some solutions, using HAProxy.

One thing I forgot to mention at that time was Cookies.
You don’t want your clients to send their cookies (understand their identity) in clear through the Internet.
This is today’s article purpose.

Actually, there is a cookie attribute called Secure which can be emit by a server. When this attribute is set, the client SHOULD not send the cookie over a clear HTTP connection.

SSL offloading Diagram


Simple SSL offloading diagram:

|--------|              |---------|           |--------|
| client |  ==HTTPS==>  | HAProxy | --HTTP--> | Server |
|--------|              |---------|           |--------|

The client uses HTTPs to get connected on HAProxy, HAProxy gets connected to the application server through HTTP.

Even if HAProxy can forward client connection mode information to the application server, the application server may not protect its cookie…
Fortunately, we can use HAProxy for this purpose.

Howto make HAProxy to protect application cookie when SSL offloading is enabled

That’s the question.

The response is as simple as the configuration below:

acl https          ssl_fc
acl secured_cookie res.hdr(Set-Cookie),lower -m sub secure
rspirep ^(set-cookie:.*) \1;\ Secure if https !secured_cookie

The configuration above sets up the Secure attribute if it has not been setup by the application server while the client was browsing the application over a ciphered connection.

Related Links

Links

Posted in Uncategorized | Leave a comment

Emulating Active/passing application clustering with HAProxy

Synopsis

HAProxy is a Load-Balancer, this is a fact. It is used to route traffic to servers to primarily ensure applications reliability.

Most of the time, the sessions are locally stored in a server. Which means that if you want to split client traffic on multiple servers, you have to ensure each user can be redirected to the server which manages his session (if the server is available, of course). HAProxy can do this in many ways: we call it persistence.
Thanks to persistence, we usually says that any application can be load-balanced… Which is true in 99% of the cases. In very rare cases, the application can’t be load-balanced. I mean that there might be a lock somewhere in the code or for some other good reasons…

In such case, to ensure high-availability, we build “active/passive” clusters, where a node can be active at a time.
HAProxy can be use in different ways to emulate an active/passive clustering mode, and this is the purpose of today’s article.

Bear in mind that by “active/passive”, I mean that 100% of the users must be forwarded to the same server. And if a fail over occurs, they must follow it in the mean time!

Diagram

Let’s use one HAProxy with a couple of servers, s1 and s2.
When starting up, s1 is master and s2 is used as backup:

  -------------
  |  HAProxy  |
  -------------
   |         `
   |active    ` backup
   |           `
 ------       ------
 | s1 |       | s2 |
 ------       ------

Configuration

Automatic failover and failback

The configuration below makes HAProxy to use s1 when available, otherwise fail over to s2 if available:

defaults
 mode http
 option http-server-close
 timeout client 20s
 timeout server 20s
 timeout connect 4s

frontend ft_app
 bind 10.0.0.100:80 name app
 default_backend bk_app

backend bk_app
 server s1 10.0.0.1:80 check
 server s2 10.0.0.2:80 check backup

The most important keyword above is “backup” on s2 configuration line.
Unfortunately, as soon as s1 comes back, then all the traffic will fail back to it again, which can be acceptable for web applications, but not for active/passive

Automatic failover without failback

The configuration below makes HAProxy to use s1 when available, otherwise fail over to s2 if available.
When a failover has occured, no failback will be processed automatically, thanks to the stick table:

peers LB
 peer LB1 10.0.0.98:1234
 peer LB2 10.0.0.99:1234

defaults
 mode http
 option http-server-close
 timeout client 20s
 timeout server 20s
 timeout connect 4s

frontend ft_app
 bind 10.0.0.100:80 name app
 default_backend bk_app

backend bk_app
 stick-table type ip size 1 peers LB
 stick on dst
 server s1 10.0.0.1:80 check
 server s2 10.0.0.2:80 check backup

The stick table will maintain persistence based on destination IP address (10.0.0.100 in this case):

show table bk_app
# table: bk_app, type: ip, size:20480, used:1
0x869154: key=10.0.0.100 use=0 exp=0 server_id=1

With such configuration, you can trigger a fail back by disabling s2 during a few second period.

Links

Posted in Aloha, architecture, HAProxy | Tagged , , | 9 Comments

HAProxy advanced Redis health check

Introduction

Redis is an opensource nosql database working on a key/value model.
One interesting feature in Redis is that it is able to write data to disk as well as a master can synchronize many slaves.

HAProxy can load-balance Redis servers with no issues at all.
There is even a built-in health check for redis in HAProxy.
Unfortunately, there was no easy way for HAProxy to detect the status of a redis server: master or slave node. Hence people usually hacks this part of the architecture.

As written in the title of this post, we’ll learn today how to make a simple Redis infrastructure thanks to newest HAProxy advanced send/expect health checks.
This feature is available in HAProxy 1.5-dev20 and above.

Purpose is to make the redis infrastructure as simple as possible and ease fail over for the web servers. HAProxy will have to detect which node is MASTER and route all the connection to it.

Redis high availability diagram with HAProxy

Below, an ascii art diagram of HAProxy load-balancing Redis servers:

+----+ +----+ +----+ +----+
| W1 | | W2 | | W3 | | W4 |   Web application servers
+----+ +----+ +----+ +----+
     \     |   |     /
      \    |   |    /
       \   |   |   /
        +---------+
        | HAProxy |
        +---------+
           /   \
       +----+ +----+
       | R1 | | R2 |           Redis servers
       +----+ +----+

The scenario is simple:
  * 4 web application servers need to store and retrieve data to/from a Redis database
  * one (better using 2) HAProxy servers which load-balance redis connections
  * 2 (at least) redis servers in an active/standby mode with replication

Configuration

Below, is the HAProxy configuration for the

defaults REDIS
 mode tcp
 timeout connect  4s
 timeout server  30s
 timeout client  30s

frontend ft_redis
 bind 10.0.0.1:6379 name redis
 default_backend bk_redis

backend bk_redis
 option tcp-check
 tcp-check send PING\r\n
 tcp-check expect string +PONG
 tcp-check send info\ replication\r\n
 tcp-check expect string role:master
 tcp-check send QUIT\r\n
 tcp-check expect string +OK
 server R1 10.0.0.11:6379 check inter 1s
 server R2 10.0.0.12:6379 check inter 1s

The HAProxy health check sequence above allows to consider the Redis master server as UP in the farm and redirect connections to it.
When the Redis master server fails, the remaining nodes elect a new one. HAProxy will detect it thanks to its health check sequence.

It does not require third party tools and make fail over transparent.

Links

Posted in architecture, HAProxy, optimization | Tagged | 7 Comments