Category Archives: layer7

What is a slow POST Attack and how to turn HAProxy into your first line of Defense?

One of the biggest security challenges that companies face in today’s modern climate is the POST attack. Unlike a more traditional “Denial-of-Service” attack, POST attacks target a servers logical resources – making them particularly powerful when executed.

What is a slow POST Attack?


In a POST attack, an attacker begins by sending a legitimate HTTP POST header to a Web server, exactly as they would under normal circumstances. The header specifies the exact size of the message body that will then follow. However, that message body is then sent at an alarmingly low rate – sometimes as slow as 1 byte per approximately two minutes. Because the entire message is technically correct and complete, the targeted server attempts to obey all specified rules – which as you would expect, can take quite awhile. The issue is that if an attacker were to establish hundreds or even thousands of these POST attacks simultaneously, it will quickly use up all server resources and make legitimate connections impossible.

How HAProxy can protect against slow POST attack?


Because POST attacks can be incredibly powerful, it’s always important to have a tool in place to identify these types of issues when they’re still in their nascent stages to prevent them from becoming much larger, more serious issues down the road. Because HAProxy was designed as an application delivery controller to manage Web application high availability and performance, it is already in an ideal position to stop these types of POST attacks in their tracks.

HAProxy Configuration Example

Because of HAProxy‘s structure and configuration flexibility, many professionals and consumers alike often use it as a security tool. Case in point: by using the following configuration example, you can easily help protect your servers against POST attacks to prevent attackers from clogging resources and ultimately harming the well-being of not only your equipment but your entire organization at the same time.

frontend ft_myapp
 [...]
 option http-buffer-request
 timeout http-request 10s

As you can see, with just a few simple modifications, HAProxy can quickly and effortlessly remove POST attacks from the list of things you have to worry about on a daily basis with regards to your mission-critical business applications and API.
The option http-buffer-request instructs HAProxy to wait for the whole DATA before forwarding it to a server and the timeout http-request 10s option tells how much time HAProxy let to a client to send the whole POST.

Thanks to its functionality as a security tool, a reverse proxy and more in addition to its intended functionality as a load balancer, it’s easy to see why HAProxy is used by some of the largest sites on the Internet including Reddit, Tumblr, GitHub and more on a daily basis.

This function is available in the following versions of HAProxy:

Related links

Links

HAProxy and container IP changes in Docker

HAProxy and Docker containers

Docker is a nice tool to handle containers: it allows building and running your apps in a simple and efficient way.

When used in production together with HAProxy, devops teams face a big challenge: how to followup a container IP change when restarting a container?

This blog article aims at giving a first answer to this question.

The version of docker used for this article is 1.8.1 (very important, since docker’s default behavior has changed in 1.9.0…)

HAProxy, webapp and docker diagram

The diagram below shows how docker runs on my laptop:

  • A docker0 network interface, with IP 172.16.0.1
  • all containers run in the subnet 172.16.0.0/16
+-------------------------------------host--------------------------------------+
|                                                                               |
|  +-----------------------docker-----------------------+ 172.16.0.0/16         |
|  |                                                    | docker0: 172.16.0.1   |
|  |  +---------+  +---------+       +----------+       |                       |
|  |  | HAProxy |  | appsrv1 |       | rsyslogd |       |                       |
|  |  +---------+  +---------+       +----------+       |                       |
|  |                                                    |                       |
|  +----------------------------------------------------+                       |
|                                                                               |
+-------------------------------------------------------------------------------+

The IP address associated to docker0 will be used to export some services.

In this article, we’ll start up 3 containers:

  1. rsyslogd: where HAProxy will send all its logs
  2. appsrv1: our application server, which may be restarted at any time
  3. haproxy: our load-balancer, which must follow-up appsrv1‘s IP

Building and running the lab

Building


First, we need rsyslogd and haproxy containers. They can be build from the following Dockerfiles:

Then run:

docker build -t blog:haproxy_dns ~/tmp/haproxy/blog/haproxy_docker_dns_link/blog_haproxy_dns/
docker build -t blog:rsyslogd ~/tmp/haproxy/blog/haproxy_docker_dns_link/blog_rsyslogd/

I consider appsrv container as yours: it’s your application.

Starting up our lab


Docker assign IPs to containers in the order they are started up, incrementing last byte for each new container.

To make it simpler, let’s restart docker first, so our container IPs are predictible:

sudo /etc/rc.d/docker restart

Then let’s start up our rsyslogd container:

docker run --detach --name rsyslogd --hostname=rsyslogd \
	--publish=172.16.0.1:8514:8514/udp \
	blog:rsyslogd

And let’s attach a terminal to it:

docker attach rsyslogd

Now, run appsrv container as appsrv1:

docker run --detach --name appsrv1 --hostname=appsrv1 demo:appsrv

And finally, let’s start HAProxy, with a docker link to appsrv1:

docker run --detach --name haproxy --hostname=haproxy \
	--link appsrv1:appsrv1 \
	blog:haproxy_dns

Docker links, /etc/hosts file updated and DNS


When using the ”–link” option, docker creates a new entry in the containers /etc/hosts file with the IP address and name provided by the ”link” directive.
Docker will also update this file when the remote container (here appsrv1) IP address is changed (IE when restarting the container).

If you’re familiar with HAProxy, you already know it doesn’t do file system IOs at run time. Furthermore, HAProxy doesn’t use /etc/hosts file directly. The glibc might use it when HAProxy asks for DNS resolution when parsing the configuration file. (read below for DNS resolution at runtime)

That said, if appsrv1 IP get changed, then /etc/hosts file is updated accordingly, then HAProxy is not aware of the change and the application may fail.
A quick solution would be to reload HAProxy process in its container, to force it taking into account the new IP.

A more reliable solution, is to use HAProxy 1.6 DNS resolution capability to follow-up the IP change. With this purpose in mind, we added 2 tools into our HAProxy container:

  1. dnsmasq: tiny software which can act as a DNS server which takes /etc/hosts file as its database
  2. inotifytools: watch changes on /etc/hosts file and force dnsmasq to reload it when necessary

I guess now you got it:

  • when appsrv1 is restarted, then docker gives it a new IP
  • Docker populates then this IP address into all /etc/hosts file required (those using ”link” directives)
  • Once populated, inotify tool detect the file change and triggers a dnsmasq reload
  • HAProxy periodically (can be configured) probes DNS and will get the new IP address quickly from dnsmasq

Docker container restart and HAProxy followup in action


At this stage, we should have a container attached to rsyslogd and we should be able to see HAProxy logging. Let’s give it a try:

curl http://172.16.0.4/

Nov 17 09:29:09 172.16.0.1 haproxy[10]: 172.16.0.1:55093 [17/Nov/2015:09:29:09.729] f_myapp b_myapp/appsrv1 0/0/0/1/1 200 858 - - ---- 1/1/0/1/0 0/0 "GET / HTTP/1.1"

Now, let’s consider your dev team delivered a new version of your application, so you build it and need to restart its running container:

docker restart appsrv1

and voilà:

==> /var/log/haproxy/events <==
Nov 17 09:29:29 172.16.0.1 haproxy[10]: b_myapp/appsrv1 changed its IP from 172.16.0.3 to 172.16.0.5 by docker/dnsmasq.
Nov 17 09:29:29 172.16.0.1 haproxy[10]: b_myapp/appsrv1 changed its IP from 172.16.0.3 to 172.16.0.5 by docker/dnsmasq.

Let’s test the application again:

curl http://172.16.0.4/

Nov 17 09:29:31 172.16.0.1 haproxy[10]: 172.16.0.1:59450 [17/Nov/2015:09:29:31.013] f_myapp b_myapp/appsrv1 0/0/0/0/0 200 858 - - ---- 1/1/0/1/0 0/0 "GET / HTTP/1.1"

Limitations


There are a few limitations in this mechanism:

  • it’s painful to maintaint ”link” directive when you have a 10s or 100s or more of containers….
  • the host computer, and computers in the host network can’t easily access our containers, because we don’t know their IPs and their hostnames are resolved in HAProxy container only
  • if we want to add more appserver in HAProxy‘s farm we still need to restart HAProxy‘s container (and update configuration accordingly)

To fix some of the issues above, we can dedicate a container to perform DNS resolution within our docker world and deliver responses to any running containers or hosts in the network. We’ll see that in a next blog article

Links

HAProxy and HTTP Strict Transport Security (HSTS) header in HTTP redirects

SSL/TLS and HSTS

SSL everywhere is on its way.
Unfortunately, many applications were written for HTTP only and switching to HTTPs is not an easy and straight forward path. Read more here about impact of TLS offloading (when a third party tool perform TLS in front of your web application servers).

A mechanism called HTTP Strict Transport Security (HSTS) has been introduced through the RFC 6797.

HSTS main purpose is to let the application server to instruct the client it’s supposed to get connected only a ciphered and secured HTTPs connection when browsing the application.
It means that of course, that both the client and the server must be compatible…
That way, the application cookie is protected on its way from the client’s browser to the remote TLS endpoint (either the load-balancer or the application server). No cookie hijacking is possible on the wire.

HAProxy configuration for Strict-Transport-Security HTTP header

HSTS header insertion in server responses

To insert the header in every server response, you can use the following HAProxy directive, in HAProxy 1.5:

 # 16000000 seconds: a bit more than 6 months
 http-response set-header Strict-Transport-Security max-age=16000000;\ includeSubDomains;\ preload;

With the upcoming HAProxy 1.6, and thanks to William’s work, we can now get rid of these ugly backslashes:

 # 16000000 seconds: a bit more than 6 months
 http-response set-header Strict-Transport-Security "max-age=16000000; includeSubDomains; preload;"

Inserting HSTS header in HTTP redirects


When HAProxy has to perform HTTP redirects, it does in at the moment of the client request, through the http-request rules.
Since we want to insert a header in the response, we can use the http-response rules. Unfortunately, these rules are enabled when HAProxy get traffic from a backend server.
Here is the trick: we do perform the http-request redirect rule in a dedicated frontend where we route traffic to. That way, our application backend or frontend can perform HSTS insertion.

A simple configuration sniplet is usually easier to explain:

frontend fe_myapp
 bind :443 ssl crt /path/to/my/cert.pem
 bind :80
 use_backend be_dummy if !{ ssl_fc }
 default_backend be_myapp

backend be_myapp
 http-response set-header Strict-Transport-Security max-age=16000000;\ includeSubDomains;\ preload;
 server s1 10.0.0.1:80

be_dummy
 server haproxy_fe_dummy_ssl_redirect 127.0.0.1:8000

frontend fe_dummy
 bind 127.0.0.1:8000
 http-request redirect scheme https

Links

Packetshield: quand votre load-balancer vous protège contre les DDOS!

Les attaques par DDOS


Il y a quelques temps, nous avions publié sur ce blog, un article expliquant comment utiliser un load-balancer pour se protéger contre les attaques de type DDOS applicatif: http://blog.haproxy.com/2012/02/27/use-a-load-balancer-as-a-first-row-of-defense-against-ddos/.

Malheureusement, les attaques sont aujourd’hui sur plusieurs vecteurs, notamment réseau et applicatif.
Leur but est toujours le même: trouver un point faible dans l’architecture et le saturer:

  1. un firewall, nombre de sessions ou paquets par secondes
  2. un load-balancer, nombre sessions établies
  3. un serveur applicatif, nombre de requêtes par secondes
  4. etc…

Dans l’article précédent, nous avions vu comment protéger les serveurs d’applications en utilisant le load-balancer, ce qui correspond à la partie applicative de l’architecture.

Pour protéger la tête de l’architecture, le firewall, il faut en général investir beaucoup d’argent dans des équipements coûteux.
Par ailleurs, dans certains cas, lorsque l’ALOHA était connecté en direct sur Internet, il ne savait se protéger que modérément contre les attaques “réseau”.

C’est pourquoi, HAProxy Technologies a développé sa propre solution de protection contre les DDOS de niveau réseau. Cet outil s’appelle Packetshield.

Les attaques de niveau réseau sont assez simples à mettre en oeuvre pour l’attaquant, mais peuvent être dévastatrices pour la cible: lorsqu’un équipement est saturé, il bloque tout le trafic.
Un équipement peut se saturer de différentes manières:

  • un trop grand nombre de paquets par secondes
  • un trop grand nombre se sessions ouvertes (TCP et/ou UDP)
  • un trop gros volume de trafic entrant

(liste non exhaustive)

NOTE: pour ce dernier cas, il faut passer par un service externe de “nettoyage” du flux, qui renverra ensuite un trafic légitime vers votre architecture.

Packetshield: Protection contre les DDOS réseau

L’ALOHA, équipé de packetshield, se place en point d’entrée de l’infrastructure.
Packetshield va agir comme un Firewall stateful en analysant les flux réseaux le traversant. Seuls le trafic considéré comme légitime sera autorisé à passer.
Différents mécanismes rentrent en jeu lors de la protection:

  • analyse des paquets: les paquets non valides sont automatiquement bloqués
  • filtrage de flux: l’administrateur peut définir ce qui est autorisé ou pas
  • analyse des flux autorisés: Packetshield fait le trie pour savoir ce qui est légitime ou pas

Contrairement aux solutions existantes qui fonctionnent dans un mode stateless et qui génèrent beaucoup de faux positifs, Packetshield garantie qu’aucune session valide ne sera bloquée.

Disponibilité de Packetshield

Packetshield est disponible dans l’ALOHA en version appliance physique depuis le firmware 7.0.
La protection de l’interface gigabit (Max 1 million de paquets par secondes) est inclu gratuitement dans l’ALOHA.
Nos clients existants n’ont qu’à mettre à jour le firmware de leur appliance pour en bénéficier. Tout nouveau client achetant l’une de nos appliances bénéficiera lui aussi de cette protection, sans surcoût.

Packetshield est aussi disponible en version 10G (14 millions de paquets traités par secondes) sur nos toutes appliances dernier modèle ALB5100, peut importe la licence choisie, du 8K au 64K.

Mode de déploiement


De part son fonctionnement stateful, Packetshield est utilisé de manière optimal dans les déploiement suivant:

Packetshield peut fonctionner en mode stateless, sans ses fonctionnalités avancé. Dans ce mode, Packetshield est compatible avec les type de déploiement suivants:

Pour plus d’information

La documentation sur la configuration de Packetshield est disponible à cette adresse: http://haproxy.com/doc/aloha/7.0/packet_flood_protection/.

Pour plus d’information, un petit mail à contact (at) haproxy (point) com.

Liens

HAProxy’s load-balancing algorithm for static content delivery with Varnish

HAProxy’s load-balancing algorithms

HAProxy supports many load-balancing algorithms which may be used in many different type of cases.
That said, cache servers, which deliver most of the time the static content from your web applications, may require some specific load-balancing algorithms.

HAProxy stands in front of your cache server for some good reasons:

  • SSL offloading (read PHK’s feeling about SSL, Varnish and HAProxy)
  • HTTP content switching capabilities
  • advanced load-balancing algorithms

The main purpose of this article is to show how HAProxy can be used to aggregate Varnish servers memory storage in some kind of “JBOD” mode (like the “Just a Bunch Of Disks“).
Main purpose of the examples delivered here are to optimize the resources on the cache, mainly its memory, in order to improve the HIT rate. This will also improve your application response time and make your site top ranked on google 🙂

Content Switching in HAProxy

This has been covered many times on this blog.
As a quick introduction for readers who are not familiar with HAProxy, let’s explain how it works.

Clients will get connected to HAProxy through a Frontend. Then HAProxy routes traffic to a backend (server farm) where the load-balancing algorithm is used to choose a server.
A frontend can points to multiple backends and the choice of a backend is made through acls and use_backend rules..
Acls can be formed using fetches. A fetch is a directive which instructs HAProxy where to get content from.

Enough theory, let’s make a practical example: splitting static and dynamic traffic using the following rules:

  • Static content is hosted on domain names starting by ‘static.’ and ‘images.’
  • Static content files extensions are ‘.jpg’ ‘.png’ ‘.gif’ ‘.css’ ‘.js’
  • Static content can match any of the rule above
  • anything which is not static is considered as dynamic

The configuration sniplet below should be integrated into the HAProxy frontend. It matches the rules above to do traffic splitting. The varnish servers will stands in the bk_static farm.

frontend ft_public
 <frontend settings>
 acl static_domain  req.hdr_beg(Host) -i static. images.
 acl static_content path_end          -i .jpg .png .gif .css .js
 use_backend bk_static if static_domain or static_content
 default_backend bk_dynamic
   
backend bk_static
 <parameters related to static content delivery>

The configuration above creates 2 named acls ‘static_domain‘ and ‘static_content‘ which are used by the used_backend rule to route the traffic to varnish servers.

HAProxy and hash based load-balancing algotithm


Later in this article, we’ll heavily used the hash based load-balancing algorithms from HAProxy.
So a few information here (non exhaustive, it would deserve a long blog article) which will be useful for people wanting to understand what happens deep inside HAProxy.

The following parameters are taken into account when computing a hash algorithm:

  • number of servers in the farm
  • weight of each server in the farm
  • status of the servers (UP or DOWN)

If any of the parameter above changes, the whole hash computation also changes, hence request may hit an other server. This may lead to a negative impact on the response time of the application (during a short period of time).
Fortunately, HAProxy allows ‘consistent’ hashing, which means that only the traffic related to the change will be impacted.
That’s why you’ll see a lot of hash-type consistent directives in the configuration samples below.

Load-Balancing varnish cache server

Now, let’s focus on the magic we can add in the bk_static server farm.

Hashing the URL

HAProxy can hash the URL to pick up a server. With this load-balancing algorithm, we guarantee that a single URL will always hit the same Varnish server.

hashing the URL path only


In the example below, HAProxy hashes the URL path, which is from the first slash ‘/’ character up to the question mark ‘?’:

backend bk_static
  balance uri
  hash-type consistent

hashing the whole url, including the query string


In some cases, the query string may contain some variables in the query string, which means we must include the query string in the hash:

backend bk_static
  balance uri whole
  hash-type consistent

Query string parameter hash


That said, in some cases (API, etc…), hashing the whole URL is not enough. We may want to hash only on a particular query string parameter.
This applies well in cases where the client can forge itself the URL and all the parameters may be randomly ordered.
The configuration below tells HAProxy to apply the hash to the query string parameter named ‘id’ (IE: /image.php?width=512&id=12&height=256)

backend bk_static
  balance url_param id
  hash-type consistent

hash on a HTTP header


HAProxy can apply the hash to a specific HTTP header field.
The example below applies it on the Host header. This can be used for people hosting many domain names with a few pages, like users dedicated pages.

backend bk_static
  balance hdr(Host)
  hash-type consistent

Compose your own hash: concatenation of Host header and URL


Nowadays, HAProxy becomes more and more flexible and we can use this flexibility in its configuration.
Imagine, in your varnish configuration, you have a storage hash key based on the concatenation of the host header and the URI, then you may want to apply the same load-balancing algorithm into HAProxy, to optimize your caches.

The configuration below creates a new HTTP header field named X-LB which contains the host header (converted to lowercase) concatenated to the request uri (converted in lowercase too).

backend bk_static
  http-request set-header X-LB %[req.hdr(Host),lower]%[req.uri,lower]
  balance hdr(X-LB)
  hash-type consistent

Conclusion


HAProxy and Varnish works very well together. Each soft can benefit from performance and flexibility of the other one.

Links

A HTTP monitor which matches multiple conditions in HAProxy

Load-Balancing and health checking

Health checking is the method to check a service availability on a server.
It is one of the most important feature of a load-balancer. How could we balance traffic amongst servers if we don’t know if the service is alive???

HAProxy and HTTP check

HAProxy can probe HTTP applications using httpchk option.
This option can be customized using the http-check expect directive to match different status codes or content.
That said, a single http-check expect rule can match.
So we can’t match a status code and the presence of a string in the page, for example.

Make HAProxy match multiple conditions for HTTP health checking

The solution is to use to the raw tcp-check and write a health check script sequence which match all the conditions.

For example, you want to ensure the server’s response has:

  • HTTP status code is 200
  • absence of keyword Error
backend myapp
[...]
 option tcp-check
 tcp-check send GET /my/check/url HTTP/1.1rn
 tcp-check send Host: myhostrn
 tcp-check send Connection: closern
 tcp-check send rn
 tcp-check expect string HTTP/1.1 200 OK
 tcp-check expect ! string Error
[...]

Links

Web application name to backend mapping in HAProxy

Synopsis

Let’s take a web application platform where many HTTP Host header points to.
Of course, this platform hosts many backends and HAProxy is used to perform content switching based on the Host header to route HTTP traffic to each backend.

HAProxy map


HAProxy 1.5 introduced a cool feature: converters. One converter type is map.
Long story made short: a map allows to map a data in input to an other one on output.

A map is stored in a flat file which is loaded by HAProxy on startup. It is composed by 2 columns, on the left the input string, on the right the output one:

in out

Basically, if you call the map above and give it the in strings, it will return out.

Mapping

Now, the interesting part of the article 🙂

As stated in introduction, we want to map hundreds of Host headers to tens of backends.

The old way of mapping: acl and use_backend rules

Before the map, we had to use acls and use_backend rules.

like below:

frontend ft_allapps
 [...]
 use_backend bk_app1 if { hdr(Host) -i app1.domain1.com app1.domain2.com }
 use_backend bk_app2 if { hdr(Host) -i app2.domain1.com app2.domain2.com }
 default_backend bk_default

Add one statement per use_backend rule.

This works nicely for a few backends and a few domain names. But this type of configuration is hardly scallable…

The new way of mapping: one map and one use_backend rule

Now we can use map to achieve the same purpose.

First, let’s create a map file called domain2backend.map, with the following content: on the left, the domain name, on the right, the backend name:

#domainname  backendname
app1.domain1.com bk_app1
app1.domain2.com bk_app1
app2.domain1.com bk_app2
app2.domain2.com bk_app2

And now, HAProxy configuration:

frontend ft_allapps
 [...]
 use_backend %[req.hdr(host),lower,map_dom(/etc/hapee-1.5/domain2backend.map,bk_default)]

Here is what HAProxy will do:

  1. req.hdr(host) ==> fetch the Host header from the HTTP request
  2. lower ==> convert the string into lowercase
  3. map_dom(/etc/hapee-1.5/domain2backend.map) ==> look for the lowercase Host header in the map and return the backend name if found. If not found, the name of a default backend is returned
  4. route traffic to the backend name returned by the map

Now, adding a new content switching rule means just add one new line in the map content (and reload HAProxy). No regexes, map data is stored in a tree, so processing time is very low compared to matching many string in many ACLs for many use_backend rules.

simple is beautiful!!!

HAProxy map content auto update


If you are an HAPEE user (and soon available for the ALOHA), you can use the lb-update content to download the content of the map automatically.
Add the following statement in your configuration:

dynamic-update
 update id domain2backend.map url https://10.0.0.1/domain2backend.map delay 60s timeout 5s retries 3 map

Links