Tag Archives: ddos

What is a slow POST Attack and how to turn HAProxy into your first line of Defense?

One of the biggest security challenges that companies face in today’s modern climate is the POST attack. Unlike a more traditional “Denial-of-Service” attack, POST attacks target a servers logical resources – making them particularly powerful when executed.

What is a slow POST Attack?


In a POST attack, an attacker begins by sending a legitimate HTTP POST header to a Web server, exactly as they would under normal circumstances. The header specifies the exact size of the message body that will then follow. However, that message body is then sent at an alarmingly low rate – sometimes as slow as 1 byte per approximately two minutes. Because the entire message is technically correct and complete, the targeted server attempts to obey all specified rules – which as you would expect, can take quite awhile. The issue is that if an attacker were to establish hundreds or even thousands of these POST attacks simultaneously, it will quickly use up all server resources and make legitimate connections impossible.

How HAProxy can protect against slow POST attack?


Because POST attacks can be incredibly powerful, it’s always important to have a tool in place to identify these types of issues when they’re still in their nascent stages to prevent them from becoming much larger, more serious issues down the road. Because HAProxy was designed as an application delivery controller to manage Web application high availability and performance, it is already in an ideal position to stop these types of POST attacks in their tracks.

HAProxy Configuration Example

Because of HAProxy‘s structure and configuration flexibility, many professionals and consumers alike often use it as a security tool. Case in point: by using the following configuration example, you can easily help protect your servers against POST attacks to prevent attackers from clogging resources and ultimately harming the well-being of not only your equipment but your entire organization at the same time.

frontend ft_myapp
 [...]
 option http-buffer-request
 timeout http-request 10s

As you can see, with just a few simple modifications, HAProxy can quickly and effortlessly remove POST attacks from the list of things you have to worry about on a daily basis with regards to your mission-critical business applications and API.
The option http-buffer-request instructs HAProxy to wait for the whole DATA before forwarding it to a server and the timeout http-request 10s option tells how much time HAProxy let to a client to send the whole POST.

Thanks to its functionality as a security tool, a reverse proxy and more in addition to its intended functionality as a load balancer, it’s easy to see why HAProxy is used by some of the largest sites on the Internet including Reddit, Tumblr, GitHub and more on a daily basis.

This function is available in the following versions of HAProxy:

Related links

Links

What’s new in HAProxy 1.6

[ANNOUNCE] HAProxy 1.6.0 released


Yesterday, 13th of October, Willy has announced the release of HAProxy 1.6.0, after 16 months of development!
First good news is that release cycle goes a bit faster and we aim to carry on making it as fast.
A total of 1156 commits from 59 people were committed since the release of 1.5.0, 16 months ago.

Please find here the official announce: [ANNOUNCE] haproxy-1.6.0 now released!.

In his mail, Willy detailed all the features that have been added to this release. The purpose of this blog article is to highlight a few of them, providing the benefits and some configuration examples.

NOTE: Most of the features below were already backported and integrated into our HAProxy Enterprise and ALOHA products.

HAProxy Enterprise is our Open Source version of HAProxy based on HAProxy community stable branch where we backport many features from dev branch and we package it to make the most stable, reliable, advanced and secured version of HAProxy. It also comes with third party software to fill the gap between a simple HAProxy process and a load-balancer (VRRP, syslog, SNMP, Route Health Injection, etc…). Cherry on the cake, we provide “enterprise” support on top of it.

NOTE 2: the list of new features introduced here is not exhaustive. Example are proposed in a quick and dirty way to teach you how to start with the feature. Don’t run those examples in production 🙂

It’s 2015, let’s use QUOTE in configuration file


Those who uses HAProxy for a long time will be happy to know that ‘\ ‘ (backslash-space) sequence is an old painful souvenir with 1.6 🙂
We can now write:

reqirep "^Host: www.(.*)" "Host: foobar\1"

or

option httpchk GET / "HTTP/1.1\r\nHost: www.domain.com\r\nConnection: close"

Lua Scripting


Maybe the biggest change that occurred is the integration of Lua.
Quote from Lua’s website: “Lua is a powerful, fast, lightweight, embeddable scripting language.“.

Basically, everyone has now the ability to extend HAProxy by writing and running their own Lua scripts. No need to write C code, maintain patches, etc…
If some Lua snipplets are very popular, we may write the equivalent feature in C and make it available in HAProxy mainline.

One of the biggest challenge Thierry faced when integrating Lua is to give it the ability to propose non-blocking processing of Lua code and non-blocking network socket management.

HAProxy requires Lua 5.3 or above.

With Lua, we can add new functions to the following HAProxy elements:

  • Service
  • Action
  • Sample-fetch
  • Converter

Compiling HAProxy and Lua


Installing LUA 5.3

cd /usr/src
curl -R -O http://www.lua.org/ftp/lua-5.3.0.tar.gz
tar zxf lua-5.3.0.tar.gz
cd lua-5.3.0
make linux
sudo make INSTALL_TOP=/opt/lua53 install

LUA 5.3 library and include files are now installed in /opt/lua53.

Compiling HAProxy with Lua support

make TARGET=linux2628 USE_OPENSSL=1 USE_PCRE=1 USE_LUA=1 LUA_LIB=/opt/lua53/lib/ LUA_INC=/opt/lua53/include/

HAProxy/Lua simple Hello world example


A simple Hello world! in Lua could be written like this:

  • The lua code in a file called hello_world.lua:
    core.register_service("hello_world", "tcp", function(applet)
       applet:send("hello world\n")
    end)
  • The haproxy configuration
    global
       lua-load hello_world.lua
    
    listen proxy
       bind 127.0.0.1:10001
       tcp-request content use-service lua.hello_world
    

More with Lua


Please read the doc, ask your questions on HAProxy‘s ML (no registration needed): haproxy@formilux.org.
Of course, we’ll write more articles later on this blog about Lua integration.

Captures


HAProxy‘s running context is very important when writing configuration. In HAProxy, each context is isolated. IE: you can’t use a request header when processing the response.
With HAProxy 1.6, this is now possible: you can declare capture slots, store data in it and use it at any time during a session.

defaults 
 mode http

frontend f_myapp
 bind :9001
 declare capture request len 32 # id=0 to store Host header
 declare capture request len 64 # id=1 to store User-Agent header
 http-request capture req.hdr(Host) id 0
 http-request capture req.hdr(User-Agent) id 1
 default_backend b_myapp

backend b_myapp
 http-response set-header Your-Host %[capture.req.hdr(0)]
 http-response set-header Your-User-Agent %[capture.req.hdr(1)]
 server s1 10.0.0.3:4444 check

Two new headers are inserted in the response:
Your-Host: 127.0.0.1:9001
Your-User-Agent: curl/7.44.0

Multiprocess, peers and stick-tables


In 1.5, we introduced “peers” to synchronize stick-table content between HAProxy servers. This feature was not compatible with multi-process mode.
We can now, in 1.6, synchronize content of the table it is stick to one process. It allows creating configurations for massive SSL processing pointing to a single backend sticked on a single process where we can use stick tables and synchronize its content.

peers article
 peer itchy 127.0.0.1:1023

global
 pidfile /tmp/haproxy.pid
 nbproc 3

defaults
 mode http

frontend f_scalessl
 bind-process 1,2
 bind :9001 ssl crt /home/bassmann/haproxy/ssl/server.pem
 default_backend bk_lo

backend bk_lo
 bind-process 1,2
 server f_myapp unix@/tmp/f_myapp send-proxy-v2

frontend f_myapp
 bind-process 3
 bind unix@/tmp/f_myapp accept-proxy
 default_backend b_myapp

backend b_myapp
 bind-process 3
 stick-table type ip size 10k peers article
 stick on src
 server s1 10.0.0.3:4444 check

Log


log-tag


It is now possible to position a syslog tag per process, frontend or backend. Purpose is to ease the job of syslog servers when classifying logs.
If no log-tag is provided, the default value is the program name.

Example applied to the configuration snipplet right above:

frontend f_scalessl
 log-tag SSL
[...]

frontend f_myapp
 log-tag CLEAR
[...]

New log format variables


New log format variables have appeared:

  • %HM: HTTP method (ex: POST)
  • %HP: HTTP request URI without query string (path)
  • %HQ: HTTP request URI query string (ex: ?bar=baz)
  • %HU: HTTP request URI (ex: /foo?bar=baz)
  • %HV: HTTP version (ex: HTTP/1.0)

Server IP resolution using DNS at runtime


In 1.5 and before, HAProxy performed DNS resolution when parsing configuration, in a synchronous mode and using the glibc (hence /etc/resolv.conf file).
Now, HAProxy can perform DNS resolution at runtime, in an asynchronous way and update server IP on the fly. This is very convenient in environment like Docker or Amazon Web Service where server IPs can be changed at any time.
Configuration example applied to docker. A dnsmasq is used as an interface between /etc/hosts file (where docker stores server IPs) and HAProxy:

resolvers docker
 nameserver dnsmasq 127.0.0.1:53

defaults
 mode http
 log global
 option httplog

frontend f_myapp
 bind :80
 default_backend b_myapp

backend b_myapp
 server s1 nginx1:80 check resolvers docker resolve-prefer ipv4

Then, let’s restart s1 with the command “docker restart nginx1” and let’s have a look at the magic in the logs:
(...) haproxy[15]: b_myapp/nginx1 changed its IP from 172.16.0.4 to 172.16.0.6 by docker/dnsmasq.

HTTP rules


New HTTP rules have appeared:

  • http-request: capture, set-method, set-uri, set-map, set-var, track-scX, sc-in-gpc0, sc-inc-gpt0, silent-drop
  • http-response: capture, set-map, set-var, sc-inc-gpc0, sc-set-gpt0, silent-drop, redirect

Variables


We often used HTTP header fields to store temporary data in HAProxy. With 1.6, we can now define variables.
A variable is available for a scope: session, transaction (request or response), request, response.
A variable name is prefixed by its scope (sess, txn, req, res), a dot ‘.’ and a tag composed by ‘a-z’, ‘A-Z’, ‘0-9’ and ‘_’.

Let’s rewrite the capture example using variables

global
 # variables memory consumption, in bytes
 tune.vars.global-max-size 1048576
 tune.vars.reqres-max-size     512
 tune.vars.sess-max-size      2048
 tune.vars.txn-max-size        256

defaults
 mode http

frontend f_myapp
 bind :9001
 http-request set-var(txn.host) req.hdr(Host)
 http-request set-var(txn.ua) req.hdr(User-Agent)
 default_backend b_myapp

backend b_myapp
 http-response set-header Your-Host %[var(txn.host)]
 http-response set-header Your-User-Agent %[var(txn.ua)]
 server s1 10.0.0.3:4444 check

mailers


Now, HAProxy can send emails when server states change (mainly goes DOWN), so your sysadmins/devos won’t sleep anymore :). It used to be able to log only before.

mailers mymailers
 mailer smtp1 192.168.0.1:587 
 mailer smtp2 192.168.0.2:587 
 
backend mybackend 
 mode tcp 
 balance roundrobin
 email-alert mailers mymailers
 email-alert from test1@horms.org
 email-alert to test2@horms.org
 server srv1 192.168.0.30:80
 server srv2 192.168.0.31:80

Processing of HTTP request body


Until 1.5 included, HAProxy could process only HTTP request headers. It can now access request body.
Simply enable the statement below in your frontend or backend to give HAProxy this ability:

option http-buffer-request

Protection against slow-POST attacks


slow-POST attacks are like the slowlorys one, except that HTTP header are sent quicly, but request body are sent very slowly.
Once enabled, the timeout http-request parameters also apply to the POSTED data.

Fetch methods


A few new fetch methods now exists to play with the body: req.body, req.body_param, req.body_len, req.body_size, etc…

Short example to detect “SELECT *” string in a request POST body, and of course how to deny it:

defaults
 mode http

frontend f_mywaf
 bind :9001
 option http-buffer-request
 http-request deny if { req.body -m reg "SELECT \*" }
 default_backend b_myapp

backend b_myapp
 server s1 10.0.0.3:4444 check

New converters


1.5 introduced the converters but only a very few of them were available.
1.6 adds many of them. The list is too long, but let’s give the most important ones: json, in_table, field, reg_sub, table_* (to access counters from stick-tables), etc…

Device Identification


Through our company, we have some customer who want us to integrate into HAProxy the ability to detect device type and characteristics and report it to the backend server.
We got a couple of contributions from 2 companies experts in this domain: 51 degrees and deviceatlas.
You can now load those libraries in HAProxy in order to fully qualify a client capabilities and set up some headers your application server can rely on to adapt content delivered to the client or let the varnish cache server use it to cache multiple flavor of the same object based on client capabilities.

More on this blog later on how to integrate each product.

Seamless server states


Prior 1.6, when being reloaded, HAProxy considers all the servers are UP until the first check is performed.
Since 1.6, we can dump server states into a flat file right before performing the reload and let the new process know where the states are stored. That way, the old and new processes owns exactly the same server states (hence seamless).
The following information are reported:

  • server IP address when resolved by DNS
  • operational state (UP/DOWN/…)
  • administrative state (MAINT/DRAIN/…)
  • Weight (including slowstart relative weight)
  • health check status
  • rise / fall current counter
  • check state (ENABLED/PAUSED/…)
  • agent check state (ENABLED/PAUSED/…)

The state could be applied globally (all server found) or per backend.
Example:
Simple HAProxy configuration:

global
 stats socket /tmp/socket
 server-state-file /tmp/server_state

backend bk
 load-server-state-from-file global
 server s1 10.0.0.3:4444 check weight 11
 server s2 10.0.0.4:4444 check weight 12

Before reloading HAProxy, we save the server states using the following command:

socat /tmp/socket - <<< "show servers state" > /tmp/server_state

Here is the content of /tmp/server_state file:

1
# <field names skipped for the blog article>
1 bk 1 s1 10.0.0.3 2 0 11 11 4 6 3 4 6 0 0
1 bk 2 s2 10.0.0.4 2 0 12 12 4 6 3 4 6 0 0

Now, let’s proceed with reload as usual.
Of course, the best option is to export the server states using the init script.

External check


HAProxy can run a script to perform complicated health checks.
Just be aware about the security concerns when enabling this feature!!

Configuration:

global
 external-check

backend b_myapp
 external-check path "/usr/bin:/bin"
 external-check command /bin/true
 server s1 10.0.0.3:4444 check

TLS / SSL


NOTE: some of the features introduced here may need a recent openssl library.

Detection of ECDSA-able clients


This has already been documented by Nenad on this blog: Serving ECC and RSA certificates on same IP with HAProxy

SSL certificate forgery on the fly


Since 1.6, HAProxy can forge SSL certificate on the fly!
Yes, you can use HAProxy with your company’s CA to inspect content.

Support of Certificate Transparency (RFC6962) TLS extension


When loading PEM files, HAProxy also checks for the presence of file at the same path suffixed by “.sctl”. If such file is found, support for Certificate Transparency (RFC6962) TLS extension is enabled.
The file must contain a valid Signed Certificate Timestamp List, as described in RFC. File is parsed to check basic syntax, but no signatures are verified.

TLS Tickets key load through stats socket


A new stats socket command is available to update TLS ticket keys at runtime. The new key is used for encryption/decryption while the old ones are used for decryption only.

Server side SNI


Many application servers now takes benefit from Server Name Indication TLS extension.
Example:

backend b_myapp_ssl
 mode http
 server s1 10.0.0.3:4444 check ssl sni req.hdr(Host)

Peers v2


The peers protocol, used to synchronized data from stick-tables between two HAProxy servers can now synchronise more than just the sample and the server-id.
It can synchronize all the tracked counters. Note that each node push its local counter to a peer. So this must be used for safe reload and server failover only.
Don’t expect to see 10 HAProxy servers to sync and aggregate counters in real time.

That said, this protocol has been extended to support different data type, so we may see more features soon relying on it 😉

HTTP connection sharing


On the road to HTTP/2, HAProxy must be able to support connection pooling.
And on the road to the connection pools, we have the ability to share a server side connection between multiple clients.
The server side connection may be used by multiple clients, until the owner (the client side connection) of this session dies. Then new connections may be established.

The new keyword is http-reuse and have different level of sharing connections:

  • never: no connections are shared
  • safe: first request of each client is sent over its own connection, subsequent request may used an other connection. It works like a regular HTTP keepalive
  • aggressive: send request to connections that has proved reliably support connection reuse (no quick connection close after a response has been sent)
  • always: send request to established connections, whatever happens. If the server was closing the connection in the mean time, the request is lost and the client must resend it.

Get rid of 408 in logs


Simply use the new option

option http-ignore-probes

.

Packetshield: quand votre load-balancer vous protĂšge contre les DDOS!

Les attaques par DDOS


Il y a quelques temps, nous avions publié sur ce blog, un article expliquant comment utiliser un load-balancer pour se protéger contre les attaques de type DDOS applicatif: http://blog.haproxy.com/2012/02/27/use-a-load-balancer-as-a-first-row-of-defense-against-ddos/.

Malheureusement, les attaques sont aujourd’hui sur plusieurs vecteurs, notamment rĂ©seau et applicatif.
Leur but est toujours le mĂȘme: trouver un point faible dans l’architecture et le saturer:

  1. un firewall, nombre de sessions ou paquets par secondes
  2. un load-balancer, nombre sessions Ă©tablies
  3. un serveur applicatif, nombre de requĂȘtes par secondes
  4. etc…

Dans l’article prĂ©cĂ©dent, nous avions vu comment protĂ©ger les serveurs d’applications en utilisant le load-balancer, ce qui correspond Ă  la partie applicative de l’architecture.

Pour protĂ©ger la tĂȘte de l’architecture, le firewall, il faut en gĂ©nĂ©ral investir beaucoup d’argent dans des Ă©quipements coĂ»teux.
Par ailleurs, dans certains cas, lorsque l’ALOHA Ă©tait connectĂ© en direct sur Internet, il ne savait se protĂ©ger que modĂ©rĂ©ment contre les attaques “rĂ©seau”.

C’est pourquoi, HAProxy Technologies a dĂ©veloppĂ© sa propre solution de protection contre les DDOS de niveau rĂ©seau. Cet outil s’appelle Packetshield.

Les attaques de niveau rĂ©seau sont assez simples Ă  mettre en oeuvre pour l’attaquant, mais peuvent ĂȘtre dĂ©vastatrices pour la cible: lorsqu’un Ă©quipement est saturĂ©, il bloque tout le trafic.
Un équipement peut se saturer de différentes maniÚres:

  • un trop grand nombre de paquets par secondes
  • un trop grand nombre se sessions ouvertes (TCP et/ou UDP)
  • un trop gros volume de trafic entrant

(liste non exhaustive)

NOTE: pour ce dernier cas, il faut passer par un service externe de “nettoyage” du flux, qui renverra ensuite un trafic lĂ©gitime vers votre architecture.

Packetshield: Protection contre les DDOS réseau

L’ALOHA, Ă©quipĂ© de packetshield, se place en point d’entrĂ©e de l’infrastructure.
Packetshield va agir comme un Firewall stateful en analysant les flux réseaux le traversant. Seuls le trafic considéré comme légitime sera autorisé à passer.
Différents mécanismes rentrent en jeu lors de la protection:

  • analyse des paquets: les paquets non valides sont automatiquement bloquĂ©s
  • filtrage de flux: l’administrateur peut dĂ©finir ce qui est autorisĂ© ou pas
  • analyse des flux autorisĂ©s: Packetshield fait le trie pour savoir ce qui est lĂ©gitime ou pas

Contrairement aux solutions existantes qui fonctionnent dans un mode stateless et qui gĂ©nĂšrent beaucoup de faux positifs, Packetshield garantie qu’aucune session valide ne sera bloquĂ©e.

Disponibilité de Packetshield

Packetshield est disponible dans l’ALOHA en version appliance physique depuis le firmware 7.0.
La protection de l’interface gigabit (Max 1 million de paquets par secondes) est inclu gratuitement dans l’ALOHA.
Nos clients existants n’ont qu’Ă  mettre Ă  jour le firmware de leur appliance pour en bĂ©nĂ©ficier. Tout nouveau client achetant l’une de nos appliances bĂ©nĂ©ficiera lui aussi de cette protection, sans surcoĂ»t.

Packetshield est aussi disponible en version 10G (14 millions de paquets traités par secondes) sur nos toutes appliances dernier modÚle ALB5100, peut importe la licence choisie, du 8K au 64K.

Mode de déploiement


De part son fonctionnement stateful, Packetshield est utilisé de maniÚre optimal dans les déploiement suivant:

Packetshield peut fonctionner en mode stateless, sans ses fonctionnalités avancé. Dans ce mode, Packetshield est compatible avec les type de déploiement suivants:

Pour plus d’information

La documentation sur la configuration de Packetshield est disponible Ă  cette adresse: http://haproxy.com/doc/aloha/7.0/packet_flood_protection/.

Pour plus d’information, un petit mail Ă  contact (at) haproxy (point) com.

Liens

Apache cdorked backdoor detection

Apache Cdorked.A backdoor

This is a pretty recent attack, using Cpanel to change the Apache httpd binary by a compromised one which embeds a backdoor.

A few articles with more details are available here:
  * http://www.welivesecurity.com/2013/04/26/linuxcdorked-new-apache-backdoor-in-the-wild-serves-blackhole/
  * http://blog.sucuri.net/2013/04/apache-binary-backdoors-on-cpanel-based-servers.html

It seems there are a few ways to detect if your server was compromised:
  1. requests with “GET_BACK;” encoded in the query string may arrive
  2. an unexpected Cookie (SECID in that case) may be sent to the server
  3. the server may answer some unexpected Location headers

Configuration

The HAProxy configuration below provides a few hints on how you can detect if you’ve been infected by the backdoor and how you can try to protect users using your services

I consider the website hostname is “www.domain.tld” and static content is delivered by “static.domain.tld”.

The configuration below can be added in the Frontend section:

# We want to capture and log the cookies sent by the client
 capture request header Cookie Len 128
# We want to capture and log the Location header sent by the server
 capture response header Location Len 128

# block any request with GET_BACK; string encoded
 http-request deny if { url_sub 4745545f4241434b3b } 
# block any request with a weird cookie
 http-request deny if { cook_cnt(SECID) ge 1 }

# block a response with a Location header for a unknown domain
 rspdeny ^Location: http://(www|static).domain.tld.*

Note that with such backdoor, you may have to monitor your logs (detection phase) first to know if you’ve been affected. Then you can update your configuration to block the attack (protection phase) and of course, you should remove the bad apache binary.

Links

wordpress CMS brute force protection with HAProxy

Brute force attacks

Brute force is a pretty simple type of attacks: it consists of massively send requests to a URL with different parameter each time. The main purpose is to try to find the right parameter combination.
Usually, brute force is used to discover login/password credentials to enter into a web application.

Fortunately, brute force are easy to detect, and latest HAProxy version have everything they need to protect any web application web form from brute forcing.
In the current article, we’ll apply the configuration on a wordpress CMS (content Management System), which was brute forced around mid of april 2013.

Note that the ALOHA Load-Balancer firmware 5.5 and above also includes the features introduced here.

WordPress login

WordPress provide the login form through the URL: /wp-login.php. Of course, it’s a regular GET.
The user just fill up the form, then the browser sends a POST on the URL /wp-login.php with form data in the body.
So basically, a brute force attacker will forge POST request on /wp-login.php and trying many different form data combination.
Below, an example of a forged request to try the credential admin/admin:

log=admin&pwd=admin&wp-submit=Log+In&redirect_to=http%3A%2F%www.domain.tld%2Fwp-admin%2F&testcookie=1

We can clearly see the log and pwd fields.

Blocking a brute force ???? Better sandboxing it smartly!!!!


It would be easy with HAProxy to drop the TCP connection, or to answer HTTP deny (403) status codes when we see somebody is abusing.
Actually, the attacker could use these information to know the maximum request rate he can achieve without being blacklisted.

The current article proposes to send abusers into a sandbox which keeps on delivering a static version of the login form, letting the abuser trying to hack your site, but actually hacking a static page 🙂
Furthermore, HAProxy will also slowdown the abusers request rate by tarpitting the request during 1s.

Brute force protection with HAProxy / ALOHA Load-Balancer

The configration is split in 2 parts:
  1. in the frontend, we store the list of blocked users
  2. in the backend, we do the brute force detection and we notify the frontend when an abuser is detected

Configuration for Brute force detection in the backend

This configuration stores a hash of 3 elements: HTTP Host header, URL path and source IP.
We’ll enable tracking only when the requests occur on the wordpress login URL (/wp-login.php) and if the method is POST.
Based on this, we can track the number of HTTP request the source IP did over a period of 20s and we can decide if the source IP did more than 5 login tentative during this period of time, then we want to flag this user as an abuser.
Basically 5 tries are allowed per 20s, over this limit, then the 6th try will make the user blocked.

[...]
  tcp-request inspect-delay 10s
  tcp-request content accept if HTTP
  # brute force protection
  acl wp_login                path_beg -i /wp-login.php
  stick-table type binary len 20 size 500 store http_req_rate(20s) peers local
  tcp-request content track-sc2  base32+src if METH_POST wp_login
  stick store-request base32+src            if METH_POST wp_login
  acl bruteforce_detection  sc2_http_req_rate gt 5
  acl flag_bruteforce       sc1_inc_gpc0      gt 0
  http-request deny if bruteforce_detection flag_bruteforce
[...]

Configuration for blocking abusers in the frontend


The configuration below detects that a user has abused the login page and then redirect him into a sandbox where HAProxy has been configured to serve a wordpress login page.
Which means the attacker will still think he is trying to brute force wordpress, but actually, he will brute force a static page !!!!! It will be impossible for him to know he has been sandboxed…
  * Frontend configuration:

  tcp-request inspect-delay 10s
  tcp-request accept if HTTP
[...]
  acl wp_login                 path_beg     -i /wp-login.php
  acl flagged_as_abuser        sc1_get_gpc0 gt 0
  stick-table type binary len 20 size 500 store gpc0 peers local
  tcp-request content track-sc1  base32+src if METH_POST wp_login
  use_backend bk_login_abusers if flagged_as_abuser
[...]

  * sandbox backend configuration:

[...]
backend bk_login_abusers
  mode http
  log global
  option httplog
  timeout tarpit 1s
  http-request tarpit
  errorfile 500 /etc/haproxy/pages/wp_fake_login.http
  errorfile 503 /etc/haproxy/pages/wp_fake_login.http
[...]

  * Errorfile content example is provided at the bottom of this article, in the Apendice section

The protection in action


Below, an extract of HAProxy logs (anonymized) which show the blocking capacity of the configuration above:

[...]
 ft_www bk_wordpress/w1 "POST /wp-login.php HTTP/1.1"
 ft_www bk_wordpress/w1 "POST /wp-login.php HTTP/1.1"
 ft_www bk_wordpress/w1 "POST /wp-login.php HTTP/1.1"
 ft_www bk_wordpress/w1 "POST /wp-login.php HTTP/1.1"
 ft_www bk_wordpress/w1 "POST /wp-login.php HTTP/1.1"
 ft_www bk_login_abusers/<NOSRV> "POST /wp-login.php HTTP/1.1"
 ft_www bk_login_abusers/<NOSRV> "POST /wp-login.php HTTP/1.1"
 ft_www bk_login_abusers/<NOSRV> "POST /wp-login.php HTTP/1.1"
[...]

5 attempts before being redirected to the sendbox, and still attempting 😉

Let’s have a look at the stick table:

# table: ft_www, type: binary, size:500, used:1
0x24f81e4: key=57FD750958BE12B3000000000000000000000000 use=0 exp=0 gpc0=1

# table: bk_wordpress, type: binary, size:500, used:1
0x24f8740: key=57FD750958BE12B3000000000000000000000000 use=0 exp=0 server_id=1 http_req_rate(20000)=6

Even if the http_req_rate decrease, as long as gpc0 is greater than 0 in the ft_www frontend stick-table, the user will be redirected to the sandbox.

Links

Appendice


  * Errorfile content, which is the wordpress login page content:
Don’t forget to change the wwww.domain.tld by your own domain, and don’t forget to update the Content-Length header using the following script from our github: errorfile_content_length

[...]
HTTP/1.0 200 OK
Server: webserver
Date: Fri, 26 Apr 2013 08:17:37 GMT
Content-Type: text/html; charset=UTF-8
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
Pragma: no-cache
Set-Cookie: wordpress_test_cookie=WP+Cookie+check; path=/
X-Frame-Options: SAMEORIGIN
Connection: close
Content-Length: 3253

<!DOCTYPE html>
        <html xmlns="http://www.w3.org/1999/xhtml" lang="en-US">
        <head>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
        <title>Bedis &rsaquo; Log In</title>
        <link rel='stylesheet' id='wp-admin-css'  href='http://www.domain.tld/wp-admin/css/wp-admin.min.css?ver=3.5.1' type='text/css' media='all' />
<link rel='stylesheet' id='buttons-css'  href='http://www.domain.tld/wp-includes/css/buttons.min.css?ver=3.5.1' type='text/css' media='all' />
<link rel='stylesheet' id='colors-fresh-css'  href='http://www.domain.tld/wp-admin/css/colors-fresh.min.css?ver=3.5.1' type='text/css' media='all' />
<link rel="stylesheet" href="http://www.domain.tld/wp-content/themes/notes-blog-core-theme/custom/login.css" type="text/css" media="screen" /><meta name='robots' content='noindex,nofollow' />
<script type="text/javascript">
addLoadEvent = function(func){if(typeof jQuery!="undefined")jQuery(document).ready(func);else if(typeof wpOnload!='function'){wpOnload=func;}else{var oldonload=wpOnload;wpOnload=function(){oldonload();func();}}};
function s(id,pos){g(id).left=pos+'px';}
function g(id){return document.getElementById(id).style;}
function shake(id,a,d){c=a.shift();s(id,c);if(a.length>0){setTimeout(function(){shake(id,a,d);},d);}else{try{g(id).position='static';wp_attempt_focus();}catch(e){}}}
addLoadEvent(function(){ var p=new Array(15,30,15,0,-15,-30,-15,0);p=p.concat(p.concat(p));var i=document.forms[0].id;g(i).position='relative';shake(i,p,20);});
</script>
        </head>
        <body class="login login-action-login wp-core-ui">
        <div id="login">
                <h1><a href="http://www.domain.tld/" title="Bedis Sites">Bedis</a></h1>
        <div id="login_error">  <strong>ERROR</strong>: Invalid username. <a href="http://www.domain.tld/wp-login.php?action=lostpassword" title="Password Lost and Found">Lost your password</a>?<br />
</div>

<form name="loginform" id="loginform" action="http://www.domain.tld/wp-login.php" method="post">
        <p>
                <label for="user_login">Username<br />
                <input type="text" name="log" id="user_login" class="input" value="" size="20" /></label>
        </p>
        <p>
                <label for="user_pass">Password<br />
                <input type="password" name="pwd" id="user_pass" class="input" value="" size="20" /></label>
        </p>
        <p class="forgetmenot"><label for="rememberme"><input name="rememberme" type="checkbox" id="rememberme" value="forever"  /> Remember Me</label></p>
        <p class="submit">
                <input type="submit" name="wp-submit" id="wp-submit" class="button button-primary button-large" value="Log In" />
                <input type="hidden" name="redirect_to" value="http://www.domain.tld/wp-admin/" />
                <input type="hidden" name="testcookie" value="1" />
        </p>
</form>

<p id="nav">
<a href="http://www.domain.tld/wp-login.php?action=lostpassword" title="Password Lost and Found">Lost your password?</a>
</p>

<script type="text/javascript">
function wp_attempt_focus(){
setTimeout( function(){ try{
d = document.getElementById('user_login');
if( d.value != '' )
d.value = '';
d.focus();
d.select();
} catch(e){}
}, 200);
}

if(typeof wpOnload=='function')wpOnload();
</script>

        <p id="backtoblog"><a href="http://www.domain.tld/" title="Are you lost?">&larr; Back to Bedis</a></p>

        </div>


                <div class="clear"></div>
        </body>
        </html>

[...]

high performance WAF platform with Naxsi and HAProxy

Synopsis

I’ve already described WAF in a previous article, where I spoke about WAF scalability with apache and modsecurity.
One of the main issue with Apache and modsecurity is the performance. To address this issue, an alternative exists: naxsi, a Web Application Firewall module for nginx.

So using Naxsi and HAProxy as a load-balancer, we’re able to build a platform which meets the following requirements:

  • Web Application Firewall: achieved by Apache and modsecurity
  • High-availability: application server and WAF monitoring, achieved by HAProxy
  • Scalability: ability to adapt capacity to the upcoming volume of traffic, achieved by HAProxy
  • DDOS protection: blind and brutal attacks protection, slowloris protection, achieved by HAProxy
  • Content-Switching: ability to route only dynamic requests to the WAF, achieved by HAProxy
  • Reliability: ability to detect capacity overusage, this is achieved by HAProxy
  • Performance: deliver response as fast as possible, achieved by the whole platform

The picture below provides a better overview:

The LAB platform is composed by 6 boxes:

  • 2 ALOHA Load-Balancers (could be replaced by HAProxy 1.5-dev)
  • 2 WAF servers: CentOS 6.0, nginx and Naxsi
  • 2 Web servers: Debian + apache + PHP + dokuwiki

Nginx and Naxsi installation on CentOS 6

Purpose of this article is not to provide such procedue. So please read this wiki article which summarizes how to install nginx and naxsi on CentOS 6.0.

Diagram

The diagram below shows the platform with HAProxy frontends (prefixed by ft_) and backends (prefixed by bk_). Each farm is composed by 2 servers.

Configuration

Nginx and Naxsi


Configure nginx as a reverse-proxy which listen in bk_waf and forward traffic to ft_web. In the mean time, naxsi is there to analyze the requests.

server {
 proxy_set_header Proxy-Connection "";
 listen       192.168.10.15:81;
 access_log  /var/log/nginx/naxsi_access.log;
 error_log  /var/log/nginx/naxsi_error.log debug;

 location / {
  include    /etc/nginx/test.rules;
  proxy_pass http://192.168.10.2:81/;
 }

 error_page 403 /403.html;
 location = /403.html {
  root /opt/nginx/html;
  internal;
 }

 location /RequestDenied {
  return 403;
 }
}

HAProxy Load-Balancer configuration


The configuration below allows the following advanced features:

  • DDOS protection on the frontend
  • abuser or attacker detection in bk_waf and blocking on the public interface (ft_waf)
  • Bypassing WAF when overusage or unavailable
######## Default values for all entries till next defaults section
defaults
  option  http-server-close
  option  dontlognull
  option  redispatch
  option  contstats
  retries 3
  timeout connect 5s
  timeout http-keep-alive 1s
  # Slowloris protection
  timeout http-request 15s
  timeout queue 30s
  timeout tarpit 1m          # tarpit hold tim
  backlog 10000

# public frontend where users get connected to
frontend ft_waf
  bind 192.168.10.2:80 name http
  mode http
  log global
  option httplog
  timeout client 25s
  maxconn 10000

  # DDOS protection
  # Use General Purpose Couter (gpc) 0 in SC1 as a global abuse counter
  # Monitors the number of request sent by an IP over a period of 10 seconds
  stick-table type ip size 1m expire 1m store gpc0,http_req_rate(10s),http_err_rate(10s)
  tcp-request connection track-sc1 src
  tcp-request connection reject if { sc1_get_gpc0 gt 0 }
  # Abuser means more than 100reqs/10s
  acl abuse sc1_http_req_rate(ft_web) ge 100
  acl flag_abuser sc1_inc_gpc0(ft_web)
  tcp-request content reject if abuse flag_abuser

  acl static path_beg /static/ /dokuwiki/images/
  acl no_waf nbsrv(bk_waf) eq 0
  acl waf_max_capacity queue(bk_waf) ge 1
  # bypass WAF farm if no WAF available
  use_backend bk_web if no_waf
  # bypass WAF farm if it reaches its capacity
  use_backend bk_web if static waf_max_capacity
  default_backend bk_waf

# WAF farm where users' traffic is routed first
backend bk_waf
  balance roundrobin
  mode http
  log global
  option httplog
  option forwardfor header X-Client-IP
  option httpchk HEAD /waf_health_check HTTP/1.0

  # If the source IP generated 10 or more http request over the defined period,
  # flag the IP as abuser on the frontend
  acl abuse sc1_http_err_rate(ft_waf) ge 10
  acl flag_abuser sc1_inc_gpc0(ft_waf)
  tcp-request content reject if abuse flag_abuser

  # Specific WAF checking: a DENY means everything is OK
  http-check expect status 403
  timeout server 25s
  default-server inter 3s rise 2 fall 3
  server waf1 192.168.10.15:81 maxconn 100 weight 10 check
  server waf2 192.168.10.16:81 maxconn 100 weight 10 check

# Traffic secured by the WAF arrives here
frontend ft_web
  bind 192.168.10.2:81 name http
  mode http
  log global
  option httplog
  timeout client 25s
  maxconn 1000
  # route health check requests to a specific backend to avoid graph pollution in ALOHA GUI
  use_backend bk_waf_health_check if { path /waf_health_check }
  default_backend bk_web

# application server farm
backend bk_web
  balance roundrobin
  mode http
  log global
  option httplog
  option forwardfor
  cookie SERVERID insert indirect nocache
  default-server inter 3s rise 2 fall 3
  option httpchk HEAD /
  # get connected on the application server using the user ip
  # provided in the X-Client-IP header setup by ft_waf frontend
  source 0.0.0.0 usesrc hdr_ip(X-Client-IP)
  timeout server 25s
  server server1 192.168.10.11:80 maxconn 100 weight 10 cookie server1 check
  server server2 192.168.10.12:80 maxconn 100 weight 10 cookie server2 check

# backend dedicated to WAF checking (to avoid graph pollution)
backend bk_waf_health_check
  balance roundrobin
  mode http
  log global
  option httplog
  option forwardfor
  default-server inter 3s rise 2 fall 3
  timeout server 25s
  server server1 192.168.10.11:80 maxconn 100 weight 10 check
  server server2 192.168.10.12:80 maxconn 100 weight 10 check

Detecting attacks


On the load-balancer


The ft_waf frontend stick table tracks two information: http_req_rate and http_err_rate which are respectively the http request rate and the http error rate generated by a single IP address.
HAProxy would automatically block an IP which has generated more than 100 requests over a period of 10s or 10 errors (WAF detection 403 responses included) in 10s. The user is blocked for 1 minute as long as he keeps on abusing.
Of course, you can setup above values to whatever you need: it is fully flexible.

To know the status of IPs in your load-balancer, just run the command below:

echo show table ft_waf | socat /var/run/haproxy.stat - 
# table: ft_waf, type: ip, size:1048576, used:1
0xc33304: key=192.168.10.254 use=0 exp=4555 gpc0=0 http_req_rate(10000)=1 http_err_rate(10000)=1

Note: The ALOHA Load-balancer does not provide watch tool, but you can monitor the content of the table in live with the command below:

while true ; do echo show table ft_waf | socat /var/run/haproxy.stat - ; sleep 2 ; clear ; done

On the Waf


Every Naxsi error log appears in /var/log/nginx/naxsi_error.log. IE:

2012/10/16 13:40:13 [error] 10556#0: *10293 NAXSI_FMT: ip=192.168.10.254&server=192.168.10.15&uri=/testphp.vulnweb.com/artists.php&total_processed=3195&total_blocked=2&zone0=ARGS&id0=1000&var_name0=artist, client: 192.168.10.254, server: , request: "GET /testphp.vulnweb.com/artists.php?artist=0+div+1+union%23foo*%2F*bar%0D%0Aselect%23foo%0D%0A1%2C2%2Ccurrent_user HTTP/1.1", host: "192.168.10.15:81"

Naxsi log line is less obvious than modsecurity one. The rule which matched os provided by the argument idX=abcde.
No false positive during the test, I had to build a request to make Naxsi match it 🙂 .

conclusion


Today, we saw it’s easy to build a scalable and performing WAF platform in front of any web application.
The WAF is able to communicate to HAProxy which IPs to automatically blacklist (throuth error rate monitoring), which is convenient since the attacker won’t bother the WAF for a certain amount of time 😉
The platform allows to detect WAF farm availability and to bypass it in case of total failure, we even saw it is possible to bypass the WAF for static content if the farm is running out of capacity. Purpose is to deliver a good end-user experience without dropping too much the security.
Note that it is possible to route all the static content to the web servers (or a static farm) directly, whatever the status of the WAF farm.
This make me say that the platform is fully scallable and flexible.
Thanks to HAProxy, the architecture is very flexible: I could switch my apache + modexurity to nginx + naxsi with no issues at all 🙂 This could be done as well for any third party waf appliances.
Note that I did not try any naxsi advanced features like learning mode and the UI as well.

Related links

Links

Scalable WAF protection with HAProxy and Apache with modsecurity

Greeting to Thomas Heil, from our German partner Olanis, for his help in Apache and modsecurity configuration assistance.

What is a Web Application Firewall (WAF)?

Years ago, it was common to protect networks using a firewall… Well known devices which filter traffic at layer 3 and 4…
Now, the failures have moved from the network stack to the application layer, making the old firewall useless and obsolete (for protection purpose I mean). We used then to deploy IDS or IPS, which tried to match attacks at a packet level. These products are usually very hard to tune.
Then the Web Application Firewall arrived: it’s a firewall aware of the layer 7 protocol in order to be more efficient when deciding to block requests (or responses).
This is because the attacks became more complicated, like the SQL Injection, Cross Site scripting, etc…

One of the most famous opensource WAF is mod_security, which works as a module on Apache webserver and IIS for a long time and has been announced recently for nginx too.
A very good alternative is naxsi, a module for nginx, still young but very promising.

On today’s article, I’ll focus on modsecurity for Apache. In a next article, I’ll build the same platform with naxsi and nginx.

Scalable WAF platform


The main problem with WAF, is that they require a lot of resources to analyse each requests headers and body. (it can even be configured to analyze the response). If you want to be able to protect all your upcoming traffic, then you must think scalability.
In the present article I’m going to explain how to build a reliable and scalable platform where WAF capacity won’t be an issue. I could add “and where WAF maintenance could be done during business hours).
Here are the basic purpose to achieve:

  • Web Application Firewall: achieved by Apache and modsecurity
  • High-availability: application server and WAF monitoring, achieved by HAProxy
  • Scalability: ability to adapt capacity to the upcoming volume of traffic, achieved by HAProxy

It would be good if the platform would achieve the following advanced features:

  • DDOS protection: blind and brutal attacks protection, slowloris protection, achieved by HAProxy
  • Content-Switching: ability to route only dynamic requests to the WAF, achieved by HAProxy
  • Reliability: ability to detect capacity overusage, this is achieved by HAProxy
  • Performance: deliver response as fast as possible, achieved by the whole platform

Web platform with WAF Diagram

The diagram below shows the platform with HAProxy frontends (prefixed by ft_) and backends (prefixed by bk_). Each farm is composed by 2 servers.

As you can see, at first, it seems all the traffic goes to the WAFs, then comes back in HAProxy before being routed to the web servers. This would be the basic configuration, meeting the following basic requirements: Web Application Firewall, High-Availability, Scalability.

Platform installation


As load-balancer, I’m going to use our well known ALOHA 🙂
The web servers are standard debian with apache and PHP, the application used on top of it is dokuwiki. I have no procedure for this one, this is very straight forward!
The WAF run on CentOS 6.3 x86_64, using modsecurity 2.5.8. The installation procedure is outside of the scope of this article, so I documented it on my personal wiki.
All of these servers are virtualized on my laptop using KVM, so NO, I won’t run performance benchmark, it would be ridiculous!

Configuration

WAF configuration


Basic configuration here, no tuning at all. The purpose is not to explain how to configure a WAF, sorry.

Apache Configuration


Modification to the file /etc/httpd/conf/httpd.conf:

Listen 192.168.10.15:81
[...]
LoadModule security2_module modules/mod_security2.so
LoadModule unique_id_module modules/mod_unique_id.so
[...]
NameVirtualHost 192.168.10.15:81
[...]
<IfModule mod_security2.c>
        SecPcreMatchLimit 1000000
        SecPcreMatchLimitRecursion 1000000
        SecDataDir logs/
</IfModule>
<VirtualHost 192.168.10.15:81>
        ServerName *
        AddDefaultCharset UTF-8

        <IfModule mod_security2.c>
                Include modsecurity.d/modsecurity_crs_10_setup.conf
                Include modsecurity.d/aloha.conf
                Include modsecurity.d/rules/*.conf

                SecRuleEngine On
                SecRequestBodyAccess On
                SecResponseBodyAccess On
        </IfModule>

        ProxyPreserveHost On
        ProxyRequests off
        ProxyVia Off
        ProxyPass / http://192.168.10.2:81/
        ProxyPassReverse / http://192.168.10.2:81/
</VirtualHost>

Basically, we just turned Apache into a reverse-proxy, accepting traffic for any server name, applying modsecurity rules before routing traffic back to HAProxy frontend dedicated to web servers.

Client IP


HAProxy works has a reverse proxy and so will use its own IP address to get connected on the WAF server. So you have to install mod_rpaf to get the client IP in the WAF for both tracking and logging.
To install mod_rpaf, follow these instructions: apache mod_rpaf installation.
Concerning its configuration, we’ll do it as below, edit the file /etc/httpd/conf.d/mod_rpaf.conf:

LoadModule rpaf_module modules/mod_rpaf-2.0.so

<IfModule rpaf_module>
        RPAFenable On
        RPAFproxy_ips 192.168.10.1 192.168.10.3
        RPAFheader X-Client-IP
</IfModule>

modsecurity custom rules

In the Apache configuration there is a directive which tells modsecurity to load a file called aloha.conf. The purpose of this file is to tell to modsecurity to deny the health check requests from HAProxy and to prevent logging them.
HAProxy will consider the WAF as operational only if it gets a 403 response to this request. (see HAProxy configuration below).
Content of the file /etc/httpd/modsecurity.d/aloha.conf:

SecRule REQUEST_FILENAME "/waf_health_check" "nolog,deny"

Load-Balancer (HAProxy) configuration for basic usage


The configuration below is the first shoot we do when deploying such platform, it is basic, simple and straight forward:

######## Default values for all entries till next defaults section
defaults
  option  http-server-close
  option  dontlognull
  option  redispatch
  option  contstats
  retries 3
  timeout connect 5s
  timeout http-keep-alive 1s
  # Slowloris protection
  timeout http-request 15s
  timeout queue 30s
  timeout tarpit 1m          # tarpit hold tim
  backlog 10000

# public frontend where users get connected to
frontend ft_waf
  bind 192.168.10.2:80 name http
  mode http
  log global
  option httplog
  timeout client 25s
  maxconn 1000
  default_backend bk_waf

# WAF farm where users' traffic is routed first
backend bk_waf
  balance roundrobin
  mode http
  log global
  option httplog
  option forwardfor header X-Client-IP
  option httpchk HEAD /waf_health_check HTTP/1.0
  # Specific WAF checking: a DENY means everything is OK
  http-check expect status 403
  timeout server 25s
  default-server inter 3s rise 2 fall 3
  server waf1 192.168.10.15:81 maxconn 100 weight 10 check
  server waf2 192.168.10.16:81 maxconn 100 weight 10 check

# Traffic secured by the WAF arrives here
frontend ft_web
  bind 192.168.10.2:81 name http
  mode http
  log global
  option httplog
  timeout client 25s
  maxconn 1000
  # route health check requests to a specific backend to avoid graph pollution in ALOHA GUI
  use_backend bk_waf_health_check if { path /waf_health_check }
  default_backend bk_web

# application server farm
backend bk_web
  balance roundrobin
  mode http
  log global
  option httplog
  option forwardfor
  cookie SERVERID insert indirect nocache
  default-server inter 3s rise 2 fall 3
  option httpchk HEAD /
  timeout server 25s
  server server1 192.168.10.11:80 maxconn 100 weight 10 cookie server1 check
  server server2 192.168.10.12:80 maxconn 100 weight 10 cookie server2 check

# backend dedicated to WAF checking (to avoid graph pollution)
backend bk_waf_health_check
  balance roundrobin
  mode http
  log global
  option httplog
  option forwardfor
  default-server inter 3s rise 2 fall 3
  timeout server 25s
  server server1 192.168.10.11:80 maxconn 100 weight 10 check
  server server2 192.168.10.12:80 maxconn 100 weight 10 check

Advanced Load-Balancing (HAProxy) configuration


We’re going now to improve a bit the platform. The picture below shows which type of protection is achieved by the load-balancer and the WAF:

The configuration below adds a few more features:

  • DDOS protection on the frontend
  • abuser or attacker detection in bk_waf and blocking on the public interface (ft_waf)
  • Bypassing WAF when overusage or unavailable

Which will allow to meet the advanced requirements: DDOS protection, Content-Switching, Reliability, Performance.

######## Default values for all entries till next defaults section
defaults
  option  http-server-close
  option  dontlognull
  option  redispatch
  option  contstats
  retries 3
  timeout connect 5s
  timeout http-keep-alive 1s
  # Slowloris protection
  timeout http-request 15s
  timeout queue 30s
  timeout tarpit 1m          # tarpit hold tim
  backlog 10000

# public frontend where users get connected to
frontend ft_waf
  bind 192.168.10.2:80 name http
  mode http
  log global
  option httplog
  timeout client 25s
  maxconn 10000

  # DDOS protection
  # Use General Purpose Couter (gpc) 0 in SC1 as a global abuse counter
  # Monitors the number of request sent by an IP over a period of 10 seconds
  stick-table type ip size 1m expire 1m store gpc0,http_req_rate(10s),http_err_rate(10s)
  tcp-request connection track-sc1 src
  tcp-request connection reject if { sc1_get_gpc0 gt 0 }
  # Abuser means more than 100reqs/10s
  acl abuse sc1_http_req_rate(ft_web) ge 100
  acl flag_abuser sc1_inc_gpc0(ft_web)
  tcp-request content reject if abuse flag_abuser

  acl static path_beg /static/ /dokuwiki/images/
  acl no_waf nbsrv(bk_waf) eq 0
  acl waf_max_capacity queue(bk_waf) ge 1
  # bypass WAF farm if no WAF available
  use_backend bk_web if no_waf
  # bypass WAF farm if it reaches its capacity
  use_backend bk_web if static waf_max_capacity
  default_backend bk_waf

# WAF farm where users' traffic is routed first
backend bk_waf
  balance roundrobin
  mode http
  log global
  option httplog
  option forwardfor header X-Client-IP
  option httpchk HEAD /waf_health_check HTTP/1.0

  # If the source IP generated 10 or more http request over the defined period,
  # flag the IP as abuser on the frontend
  acl abuse sc1_http_err_rate(ft_waf) ge 10
  acl flag_abuser sc1_inc_gpc0(ft_waf)
  tcp-request content reject if abuse flag_abuser

  # Specific WAF checking: a DENY means everything is OK
  http-check expect status 403
  timeout server 25s
  default-server inter 3s rise 2 fall 3
  server waf1 192.168.10.15:81 maxconn 100 weight 10 check
  server waf2 192.168.10.16:81 maxconn 100 weight 10 check

# Traffic secured by the WAF arrives here
frontend ft_web
  bind 192.168.10.2:81 name http
  mode http
  log global
  option httplog
  timeout client 25s
  maxconn 1000
  # route health check requests to a specific backend to avoid graph pollution in ALOHA GUI
  use_backend bk_waf_health_check if { path /waf_health_check }
  default_backend bk_web

# application server farm
backend bk_web
  balance roundrobin
  mode http
  log global
  option httplog
  option forwardfor
  cookie SERVERID insert indirect nocache
  default-server inter 3s rise 2 fall 3
  option httpchk HEAD /
  # get connected on the application server using the user ip
  # provided in the X-Client-IP header setup by ft_waf frontend
  source 0.0.0.0 usesrc hdr_ip(X-Client-IP)
  timeout server 25s
  server server1 192.168.10.11:80 maxconn 100 weight 10 cookie server1 check
  server server2 192.168.10.12:80 maxconn 100 weight 10 cookie server2 check

# backend dedicated to WAF checking (to avoid graph pollution)
backend bk_waf_health_check
  balance roundrobin
  mode http
  log global
  option httplog
  option forwardfor
  default-server inter 3s rise 2 fall 3
  timeout server 25s
  server server1 192.168.10.11:80 maxconn 100 weight 10 check
  server server2 192.168.10.12:80 maxconn 100 weight 10 check

Detecting attacks


On the load-balancer


The ft_waf frontend stick table tracks two information: http_req_rate and http_err_rate which are respectively the http request rate and the http error rate generated by a single IP address.
HAProxy would automatically block an IP which has generated more than 100 requests over a period of 10s or 10 errors (WAF detection 403 responses included) in 10s. The user is blocked for 1 minute as long as he keeps on abusing.
Of course, you can setup above values to whatever you need: it is fully flexible.

To know the status of IPs in your load-balancer, just run the command below:

echo show table ft_waf | socat /var/run/haproxy.stat - 
# table: ft_waf, type: ip, size:1048576, used:1
0xc33304: key=192.168.10.254 use=0 exp=4555 gpc0=0 http_req_rate(10000)=1 http_err_rate(10000)=1

Note: The ALOHA Load-balancer does not provide watch, but you can monitor the content of the table in live with the command below:

while true ; do echo show table ft_waf | socat /var/run/haproxy.stat - ; sleep 2 ; clear ; done

On the Waf


I have not setup anything particular on WAF logging, so every errors appears in /var/log/httpd/error_log. IE:

[Fri Oct 12 10:48:21 2012] [error] [client 192.168.10.254] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?:(?:[\\;\\|\\`]\\W*?\\bcc|\\b(wget|curl))\\b|\\/cc(?:[\\'\"\\|\\;\\`\\-\\s]|$))" at REQUEST_FILENAME. [file "/etc/httpd/modsecurity.d/rules/modsecurity_crs_40_generic_attacks.conf"] [line "25"] [id "950907"] [rev "2.2.5"] [msg "System Command Injection"] [data "/cc-"] [severity "CRITICAL"] [tag "WEB_ATTACK/COMMAND_INJECTION"] [tag "WASCTC/WASC-31"] [tag "OWASP_TOP_10/A1"] [tag "PCI/6.5.2"] [hostname "mywiki"] [uri "/dokuwiki/lib/images/license/button/cc-by-sa.png"] [unique_id "UHfZVcCoCg8AAApVAzsAAAAA"]

Seems to be a false positive 🙂

Conclusion


Today, we saw it’s easy to build a scalable and well performing WAF platform in front of our web application.
The WAF is able to communicate to HAProxy which IPs to automatically blacklist (throuth error rate monitoring), which is convenient since the attacker won’t bother the WAF for a certain amount of time 😉
The platform allows to detect WAF farm availability and to bypass it in case of total failure, we even saw it is possible to bypass the WAF for static content if the farm is running out of capacity. Purpose is to deliver a good end-user experience without dropping too much the security.
Note that it is possible to route all the static content to the web servers (or a static farm) directly, whatever the status of the WAF farm.
This make me say that the platform is fully scallable and flexible.
Also, bear in mind to monitor your WAF logs, as shown in the example above, there was a false positive preventing an image to be loaded from dokuwiki.

Related links

Links