Tag Archives: mod_security

HAProxy and HTTP Strict Transport Security (HSTS) header in HTTP redirects

SSL/TLS and HSTS

SSL everywhere is on its way.
Unfortunately, many applications were written for HTTP only and switching to HTTPs is not an easy and straight forward path. Read more here about impact of TLS offloading (when a third party tool perform TLS in front of your web application servers).

A mechanism called HTTP Strict Transport Security (HSTS) has been introduced through the RFC 6797.

HSTS main purpose is to let the application server to instruct the client it’s supposed to get connected only a ciphered and secured HTTPs connection when browsing the application.
It means that of course, that both the client and the server must be compatible…
That way, the application cookie is protected on its way from the client’s browser to the remote TLS endpoint (either the load-balancer or the application server). No cookie hijacking is possible on the wire.

HAProxy configuration for Strict-Transport-Security HTTP header

HSTS header insertion in server responses

To insert the header in every server response, you can use the following HAProxy directive, in HAProxy 1.5:

 # 16000000 seconds: a bit more than 6 months
 http-response set-header Strict-Transport-Security max-age=16000000;\ includeSubDomains;\ preload;

With the upcoming HAProxy 1.6, and thanks to William’s work, we can now get rid of these ugly backslashes:

 # 16000000 seconds: a bit more than 6 months
 http-response set-header Strict-Transport-Security "max-age=16000000; includeSubDomains; preload;"

Inserting HSTS header in HTTP redirects


When HAProxy has to perform HTTP redirects, it does in at the moment of the client request, through the http-request rules.
Since we want to insert a header in the response, we can use the http-response rules. Unfortunately, these rules are enabled when HAProxy get traffic from a backend server.
Here is the trick: we do perform the http-request redirect rule in a dedicated frontend where we route traffic to. That way, our application backend or frontend can perform HSTS insertion.

A simple configuration sniplet is usually easier to explain:

frontend fe_myapp
 bind :443 ssl crt /path/to/my/cert.pem
 bind :80
 use_backend be_dummy if !{ ssl_fc }
 default_backend be_myapp

backend be_myapp
 http-response set-header Strict-Transport-Security max-age=16000000;\ includeSubDomains;\ preload;
 server s1 10.0.0.1:80

be_dummy
 server haproxy_fe_dummy_ssl_redirect 127.0.0.1:8000

frontend fe_dummy
 bind 127.0.0.1:8000
 http-request redirect scheme https

Links

wordpress CMS brute force protection with HAProxy

Brute force attacks

Brute force is a pretty simple type of attacks: it consists of massively send requests to a URL with different parameter each time. The main purpose is to try to find the right parameter combination.
Usually, brute force is used to discover login/password credentials to enter into a web application.

Fortunately, brute force are easy to detect, and latest HAProxy version have everything they need to protect any web application web form from brute forcing.
In the current article, we’ll apply the configuration on a wordpress CMS (content Management System), which was brute forced around mid of april 2013.

Note that the ALOHA Load-Balancer firmware 5.5 and above also includes the features introduced here.

WordPress login

WordPress provide the login form through the URL: /wp-login.php. Of course, it’s a regular GET.
The user just fill up the form, then the browser sends a POST on the URL /wp-login.php with form data in the body.
So basically, a brute force attacker will forge POST request on /wp-login.php and trying many different form data combination.
Below, an example of a forged request to try the credential admin/admin:

log=admin&pwd=admin&wp-submit=Log+In&redirect_to=http%3A%2F%www.domain.tld%2Fwp-admin%2F&testcookie=1

We can clearly see the log and pwd fields.

Blocking a brute force ???? Better sandboxing it smartly!!!!


It would be easy with HAProxy to drop the TCP connection, or to answer HTTP deny (403) status codes when we see somebody is abusing.
Actually, the attacker could use these information to know the maximum request rate he can achieve without being blacklisted.

The current article proposes to send abusers into a sandbox which keeps on delivering a static version of the login form, letting the abuser trying to hack your site, but actually hacking a static page 🙂
Furthermore, HAProxy will also slowdown the abusers request rate by tarpitting the request during 1s.

Brute force protection with HAProxy / ALOHA Load-Balancer

The configration is split in 2 parts:
  1. in the frontend, we store the list of blocked users
  2. in the backend, we do the brute force detection and we notify the frontend when an abuser is detected

Configuration for Brute force detection in the backend

This configuration stores a hash of 3 elements: HTTP Host header, URL path and source IP.
We’ll enable tracking only when the requests occur on the wordpress login URL (/wp-login.php) and if the method is POST.
Based on this, we can track the number of HTTP request the source IP did over a period of 20s and we can decide if the source IP did more than 5 login tentative during this period of time, then we want to flag this user as an abuser.
Basically 5 tries are allowed per 20s, over this limit, then the 6th try will make the user blocked.

[...]
  tcp-request inspect-delay 10s
  tcp-request content accept if HTTP
  # brute force protection
  acl wp_login                path_beg -i /wp-login.php
  stick-table type binary len 20 size 500 store http_req_rate(20s) peers local
  tcp-request content track-sc2  base32+src if METH_POST wp_login
  stick store-request base32+src            if METH_POST wp_login
  acl bruteforce_detection  sc2_http_req_rate gt 5
  acl flag_bruteforce       sc1_inc_gpc0      gt 0
  http-request deny if bruteforce_detection flag_bruteforce
[...]

Configuration for blocking abusers in the frontend


The configuration below detects that a user has abused the login page and then redirect him into a sandbox where HAProxy has been configured to serve a wordpress login page.
Which means the attacker will still think he is trying to brute force wordpress, but actually, he will brute force a static page !!!!! It will be impossible for him to know he has been sandboxed…
  * Frontend configuration:

  tcp-request inspect-delay 10s
  tcp-request accept if HTTP
[...]
  acl wp_login                 path_beg     -i /wp-login.php
  acl flagged_as_abuser        sc1_get_gpc0 gt 0
  stick-table type binary len 20 size 500 store gpc0 peers local
  tcp-request content track-sc1  base32+src if METH_POST wp_login
  use_backend bk_login_abusers if flagged_as_abuser
[...]

  * sandbox backend configuration:

[...]
backend bk_login_abusers
  mode http
  log global
  option httplog
  timeout tarpit 1s
  http-request tarpit
  errorfile 500 /etc/haproxy/pages/wp_fake_login.http
  errorfile 503 /etc/haproxy/pages/wp_fake_login.http
[...]

  * Errorfile content example is provided at the bottom of this article, in the Apendice section

The protection in action


Below, an extract of HAProxy logs (anonymized) which show the blocking capacity of the configuration above:

[...]
 ft_www bk_wordpress/w1 "POST /wp-login.php HTTP/1.1"
 ft_www bk_wordpress/w1 "POST /wp-login.php HTTP/1.1"
 ft_www bk_wordpress/w1 "POST /wp-login.php HTTP/1.1"
 ft_www bk_wordpress/w1 "POST /wp-login.php HTTP/1.1"
 ft_www bk_wordpress/w1 "POST /wp-login.php HTTP/1.1"
 ft_www bk_login_abusers/<NOSRV> "POST /wp-login.php HTTP/1.1"
 ft_www bk_login_abusers/<NOSRV> "POST /wp-login.php HTTP/1.1"
 ft_www bk_login_abusers/<NOSRV> "POST /wp-login.php HTTP/1.1"
[...]

5 attempts before being redirected to the sendbox, and still attempting 😉

Let’s have a look at the stick table:

# table: ft_www, type: binary, size:500, used:1
0x24f81e4: key=57FD750958BE12B3000000000000000000000000 use=0 exp=0 gpc0=1

# table: bk_wordpress, type: binary, size:500, used:1
0x24f8740: key=57FD750958BE12B3000000000000000000000000 use=0 exp=0 server_id=1 http_req_rate(20000)=6

Even if the http_req_rate decrease, as long as gpc0 is greater than 0 in the ft_www frontend stick-table, the user will be redirected to the sandbox.

Links

Appendice


  * Errorfile content, which is the wordpress login page content:
Don’t forget to change the wwww.domain.tld by your own domain, and don’t forget to update the Content-Length header using the following script from our github: errorfile_content_length

[...]
HTTP/1.0 200 OK
Server: webserver
Date: Fri, 26 Apr 2013 08:17:37 GMT
Content-Type: text/html; charset=UTF-8
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
Pragma: no-cache
Set-Cookie: wordpress_test_cookie=WP+Cookie+check; path=/
X-Frame-Options: SAMEORIGIN
Connection: close
Content-Length: 3253

<!DOCTYPE html>
        <html xmlns="http://www.w3.org/1999/xhtml" lang="en-US">
        <head>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
        <title>Bedis &rsaquo; Log In</title>
        <link rel='stylesheet' id='wp-admin-css'  href='http://www.domain.tld/wp-admin/css/wp-admin.min.css?ver=3.5.1' type='text/css' media='all' />
<link rel='stylesheet' id='buttons-css'  href='http://www.domain.tld/wp-includes/css/buttons.min.css?ver=3.5.1' type='text/css' media='all' />
<link rel='stylesheet' id='colors-fresh-css'  href='http://www.domain.tld/wp-admin/css/colors-fresh.min.css?ver=3.5.1' type='text/css' media='all' />
<link rel="stylesheet" href="http://www.domain.tld/wp-content/themes/notes-blog-core-theme/custom/login.css" type="text/css" media="screen" /><meta name='robots' content='noindex,nofollow' />
<script type="text/javascript">
addLoadEvent = function(func){if(typeof jQuery!="undefined")jQuery(document).ready(func);else if(typeof wpOnload!='function'){wpOnload=func;}else{var oldonload=wpOnload;wpOnload=function(){oldonload();func();}}};
function s(id,pos){g(id).left=pos+'px';}
function g(id){return document.getElementById(id).style;}
function shake(id,a,d){c=a.shift();s(id,c);if(a.length>0){setTimeout(function(){shake(id,a,d);},d);}else{try{g(id).position='static';wp_attempt_focus();}catch(e){}}}
addLoadEvent(function(){ var p=new Array(15,30,15,0,-15,-30,-15,0);p=p.concat(p.concat(p));var i=document.forms[0].id;g(i).position='relative';shake(i,p,20);});
</script>
        </head>
        <body class="login login-action-login wp-core-ui">
        <div id="login">
                <h1><a href="http://www.domain.tld/" title="Bedis Sites">Bedis</a></h1>
        <div id="login_error">  <strong>ERROR</strong>: Invalid username. <a href="http://www.domain.tld/wp-login.php?action=lostpassword" title="Password Lost and Found">Lost your password</a>?<br />
</div>

<form name="loginform" id="loginform" action="http://www.domain.tld/wp-login.php" method="post">
        <p>
                <label for="user_login">Username<br />
                <input type="text" name="log" id="user_login" class="input" value="" size="20" /></label>
        </p>
        <p>
                <label for="user_pass">Password<br />
                <input type="password" name="pwd" id="user_pass" class="input" value="" size="20" /></label>
        </p>
        <p class="forgetmenot"><label for="rememberme"><input name="rememberme" type="checkbox" id="rememberme" value="forever"  /> Remember Me</label></p>
        <p class="submit">
                <input type="submit" name="wp-submit" id="wp-submit" class="button button-primary button-large" value="Log In" />
                <input type="hidden" name="redirect_to" value="http://www.domain.tld/wp-admin/" />
                <input type="hidden" name="testcookie" value="1" />
        </p>
</form>

<p id="nav">
<a href="http://www.domain.tld/wp-login.php?action=lostpassword" title="Password Lost and Found">Lost your password?</a>
</p>

<script type="text/javascript">
function wp_attempt_focus(){
setTimeout( function(){ try{
d = document.getElementById('user_login');
if( d.value != '' )
d.value = '';
d.focus();
d.select();
} catch(e){}
}, 200);
}

if(typeof wpOnload=='function')wpOnload();
</script>

        <p id="backtoblog"><a href="http://www.domain.tld/" title="Are you lost?">&larr; Back to Bedis</a></p>

        </div>


                <div class="clear"></div>
        </body>
        </html>

[...]

Scalable WAF protection with HAProxy and Apache with modsecurity

Greeting to Thomas Heil, from our German partner Olanis, for his help in Apache and modsecurity configuration assistance.

What is a Web Application Firewall (WAF)?

Years ago, it was common to protect networks using a firewall… Well known devices which filter traffic at layer 3 and 4…
Now, the failures have moved from the network stack to the application layer, making the old firewall useless and obsolete (for protection purpose I mean). We used then to deploy IDS or IPS, which tried to match attacks at a packet level. These products are usually very hard to tune.
Then the Web Application Firewall arrived: it’s a firewall aware of the layer 7 protocol in order to be more efficient when deciding to block requests (or responses).
This is because the attacks became more complicated, like the SQL Injection, Cross Site scripting, etc…

One of the most famous opensource WAF is mod_security, which works as a module on Apache webserver and IIS for a long time and has been announced recently for nginx too.
A very good alternative is naxsi, a module for nginx, still young but very promising.

On today’s article, I’ll focus on modsecurity for Apache. In a next article, I’ll build the same platform with naxsi and nginx.

Scalable WAF platform


The main problem with WAF, is that they require a lot of resources to analyse each requests headers and body. (it can even be configured to analyze the response). If you want to be able to protect all your upcoming traffic, then you must think scalability.
In the present article I’m going to explain how to build a reliable and scalable platform where WAF capacity won’t be an issue. I could add “and where WAF maintenance could be done during business hours).
Here are the basic purpose to achieve:

  • Web Application Firewall: achieved by Apache and modsecurity
  • High-availability: application server and WAF monitoring, achieved by HAProxy
  • Scalability: ability to adapt capacity to the upcoming volume of traffic, achieved by HAProxy

It would be good if the platform would achieve the following advanced features:

  • DDOS protection: blind and brutal attacks protection, slowloris protection, achieved by HAProxy
  • Content-Switching: ability to route only dynamic requests to the WAF, achieved by HAProxy
  • Reliability: ability to detect capacity overusage, this is achieved by HAProxy
  • Performance: deliver response as fast as possible, achieved by the whole platform

Web platform with WAF Diagram

The diagram below shows the platform with HAProxy frontends (prefixed by ft_) and backends (prefixed by bk_). Each farm is composed by 2 servers.

As you can see, at first, it seems all the traffic goes to the WAFs, then comes back in HAProxy before being routed to the web servers. This would be the basic configuration, meeting the following basic requirements: Web Application Firewall, High-Availability, Scalability.

Platform installation


As load-balancer, I’m going to use our well known ALOHA 🙂
The web servers are standard debian with apache and PHP, the application used on top of it is dokuwiki. I have no procedure for this one, this is very straight forward!
The WAF run on CentOS 6.3 x86_64, using modsecurity 2.5.8. The installation procedure is outside of the scope of this article, so I documented it on my personal wiki.
All of these servers are virtualized on my laptop using KVM, so NO, I won’t run performance benchmark, it would be ridiculous!

Configuration

WAF configuration


Basic configuration here, no tuning at all. The purpose is not to explain how to configure a WAF, sorry.

Apache Configuration


Modification to the file /etc/httpd/conf/httpd.conf:

Listen 192.168.10.15:81
[...]
LoadModule security2_module modules/mod_security2.so
LoadModule unique_id_module modules/mod_unique_id.so
[...]
NameVirtualHost 192.168.10.15:81
[...]
<IfModule mod_security2.c>
        SecPcreMatchLimit 1000000
        SecPcreMatchLimitRecursion 1000000
        SecDataDir logs/
</IfModule>
<VirtualHost 192.168.10.15:81>
        ServerName *
        AddDefaultCharset UTF-8

        <IfModule mod_security2.c>
                Include modsecurity.d/modsecurity_crs_10_setup.conf
                Include modsecurity.d/aloha.conf
                Include modsecurity.d/rules/*.conf

                SecRuleEngine On
                SecRequestBodyAccess On
                SecResponseBodyAccess On
        </IfModule>

        ProxyPreserveHost On
        ProxyRequests off
        ProxyVia Off
        ProxyPass / http://192.168.10.2:81/
        ProxyPassReverse / http://192.168.10.2:81/
</VirtualHost>

Basically, we just turned Apache into a reverse-proxy, accepting traffic for any server name, applying modsecurity rules before routing traffic back to HAProxy frontend dedicated to web servers.

Client IP


HAProxy works has a reverse proxy and so will use its own IP address to get connected on the WAF server. So you have to install mod_rpaf to get the client IP in the WAF for both tracking and logging.
To install mod_rpaf, follow these instructions: apache mod_rpaf installation.
Concerning its configuration, we’ll do it as below, edit the file /etc/httpd/conf.d/mod_rpaf.conf:

LoadModule rpaf_module modules/mod_rpaf-2.0.so

<IfModule rpaf_module>
        RPAFenable On
        RPAFproxy_ips 192.168.10.1 192.168.10.3
        RPAFheader X-Client-IP
</IfModule>

modsecurity custom rules

In the Apache configuration there is a directive which tells modsecurity to load a file called aloha.conf. The purpose of this file is to tell to modsecurity to deny the health check requests from HAProxy and to prevent logging them.
HAProxy will consider the WAF as operational only if it gets a 403 response to this request. (see HAProxy configuration below).
Content of the file /etc/httpd/modsecurity.d/aloha.conf:

SecRule REQUEST_FILENAME "/waf_health_check" "nolog,deny"

Load-Balancer (HAProxy) configuration for basic usage


The configuration below is the first shoot we do when deploying such platform, it is basic, simple and straight forward:

######## Default values for all entries till next defaults section
defaults
  option  http-server-close
  option  dontlognull
  option  redispatch
  option  contstats
  retries 3
  timeout connect 5s
  timeout http-keep-alive 1s
  # Slowloris protection
  timeout http-request 15s
  timeout queue 30s
  timeout tarpit 1m          # tarpit hold tim
  backlog 10000

# public frontend where users get connected to
frontend ft_waf
  bind 192.168.10.2:80 name http
  mode http
  log global
  option httplog
  timeout client 25s
  maxconn 1000
  default_backend bk_waf

# WAF farm where users' traffic is routed first
backend bk_waf
  balance roundrobin
  mode http
  log global
  option httplog
  option forwardfor header X-Client-IP
  option httpchk HEAD /waf_health_check HTTP/1.0
  # Specific WAF checking: a DENY means everything is OK
  http-check expect status 403
  timeout server 25s
  default-server inter 3s rise 2 fall 3
  server waf1 192.168.10.15:81 maxconn 100 weight 10 check
  server waf2 192.168.10.16:81 maxconn 100 weight 10 check

# Traffic secured by the WAF arrives here
frontend ft_web
  bind 192.168.10.2:81 name http
  mode http
  log global
  option httplog
  timeout client 25s
  maxconn 1000
  # route health check requests to a specific backend to avoid graph pollution in ALOHA GUI
  use_backend bk_waf_health_check if { path /waf_health_check }
  default_backend bk_web

# application server farm
backend bk_web
  balance roundrobin
  mode http
  log global
  option httplog
  option forwardfor
  cookie SERVERID insert indirect nocache
  default-server inter 3s rise 2 fall 3
  option httpchk HEAD /
  timeout server 25s
  server server1 192.168.10.11:80 maxconn 100 weight 10 cookie server1 check
  server server2 192.168.10.12:80 maxconn 100 weight 10 cookie server2 check

# backend dedicated to WAF checking (to avoid graph pollution)
backend bk_waf_health_check
  balance roundrobin
  mode http
  log global
  option httplog
  option forwardfor
  default-server inter 3s rise 2 fall 3
  timeout server 25s
  server server1 192.168.10.11:80 maxconn 100 weight 10 check
  server server2 192.168.10.12:80 maxconn 100 weight 10 check

Advanced Load-Balancing (HAProxy) configuration


We’re going now to improve a bit the platform. The picture below shows which type of protection is achieved by the load-balancer and the WAF:

The configuration below adds a few more features:

  • DDOS protection on the frontend
  • abuser or attacker detection in bk_waf and blocking on the public interface (ft_waf)
  • Bypassing WAF when overusage or unavailable

Which will allow to meet the advanced requirements: DDOS protection, Content-Switching, Reliability, Performance.

######## Default values for all entries till next defaults section
defaults
  option  http-server-close
  option  dontlognull
  option  redispatch
  option  contstats
  retries 3
  timeout connect 5s
  timeout http-keep-alive 1s
  # Slowloris protection
  timeout http-request 15s
  timeout queue 30s
  timeout tarpit 1m          # tarpit hold tim
  backlog 10000

# public frontend where users get connected to
frontend ft_waf
  bind 192.168.10.2:80 name http
  mode http
  log global
  option httplog
  timeout client 25s
  maxconn 10000

  # DDOS protection
  # Use General Purpose Couter (gpc) 0 in SC1 as a global abuse counter
  # Monitors the number of request sent by an IP over a period of 10 seconds
  stick-table type ip size 1m expire 1m store gpc0,http_req_rate(10s),http_err_rate(10s)
  tcp-request connection track-sc1 src
  tcp-request connection reject if { sc1_get_gpc0 gt 0 }
  # Abuser means more than 100reqs/10s
  acl abuse sc1_http_req_rate(ft_web) ge 100
  acl flag_abuser sc1_inc_gpc0(ft_web)
  tcp-request content reject if abuse flag_abuser

  acl static path_beg /static/ /dokuwiki/images/
  acl no_waf nbsrv(bk_waf) eq 0
  acl waf_max_capacity queue(bk_waf) ge 1
  # bypass WAF farm if no WAF available
  use_backend bk_web if no_waf
  # bypass WAF farm if it reaches its capacity
  use_backend bk_web if static waf_max_capacity
  default_backend bk_waf

# WAF farm where users' traffic is routed first
backend bk_waf
  balance roundrobin
  mode http
  log global
  option httplog
  option forwardfor header X-Client-IP
  option httpchk HEAD /waf_health_check HTTP/1.0

  # If the source IP generated 10 or more http request over the defined period,
  # flag the IP as abuser on the frontend
  acl abuse sc1_http_err_rate(ft_waf) ge 10
  acl flag_abuser sc1_inc_gpc0(ft_waf)
  tcp-request content reject if abuse flag_abuser

  # Specific WAF checking: a DENY means everything is OK
  http-check expect status 403
  timeout server 25s
  default-server inter 3s rise 2 fall 3
  server waf1 192.168.10.15:81 maxconn 100 weight 10 check
  server waf2 192.168.10.16:81 maxconn 100 weight 10 check

# Traffic secured by the WAF arrives here
frontend ft_web
  bind 192.168.10.2:81 name http
  mode http
  log global
  option httplog
  timeout client 25s
  maxconn 1000
  # route health check requests to a specific backend to avoid graph pollution in ALOHA GUI
  use_backend bk_waf_health_check if { path /waf_health_check }
  default_backend bk_web

# application server farm
backend bk_web
  balance roundrobin
  mode http
  log global
  option httplog
  option forwardfor
  cookie SERVERID insert indirect nocache
  default-server inter 3s rise 2 fall 3
  option httpchk HEAD /
  # get connected on the application server using the user ip
  # provided in the X-Client-IP header setup by ft_waf frontend
  source 0.0.0.0 usesrc hdr_ip(X-Client-IP)
  timeout server 25s
  server server1 192.168.10.11:80 maxconn 100 weight 10 cookie server1 check
  server server2 192.168.10.12:80 maxconn 100 weight 10 cookie server2 check

# backend dedicated to WAF checking (to avoid graph pollution)
backend bk_waf_health_check
  balance roundrobin
  mode http
  log global
  option httplog
  option forwardfor
  default-server inter 3s rise 2 fall 3
  timeout server 25s
  server server1 192.168.10.11:80 maxconn 100 weight 10 check
  server server2 192.168.10.12:80 maxconn 100 weight 10 check

Detecting attacks


On the load-balancer


The ft_waf frontend stick table tracks two information: http_req_rate and http_err_rate which are respectively the http request rate and the http error rate generated by a single IP address.
HAProxy would automatically block an IP which has generated more than 100 requests over a period of 10s or 10 errors (WAF detection 403 responses included) in 10s. The user is blocked for 1 minute as long as he keeps on abusing.
Of course, you can setup above values to whatever you need: it is fully flexible.

To know the status of IPs in your load-balancer, just run the command below:

echo show table ft_waf | socat /var/run/haproxy.stat - 
# table: ft_waf, type: ip, size:1048576, used:1
0xc33304: key=192.168.10.254 use=0 exp=4555 gpc0=0 http_req_rate(10000)=1 http_err_rate(10000)=1

Note: The ALOHA Load-balancer does not provide watch, but you can monitor the content of the table in live with the command below:

while true ; do echo show table ft_waf | socat /var/run/haproxy.stat - ; sleep 2 ; clear ; done

On the Waf


I have not setup anything particular on WAF logging, so every errors appears in /var/log/httpd/error_log. IE:

[Fri Oct 12 10:48:21 2012] [error] [client 192.168.10.254] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?:(?:[\\;\\|\\`]\\W*?\\bcc|\\b(wget|curl))\\b|\\/cc(?:[\\'\"\\|\\;\\`\\-\\s]|$))" at REQUEST_FILENAME. [file "/etc/httpd/modsecurity.d/rules/modsecurity_crs_40_generic_attacks.conf"] [line "25"] [id "950907"] [rev "2.2.5"] [msg "System Command Injection"] [data "/cc-"] [severity "CRITICAL"] [tag "WEB_ATTACK/COMMAND_INJECTION"] [tag "WASCTC/WASC-31"] [tag "OWASP_TOP_10/A1"] [tag "PCI/6.5.2"] [hostname "mywiki"] [uri "/dokuwiki/lib/images/license/button/cc-by-sa.png"] [unique_id "UHfZVcCoCg8AAApVAzsAAAAA"]

Seems to be a false positive 🙂

Conclusion


Today, we saw it’s easy to build a scalable and well performing WAF platform in front of our web application.
The WAF is able to communicate to HAProxy which IPs to automatically blacklist (throuth error rate monitoring), which is convenient since the attacker won’t bother the WAF for a certain amount of time 😉
The platform allows to detect WAF farm availability and to bypass it in case of total failure, we even saw it is possible to bypass the WAF for static content if the farm is running out of capacity. Purpose is to deliver a good end-user experience without dropping too much the security.
Note that it is possible to route all the static content to the web servers (or a static farm) directly, whatever the status of the WAF farm.
This make me say that the platform is fully scallable and flexible.
Also, bear in mind to monitor your WAF logs, as shown in the example above, there was a false positive preventing an image to be loaded from dokuwiki.

Related links

Links