Tag Archives: dos

What is a slow POST Attack and how to turn HAProxy into your first line of Defense?

One of the biggest security challenges that companies face in today’s modern climate is the POST attack. Unlike a more traditional “Denial-of-Service” attack, POST attacks target a servers logical resources – making them particularly powerful when executed.

What is a slow POST Attack?


In a POST attack, an attacker begins by sending a legitimate HTTP POST header to a Web server, exactly as they would under normal circumstances. The header specifies the exact size of the message body that will then follow. However, that message body is then sent at an alarmingly low rate – sometimes as slow as 1 byte per approximately two minutes. Because the entire message is technically correct and complete, the targeted server attempts to obey all specified rules – which as you would expect, can take quite awhile. The issue is that if an attacker were to establish hundreds or even thousands of these POST attacks simultaneously, it will quickly use up all server resources and make legitimate connections impossible.

How HAProxy can protect against slow POST attack?


Because POST attacks can be incredibly powerful, it’s always important to have a tool in place to identify these types of issues when they’re still in their nascent stages to prevent them from becoming much larger, more serious issues down the road. Because HAProxy was designed as an application delivery controller to manage Web application high availability and performance, it is already in an ideal position to stop these types of POST attacks in their tracks.

HAProxy Configuration Example

Because of HAProxy‘s structure and configuration flexibility, many professionals and consumers alike often use it as a security tool. Case in point: by using the following configuration example, you can easily help protect your servers against POST attacks to prevent attackers from clogging resources and ultimately harming the well-being of not only your equipment but your entire organization at the same time.

frontend ft_myapp
 [...]
 option http-buffer-request
 timeout http-request 10s

As you can see, with just a few simple modifications, HAProxy can quickly and effortlessly remove POST attacks from the list of things you have to worry about on a daily basis with regards to your mission-critical business applications and API.
The option http-buffer-request instructs HAProxy to wait for the whole DATA before forwarding it to a server and the timeout http-request 10s option tells how much time HAProxy let to a client to send the whole POST.

Thanks to its functionality as a security tool, a reverse proxy and more in addition to its intended functionality as a load balancer, it’s easy to see why HAProxy is used by some of the largest sites on the Internet including Reddit, Tumblr, GitHub and more on a daily basis.

This function is available in the following versions of HAProxy:

Related links

Links

Packetshield: quand votre load-balancer vous protège contre les DDOS!

Les attaques par DDOS


Il y a quelques temps, nous avions publié sur ce blog, un article expliquant comment utiliser un load-balancer pour se protéger contre les attaques de type DDOS applicatif: http://blog.haproxy.com/2012/02/27/use-a-load-balancer-as-a-first-row-of-defense-against-ddos/.

Malheureusement, les attaques sont aujourd’hui sur plusieurs vecteurs, notamment réseau et applicatif.
Leur but est toujours le même: trouver un point faible dans l’architecture et le saturer:

  1. un firewall, nombre de sessions ou paquets par secondes
  2. un load-balancer, nombre sessions établies
  3. un serveur applicatif, nombre de requêtes par secondes
  4. etc…

Dans l’article précédent, nous avions vu comment protéger les serveurs d’applications en utilisant le load-balancer, ce qui correspond à la partie applicative de l’architecture.

Pour protéger la tête de l’architecture, le firewall, il faut en général investir beaucoup d’argent dans des équipements coûteux.
Par ailleurs, dans certains cas, lorsque l’ALOHA était connecté en direct sur Internet, il ne savait se protéger que modérément contre les attaques “réseau”.

C’est pourquoi, HAProxy Technologies a développé sa propre solution de protection contre les DDOS de niveau réseau. Cet outil s’appelle Packetshield.

Les attaques de niveau réseau sont assez simples à mettre en oeuvre pour l’attaquant, mais peuvent être dévastatrices pour la cible: lorsqu’un équipement est saturé, il bloque tout le trafic.
Un équipement peut se saturer de différentes manières:

  • un trop grand nombre de paquets par secondes
  • un trop grand nombre se sessions ouvertes (TCP et/ou UDP)
  • un trop gros volume de trafic entrant

(liste non exhaustive)

NOTE: pour ce dernier cas, il faut passer par un service externe de “nettoyage” du flux, qui renverra ensuite un trafic légitime vers votre architecture.

Packetshield: Protection contre les DDOS réseau

L’ALOHA, équipé de packetshield, se place en point d’entrée de l’infrastructure.
Packetshield va agir comme un Firewall stateful en analysant les flux réseaux le traversant. Seuls le trafic considéré comme légitime sera autorisé à passer.
Différents mécanismes rentrent en jeu lors de la protection:

  • analyse des paquets: les paquets non valides sont automatiquement bloqués
  • filtrage de flux: l’administrateur peut définir ce qui est autorisé ou pas
  • analyse des flux autorisés: Packetshield fait le trie pour savoir ce qui est légitime ou pas

Contrairement aux solutions existantes qui fonctionnent dans un mode stateless et qui génèrent beaucoup de faux positifs, Packetshield garantie qu’aucune session valide ne sera bloquée.

Disponibilité de Packetshield

Packetshield est disponible dans l’ALOHA en version appliance physique depuis le firmware 7.0.
La protection de l’interface gigabit (Max 1 million de paquets par secondes) est inclu gratuitement dans l’ALOHA.
Nos clients existants n’ont qu’à mettre à jour le firmware de leur appliance pour en bénéficier. Tout nouveau client achetant l’une de nos appliances bénéficiera lui aussi de cette protection, sans surcoût.

Packetshield est aussi disponible en version 10G (14 millions de paquets traités par secondes) sur nos toutes appliances dernier modèle ALB5100, peut importe la licence choisie, du 8K au 64K.

Mode de déploiement


De part son fonctionnement stateful, Packetshield est utilisé de manière optimal dans les déploiement suivant:

Packetshield peut fonctionner en mode stateless, sans ses fonctionnalités avancé. Dans ce mode, Packetshield est compatible avec les type de déploiement suivants:

Pour plus d’information

La documentation sur la configuration de Packetshield est disponible à cette adresse: http://haproxy.com/doc/aloha/7.0/packet_flood_protection/.

Pour plus d’information, un petit mail à contact (at) haproxy (point) com.

Liens

Apache cdorked backdoor detection

Apache Cdorked.A backdoor

This is a pretty recent attack, using Cpanel to change the Apache httpd binary by a compromised one which embeds a backdoor.

A few articles with more details are available here:
  * http://www.welivesecurity.com/2013/04/26/linuxcdorked-new-apache-backdoor-in-the-wild-serves-blackhole/
  * http://blog.sucuri.net/2013/04/apache-binary-backdoors-on-cpanel-based-servers.html

It seems there are a few ways to detect if your server was compromised:
  1. requests with “GET_BACK;” encoded in the query string may arrive
  2. an unexpected Cookie (SECID in that case) may be sent to the server
  3. the server may answer some unexpected Location headers

Configuration

The HAProxy configuration below provides a few hints on how you can detect if you’ve been infected by the backdoor and how you can try to protect users using your services

I consider the website hostname is “www.domain.tld” and static content is delivered by “static.domain.tld”.

The configuration below can be added in the Frontend section:

# We want to capture and log the cookies sent by the client
 capture request header Cookie Len 128
# We want to capture and log the Location header sent by the server
 capture response header Location Len 128

# block any request with GET_BACK; string encoded
 http-request deny if { url_sub 4745545f4241434b3b } 
# block any request with a weird cookie
 http-request deny if { cook_cnt(SECID) ge 1 }

# block a response with a Location header for a unknown domain
 rspdeny ^Location: http://(www|static).domain.tld.*

Note that with such backdoor, you may have to monitor your logs (detection phase) first to know if you’ve been affected. Then you can update your configuration to block the attack (protection phase) and of course, you should remove the bad apache binary.

Links

wordpress CMS brute force protection with HAProxy

Brute force attacks

Brute force is a pretty simple type of attacks: it consists of massively send requests to a URL with different parameter each time. The main purpose is to try to find the right parameter combination.
Usually, brute force is used to discover login/password credentials to enter into a web application.

Fortunately, brute force are easy to detect, and latest HAProxy version have everything they need to protect any web application web form from brute forcing.
In the current article, we’ll apply the configuration on a wordpress CMS (content Management System), which was brute forced around mid of april 2013.

Note that the ALOHA Load-Balancer firmware 5.5 and above also includes the features introduced here.

WordPress login

WordPress provide the login form through the URL: /wp-login.php. Of course, it’s a regular GET.
The user just fill up the form, then the browser sends a POST on the URL /wp-login.php with form data in the body.
So basically, a brute force attacker will forge POST request on /wp-login.php and trying many different form data combination.
Below, an example of a forged request to try the credential admin/admin:

log=admin&pwd=admin&wp-submit=Log+In&redirect_to=http%3A%2F%www.domain.tld%2Fwp-admin%2F&testcookie=1

We can clearly see the log and pwd fields.

Blocking a brute force ???? Better sandboxing it smartly!!!!


It would be easy with HAProxy to drop the TCP connection, or to answer HTTP deny (403) status codes when we see somebody is abusing.
Actually, the attacker could use these information to know the maximum request rate he can achieve without being blacklisted.

The current article proposes to send abusers into a sandbox which keeps on delivering a static version of the login form, letting the abuser trying to hack your site, but actually hacking a static page 🙂
Furthermore, HAProxy will also slowdown the abusers request rate by tarpitting the request during 1s.

Brute force protection with HAProxy / ALOHA Load-Balancer

The configration is split in 2 parts:
  1. in the frontend, we store the list of blocked users
  2. in the backend, we do the brute force detection and we notify the frontend when an abuser is detected

Configuration for Brute force detection in the backend

This configuration stores a hash of 3 elements: HTTP Host header, URL path and source IP.
We’ll enable tracking only when the requests occur on the wordpress login URL (/wp-login.php) and if the method is POST.
Based on this, we can track the number of HTTP request the source IP did over a period of 20s and we can decide if the source IP did more than 5 login tentative during this period of time, then we want to flag this user as an abuser.
Basically 5 tries are allowed per 20s, over this limit, then the 6th try will make the user blocked.

[...]
  tcp-request inspect-delay 10s
  tcp-request content accept if HTTP
  # brute force protection
  acl wp_login                path_beg -i /wp-login.php
  stick-table type binary len 20 size 500 store http_req_rate(20s) peers local
  tcp-request content track-sc2  base32+src if METH_POST wp_login
  stick store-request base32+src            if METH_POST wp_login
  acl bruteforce_detection  sc2_http_req_rate gt 5
  acl flag_bruteforce       sc1_inc_gpc0      gt 0
  http-request deny if bruteforce_detection flag_bruteforce
[...]

Configuration for blocking abusers in the frontend


The configuration below detects that a user has abused the login page and then redirect him into a sandbox where HAProxy has been configured to serve a wordpress login page.
Which means the attacker will still think he is trying to brute force wordpress, but actually, he will brute force a static page !!!!! It will be impossible for him to know he has been sandboxed…
  * Frontend configuration:

  tcp-request inspect-delay 10s
  tcp-request accept if HTTP
[...]
  acl wp_login                 path_beg     -i /wp-login.php
  acl flagged_as_abuser        sc1_get_gpc0 gt 0
  stick-table type binary len 20 size 500 store gpc0 peers local
  tcp-request content track-sc1  base32+src if METH_POST wp_login
  use_backend bk_login_abusers if flagged_as_abuser
[...]

  * sandbox backend configuration:

[...]
backend bk_login_abusers
  mode http
  log global
  option httplog
  timeout tarpit 1s
  http-request tarpit
  errorfile 500 /etc/haproxy/pages/wp_fake_login.http
  errorfile 503 /etc/haproxy/pages/wp_fake_login.http
[...]

  * Errorfile content example is provided at the bottom of this article, in the Apendice section

The protection in action


Below, an extract of HAProxy logs (anonymized) which show the blocking capacity of the configuration above:

[...]
 ft_www bk_wordpress/w1 "POST /wp-login.php HTTP/1.1"
 ft_www bk_wordpress/w1 "POST /wp-login.php HTTP/1.1"
 ft_www bk_wordpress/w1 "POST /wp-login.php HTTP/1.1"
 ft_www bk_wordpress/w1 "POST /wp-login.php HTTP/1.1"
 ft_www bk_wordpress/w1 "POST /wp-login.php HTTP/1.1"
 ft_www bk_login_abusers/<NOSRV> "POST /wp-login.php HTTP/1.1"
 ft_www bk_login_abusers/<NOSRV> "POST /wp-login.php HTTP/1.1"
 ft_www bk_login_abusers/<NOSRV> "POST /wp-login.php HTTP/1.1"
[...]

5 attempts before being redirected to the sendbox, and still attempting 😉

Let’s have a look at the stick table:

# table: ft_www, type: binary, size:500, used:1
0x24f81e4: key=57FD750958BE12B3000000000000000000000000 use=0 exp=0 gpc0=1

# table: bk_wordpress, type: binary, size:500, used:1
0x24f8740: key=57FD750958BE12B3000000000000000000000000 use=0 exp=0 server_id=1 http_req_rate(20000)=6

Even if the http_req_rate decrease, as long as gpc0 is greater than 0 in the ft_www frontend stick-table, the user will be redirected to the sandbox.

Links

Appendice


  * Errorfile content, which is the wordpress login page content:
Don’t forget to change the wwww.domain.tld by your own domain, and don’t forget to update the Content-Length header using the following script from our github: errorfile_content_length

[...]
HTTP/1.0 200 OK
Server: webserver
Date: Fri, 26 Apr 2013 08:17:37 GMT
Content-Type: text/html; charset=UTF-8
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
Pragma: no-cache
Set-Cookie: wordpress_test_cookie=WP+Cookie+check; path=/
X-Frame-Options: SAMEORIGIN
Connection: close
Content-Length: 3253

<!DOCTYPE html>
        <html xmlns="http://www.w3.org/1999/xhtml" lang="en-US">
        <head>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
        <title>Bedis &rsaquo; Log In</title>
        <link rel='stylesheet' id='wp-admin-css'  href='http://www.domain.tld/wp-admin/css/wp-admin.min.css?ver=3.5.1' type='text/css' media='all' />
<link rel='stylesheet' id='buttons-css'  href='http://www.domain.tld/wp-includes/css/buttons.min.css?ver=3.5.1' type='text/css' media='all' />
<link rel='stylesheet' id='colors-fresh-css'  href='http://www.domain.tld/wp-admin/css/colors-fresh.min.css?ver=3.5.1' type='text/css' media='all' />
<link rel="stylesheet" href="http://www.domain.tld/wp-content/themes/notes-blog-core-theme/custom/login.css" type="text/css" media="screen" /><meta name='robots' content='noindex,nofollow' />
<script type="text/javascript">
addLoadEvent = function(func){if(typeof jQuery!="undefined")jQuery(document).ready(func);else if(typeof wpOnload!='function'){wpOnload=func;}else{var oldonload=wpOnload;wpOnload=function(){oldonload();func();}}};
function s(id,pos){g(id).left=pos+'px';}
function g(id){return document.getElementById(id).style;}
function shake(id,a,d){c=a.shift();s(id,c);if(a.length>0){setTimeout(function(){shake(id,a,d);},d);}else{try{g(id).position='static';wp_attempt_focus();}catch(e){}}}
addLoadEvent(function(){ var p=new Array(15,30,15,0,-15,-30,-15,0);p=p.concat(p.concat(p));var i=document.forms[0].id;g(i).position='relative';shake(i,p,20);});
</script>
        </head>
        <body class="login login-action-login wp-core-ui">
        <div id="login">
                <h1><a href="http://www.domain.tld/" title="Bedis Sites">Bedis</a></h1>
        <div id="login_error">  <strong>ERROR</strong>: Invalid username. <a href="http://www.domain.tld/wp-login.php?action=lostpassword" title="Password Lost and Found">Lost your password</a>?<br />
</div>

<form name="loginform" id="loginform" action="http://www.domain.tld/wp-login.php" method="post">
        <p>
                <label for="user_login">Username<br />
                <input type="text" name="log" id="user_login" class="input" value="" size="20" /></label>
        </p>
        <p>
                <label for="user_pass">Password<br />
                <input type="password" name="pwd" id="user_pass" class="input" value="" size="20" /></label>
        </p>
        <p class="forgetmenot"><label for="rememberme"><input name="rememberme" type="checkbox" id="rememberme" value="forever"  /> Remember Me</label></p>
        <p class="submit">
                <input type="submit" name="wp-submit" id="wp-submit" class="button button-primary button-large" value="Log In" />
                <input type="hidden" name="redirect_to" value="http://www.domain.tld/wp-admin/" />
                <input type="hidden" name="testcookie" value="1" />
        </p>
</form>

<p id="nav">
<a href="http://www.domain.tld/wp-login.php?action=lostpassword" title="Password Lost and Found">Lost your password?</a>
</p>

<script type="text/javascript">
function wp_attempt_focus(){
setTimeout( function(){ try{
d = document.getElementById('user_login');
if( d.value != '' )
d.value = '';
d.focus();
d.select();
} catch(e){}
}, 200);
}

if(typeof wpOnload=='function')wpOnload();
</script>

        <p id="backtoblog"><a href="http://www.domain.tld/" title="Are you lost?">&larr; Back to Bedis</a></p>

        </div>


                <div class="clear"></div>
        </body>
        </html>

[...]

Microsoft Remote Desktop Services (RDS) Load-Balancing and protection

RDS, RDP, TSE, remoteapp

Whatever you call it, it’s the remote desktop protocol from Microsoft, which has been renamed during the product life.
Basically, it allows users to get connected on remote server and run an application or a full desktop remotely.

It’s more and more fashion nowadays, because it’s easier for IT teams to patch and upgrade a single server hosting many remote desktops than maintaining many laptop individually.

VDI or Virtual Desktop Infrastructure


A Virtual Desktop Infrastructure is an architecture purposely built to export desktops to end users. It can use different type of product, such as Citrix or VMWare View.
Of course, you could build yours using Microsoft RDS as well as the ALOHA Load-Balancer to improve scalability and security of such platform.

Diagram

In a previous article, we already demonstrate how good the ALOHA Load-Balancer is to smartly load-balance Microsoft RDS.
Let’s start again from the same platform (I know, I’m a lazy engineer!):
rdp infrastructure

Windows Broker


When using the ALOHA Load-Balancer, you don’t need a windows broker anymore: the ALOHA is able to balance a new user to the less loaded server, as well as resume a broken session.
When an ALOHA outage occurs and a failover from the master to the slave happens, then all the sessions are resumed: the ALOHAs can share session affinity within a cluster, ensuring nobody will notice the issue.

Making an RDS platforms more secured


It’s easy to load-balance RDS protocol using the mstshash cookie, since microsoft described his protocol: it allows advanced persistence. Basically, each user owns a dedicated mstshash cookie value.

That said, being able to detect weird behavior or service abusers would be much better.

Actually, the ALOHA Load-Balancer allows you to monitor each source IP address, and, much better, each mstshash cookie value.
For any object, we can monitor the connection rate over a period of time, the number of current connection, the bytes in and out (over a period of time), etc…

Now, it’s up to you to protect your VDI infrastructure using the ALOHA Load-Balancer. In example:
  * counting the number of active connections per user (mstshash cookie)
  * counting the number of connection tries over the last 5 minutes per user

The above will allow you to detect the following problems / misbehavior on a VDI platform:
  * a user being abnormally connected too many times
  * a user trying to get connected to quickly, or maybe somebody is trying to discover someone else’s password 😉
  * etc…

Configuration

The configuration below explains how to create a highly-available VDI infrastructure with advanced persistence and improved security:

# for High-Availability purpose
peers aloha
  peer aloha1 10.0.0.16:1024
  peer aloha2 10.0.0.17:1024

# RDP / TSE configuration
frontend ft_rdp
  bind 10.0.0.18:3389 name rdp
  mode tcp
  timeout client 1h
  option tcplog
  option log global
  option tcpka
  # Protection 1
  # wait up to 5s for the mstshash cookie, and reject the client if none
  tcp-request inspect-delay 5s
  tcp-request content accept if RDP_COOKIE

  default_backend bk_rdp

backend bk_rdp
  mode tcp
  balance roundrobin
  # Options
  timeout server 1h
  timeout connect 4s
  option redispatch
  option tcpka
  option tcplog
  option log global
  # sticky persistence + monitoring based on mstshash Cookie
  #    established connection
  #    connection tries over the last minute
  stick-table type string len 32 size 10k expire 1d peers aloha store conn_cur,conn_rate(1m)
  stick on rdp_cookie(mstshash)

  # Protection 2
  # Each user is supposed to get a single active connection at a time, block the second one
  tcp-request content reject if { sc1_conn_cur ge 2 }

  # Protection 3
  # if a user tried to get connected at least 10 times over the last minute, 
  # it could be a brute force
  tcp-request content reject if { sc1_conn_rate ge 10 }

  # Server farm
  server tse1 10.0.0.23:3389 weight 10 check inter 2s rise 2 fall 3
  server tse2 10.0.0.24:3389 weight 10 check inter 2s rise 2 fall 3
  server tse3 10.0.0.25:3389 weight 10 check inter 2s rise 2 fall 3
  server tse4 10.0.0.26:3389 weight 10 check inter 2s rise 2 fall 3

Troubleshouting RDS connections issues with the ALOHA


The command below, when run from your ALOHA Load-Balancer CLI, will print out (and refresh every 1 second) the content of the stick-table with the following information:
  * table name, table size and number of current entries in the table
  * mstshash cookie value as sent by RDS client: field key
  * affected server id: field server_id
  * monitored counter and values, fields conn_rate and conn_cur

while true ; do clear ; echo show table bk_rdp | socat /tmp/haproxy.sock - ; sleep 1 ; done

stick-table output in a normal case


Below, an example of the output when everything goes well:

# table: bk_rdp, type: string, size:20480, used:2
0x912f84: key=Administrator use=1 exp=0 server_id=0 conn_rate(60000)=1 conn_cur=1
0x91ba14: key=XLC\Admin use=1 exp=0 server_id=1 conn_rate(60000)=1 conn_cur=1

stick-table output tracking a RDS abuser

Below, an example of the output when somebody tries many connections to discover someone’s password:

# table: bk_rdp, type: string, size:20480, used:2
0x912f84: key=Administrator use=1 exp=0 server_id=0 conn_rate(60000)=1 conn_cur=1
0x91ba14: key=XLC\Admin use=1 exp=0 server_id=1 conn_rate(60000)=1 conn_cur=1
0x91ca12: key=XLC\user1 use=1 exp=0 server_id=2 conn_rate(60000)=15 conn_cur=0

NOTE: in this case, the client has been blocked and the user user1 from the domain XLC can’t get connected for 1 minute (conn_rate monitoring period). I agree, this type of protection can lead to DOS your users…
We can easily configure the load-balancer to track source IP connection rate and is a single source IP tries to connect too often on our RDS farm, then we can block it before he can access it. (this type of attack would succeed only if the user is not only connected on the VDI platform)
Combining both source IP and mstshash cookie allows us to improve security on the VDI platform.

Securing a RDS platform combining both mstshash cookie and source IP address


The configuration below improve the protection by combining source IP address and RDS cookie.
The purpose is to block an abuser at a lower layer (IP) to avoid him making the Load-Balancer to blacklist regular user, even if this is temporary.

# for High-Availability purpose
peers aloha
  peer aloha1 10.0.0.16:1024
  peer aloha2 10.0.0.17:1024

# RDP / TSE configuration
frontend ft_rdp
  mode tcp
  bind 10.0.0.18:3389 name rdp
  timeout client 1h
  option tcpka
  log global
  option tcplog
  default_backend bk_rdp

backend bk_rdp
  mode tcp
  balance roundrobin

  # Options
  timeout server 1h
  timeout connect 4s
  option redispatch
  option tcpka
  log global
  option tcplog

  # sticky persistence + monitoring based on mstshash Cookie
  #    established connection
  #    connection tries over the last minute
  stick-table type string len 32 size 10k expire 1d peers aloha store conn_cur,conn_rate(1m)
  stick on rdp_cookie(mstshash)

  # Protection 1
  # wait up to 5s for the mstshash cookie, and reject the client if none
  tcp-request inspect-delay 5s
  tcp-request content accept if RDP_COOKIE
  tcp-request content track-sc1 rdp_cookie(mstshash) table bk_rdp
  tcp-request content track-sc2 src                  table sourceip

# IP layer protection
  #  Protection 2
  #   each single IP can have up to 2 connections on the VDI infrastructure
  tcp-request content reject if { sc2_conn_cur ge 2 }

  #  Protection 3
  #   each single IP can try up to 5 connections in a single minute
  tcp-request content reject if { sc2_conn_rate ge 10 }

# RDS / TSE application layer protection
  #  Protection 4
  #   Each user is supposed to get a single active connection at a time, block the second one
  tcp-request content reject if { sc1_conn_cur ge 2 }

  #  Protection 5
  #   if a user tried to get connected at least 10 times over the last minute, 
  #   it could be a brute force
  tcp-request content reject if { sc1_conn_rate ge 10 }

  # Server farm
  server tse1 10.0.0.23:3389 weight 10 check inter 2s rise 2 fall 3
  server tse2 10.0.0.24:3389 weight 10 check inter 2s rise 2 fall 3
  server tse3 10.0.0.25:3389 weight 10 check inter 2s rise 2 fall 3
  server tse4 10.0.0.26:3389 weight 10 check inter 2s rise 2 fall 3

backend sourceip
 stick-table type ip size 20k store conn_cur,conn_rate(1m)

Note that if you have remote sites using your VDI infrastructure, it is also possible to manage a source IP white list. And the same for the mstshash cookies: you can allow some users (like Administrator) to get connected multiple time on the VDI infrastructure.

Source IP and mstshash cookie combined protection


To observe the content of both tables (and still refreshed every 1s), just run the command below on your ALOHA CLI:

while true ; do clear ; echo show table bk_rdp | socat /tmp/haproxy.sock - ; echo "=====" ; echo show table sourceip | socat /tmp/haproxy.sock - ; sleep 1 ; done

And the result:

# table: bk_rdp, type: string, size:20480, used:2
0x23f1424: key=Administrator use=1 exp=0 server_id=0 conn_rate(60000)=0 conn_cur=1
0x2402904: key=XLC\Admin use=1 exp=0 server_id=1 conn_rate(60000)=1 conn_cur=1

=====
# table: sourceip, type: ip, size:20480, used:2
0x23f9f70: key=10.0.4.215 use=2 exp=0 conn_rate(60000)=2 conn_cur=2
0x23f1500: key=10.0.3.20 use=1 exp=0 conn_rate(60000)=0 conn_cur=1

Important Note about Microsoft RDS clients


As Mathew Levett stated on LoadBalancer.org’s blog (here is the article: http://blog.loadbalancer.org/microsoft-drops-support-for-mstshash-cookies/), latest version of Microsoft RDS clients don’t send the full user’s information into the mstshash cookie. Worst, sometime, the client don’t even send the cookie!!!!

In the example we saw above, there was 2 users connected on the infrastructure: the Administrator user was connected from a Linux client using xfreerdp tool, and we saw the cookie properly in the stick table. Second user, Administrator, from the Windows domain XLC, as his credential truncated in the stick table (Value: XLC\Admin), it was getting connected using Windows 2012 Server Enterprise RDS client.

As lb.org said, hopefully Microsoft will fix this issue. Or maybe there are some alternative RDS client for Windows…

To bypass this issue, we can take advantage of the ALOHA: we can do a persistence based on the source IP address when there is no mstshash cookies in the client connection information.
Just add the line below, right after the stick on rdp_cookie(mstshash):

  stick on src

With such configuration, the ALOHA Load-Balancer will stick on mstshash cookie, if present, and will failover on source IP address otherwise.

Conclusion


Virtualisation Desktop Infrastructure is a very fashion way of providing access to end users to a company’s applications.
If you don’t use the right product to build your RDS / TSE / RemoteApp infrastructure, you may be in trouble at some point.

The ALOHA Load-Balancer, thanks to its flexible configuration has many advantages

You can freely give a try to the ALOHA Load-Balancer, it is available from our website for download, allowing up to 10 connections with no time limitation: Download the ALOHA Load-Balancer

Links

high performance WAF platform with Naxsi and HAProxy

Synopsis

I’ve already described WAF in a previous article, where I spoke about WAF scalability with apache and modsecurity.
One of the main issue with Apache and modsecurity is the performance. To address this issue, an alternative exists: naxsi, a Web Application Firewall module for nginx.

So using Naxsi and HAProxy as a load-balancer, we’re able to build a platform which meets the following requirements:

  • Web Application Firewall: achieved by Apache and modsecurity
  • High-availability: application server and WAF monitoring, achieved by HAProxy
  • Scalability: ability to adapt capacity to the upcoming volume of traffic, achieved by HAProxy
  • DDOS protection: blind and brutal attacks protection, slowloris protection, achieved by HAProxy
  • Content-Switching: ability to route only dynamic requests to the WAF, achieved by HAProxy
  • Reliability: ability to detect capacity overusage, this is achieved by HAProxy
  • Performance: deliver response as fast as possible, achieved by the whole platform

The picture below provides a better overview:

The LAB platform is composed by 6 boxes:

  • 2 ALOHA Load-Balancers (could be replaced by HAProxy 1.5-dev)
  • 2 WAF servers: CentOS 6.0, nginx and Naxsi
  • 2 Web servers: Debian + apache + PHP + dokuwiki

Nginx and Naxsi installation on CentOS 6

Purpose of this article is not to provide such procedue. So please read this wiki article which summarizes how to install nginx and naxsi on CentOS 6.0.

Diagram

The diagram below shows the platform with HAProxy frontends (prefixed by ft_) and backends (prefixed by bk_). Each farm is composed by 2 servers.

Configuration

Nginx and Naxsi


Configure nginx as a reverse-proxy which listen in bk_waf and forward traffic to ft_web. In the mean time, naxsi is there to analyze the requests.

server {
 proxy_set_header Proxy-Connection "";
 listen       192.168.10.15:81;
 access_log  /var/log/nginx/naxsi_access.log;
 error_log  /var/log/nginx/naxsi_error.log debug;

 location / {
  include    /etc/nginx/test.rules;
  proxy_pass http://192.168.10.2:81/;
 }

 error_page 403 /403.html;
 location = /403.html {
  root /opt/nginx/html;
  internal;
 }

 location /RequestDenied {
  return 403;
 }
}

HAProxy Load-Balancer configuration


The configuration below allows the following advanced features:

  • DDOS protection on the frontend
  • abuser or attacker detection in bk_waf and blocking on the public interface (ft_waf)
  • Bypassing WAF when overusage or unavailable
######## Default values for all entries till next defaults section
defaults
  option  http-server-close
  option  dontlognull
  option  redispatch
  option  contstats
  retries 3
  timeout connect 5s
  timeout http-keep-alive 1s
  # Slowloris protection
  timeout http-request 15s
  timeout queue 30s
  timeout tarpit 1m          # tarpit hold tim
  backlog 10000

# public frontend where users get connected to
frontend ft_waf
  bind 192.168.10.2:80 name http
  mode http
  log global
  option httplog
  timeout client 25s
  maxconn 10000

  # DDOS protection
  # Use General Purpose Couter (gpc) 0 in SC1 as a global abuse counter
  # Monitors the number of request sent by an IP over a period of 10 seconds
  stick-table type ip size 1m expire 1m store gpc0,http_req_rate(10s),http_err_rate(10s)
  tcp-request connection track-sc1 src
  tcp-request connection reject if { sc1_get_gpc0 gt 0 }
  # Abuser means more than 100reqs/10s
  acl abuse sc1_http_req_rate(ft_web) ge 100
  acl flag_abuser sc1_inc_gpc0(ft_web)
  tcp-request content reject if abuse flag_abuser

  acl static path_beg /static/ /dokuwiki/images/
  acl no_waf nbsrv(bk_waf) eq 0
  acl waf_max_capacity queue(bk_waf) ge 1
  # bypass WAF farm if no WAF available
  use_backend bk_web if no_waf
  # bypass WAF farm if it reaches its capacity
  use_backend bk_web if static waf_max_capacity
  default_backend bk_waf

# WAF farm where users' traffic is routed first
backend bk_waf
  balance roundrobin
  mode http
  log global
  option httplog
  option forwardfor header X-Client-IP
  option httpchk HEAD /waf_health_check HTTP/1.0

  # If the source IP generated 10 or more http request over the defined period,
  # flag the IP as abuser on the frontend
  acl abuse sc1_http_err_rate(ft_waf) ge 10
  acl flag_abuser sc1_inc_gpc0(ft_waf)
  tcp-request content reject if abuse flag_abuser

  # Specific WAF checking: a DENY means everything is OK
  http-check expect status 403
  timeout server 25s
  default-server inter 3s rise 2 fall 3
  server waf1 192.168.10.15:81 maxconn 100 weight 10 check
  server waf2 192.168.10.16:81 maxconn 100 weight 10 check

# Traffic secured by the WAF arrives here
frontend ft_web
  bind 192.168.10.2:81 name http
  mode http
  log global
  option httplog
  timeout client 25s
  maxconn 1000
  # route health check requests to a specific backend to avoid graph pollution in ALOHA GUI
  use_backend bk_waf_health_check if { path /waf_health_check }
  default_backend bk_web

# application server farm
backend bk_web
  balance roundrobin
  mode http
  log global
  option httplog
  option forwardfor
  cookie SERVERID insert indirect nocache
  default-server inter 3s rise 2 fall 3
  option httpchk HEAD /
  # get connected on the application server using the user ip
  # provided in the X-Client-IP header setup by ft_waf frontend
  source 0.0.0.0 usesrc hdr_ip(X-Client-IP)
  timeout server 25s
  server server1 192.168.10.11:80 maxconn 100 weight 10 cookie server1 check
  server server2 192.168.10.12:80 maxconn 100 weight 10 cookie server2 check

# backend dedicated to WAF checking (to avoid graph pollution)
backend bk_waf_health_check
  balance roundrobin
  mode http
  log global
  option httplog
  option forwardfor
  default-server inter 3s rise 2 fall 3
  timeout server 25s
  server server1 192.168.10.11:80 maxconn 100 weight 10 check
  server server2 192.168.10.12:80 maxconn 100 weight 10 check

Detecting attacks


On the load-balancer


The ft_waf frontend stick table tracks two information: http_req_rate and http_err_rate which are respectively the http request rate and the http error rate generated by a single IP address.
HAProxy would automatically block an IP which has generated more than 100 requests over a period of 10s or 10 errors (WAF detection 403 responses included) in 10s. The user is blocked for 1 minute as long as he keeps on abusing.
Of course, you can setup above values to whatever you need: it is fully flexible.

To know the status of IPs in your load-balancer, just run the command below:

echo show table ft_waf | socat /var/run/haproxy.stat - 
# table: ft_waf, type: ip, size:1048576, used:1
0xc33304: key=192.168.10.254 use=0 exp=4555 gpc0=0 http_req_rate(10000)=1 http_err_rate(10000)=1

Note: The ALOHA Load-balancer does not provide watch tool, but you can monitor the content of the table in live with the command below:

while true ; do echo show table ft_waf | socat /var/run/haproxy.stat - ; sleep 2 ; clear ; done

On the Waf


Every Naxsi error log appears in /var/log/nginx/naxsi_error.log. IE:

2012/10/16 13:40:13 [error] 10556#0: *10293 NAXSI_FMT: ip=192.168.10.254&server=192.168.10.15&uri=/testphp.vulnweb.com/artists.php&total_processed=3195&total_blocked=2&zone0=ARGS&id0=1000&var_name0=artist, client: 192.168.10.254, server: , request: "GET /testphp.vulnweb.com/artists.php?artist=0+div+1+union%23foo*%2F*bar%0D%0Aselect%23foo%0D%0A1%2C2%2Ccurrent_user HTTP/1.1", host: "192.168.10.15:81"

Naxsi log line is less obvious than modsecurity one. The rule which matched os provided by the argument idX=abcde.
No false positive during the test, I had to build a request to make Naxsi match it 🙂 .

conclusion


Today, we saw it’s easy to build a scalable and performing WAF platform in front of any web application.
The WAF is able to communicate to HAProxy which IPs to automatically blacklist (throuth error rate monitoring), which is convenient since the attacker won’t bother the WAF for a certain amount of time 😉
The platform allows to detect WAF farm availability and to bypass it in case of total failure, we even saw it is possible to bypass the WAF for static content if the farm is running out of capacity. Purpose is to deliver a good end-user experience without dropping too much the security.
Note that it is possible to route all the static content to the web servers (or a static farm) directly, whatever the status of the WAF farm.
This make me say that the platform is fully scallable and flexible.
Thanks to HAProxy, the architecture is very flexible: I could switch my apache + modexurity to nginx + naxsi with no issues at all 🙂 This could be done as well for any third party waf appliances.
Note that I did not try any naxsi advanced features like learning mode and the UI as well.

Related links

Links

HTTP request flood mitigation

In a recent article, we saw how we can use a load-balancer as a first row of defense against DDOS.

The purpose of the present article to provide a configuration to protect your applications against HTTP request flood.

The configuration below allows only 10 requests per source IP over a period of 10s for the dynamic part of the website.
If the user go above this limit, it gets blacklisted until he stops browsing the website for 10 seconds.
HAProxy would return him a 403 for requests over an established connection and would refuse any new connection from this user.

# On Aloha, the global section is already setup for you
# and the haproxy stats socket is available at /var/run/haproxy.stats
global
  stats socket ./haproxy.stats level admin

defaults
  option http-server-close
  mode http
  timeout http-request 5s
  timeout connect 5s
  timeout server 10s
  timeout client 30s

# On Aloha, you don't need to set up the stats page, the GUI already provides
# all the necessary information
listen stats
  bind 0.0.0.0:8880
  stats enable
  stats hide-version
  stats uri     /
  stats realm   HAProxy Statistics
  stats auth    admin:admin

frontend ft_web
  bind 0.0.0.0:8080

  # Use General Purpose Couter (gpc) 0 in SC1 as a global abuse counter
  # Monitors the number of request sent by an IP over a period of 10 seconds
  stick-table type ip size 1m expire 10s store gpc0,http_req_rate(10s)
  tcp-request connection track-sc1 src
  # refuses a new connection from an abuser
  tcp-request content reject if { src_get_gpc0 gt 0 }
  # returns a 403 for requests in an established connection
  http-request deny if { src_get_gpc0 gt 0 }

  # Split static and dynamic traffic since these requests have different impacts on the servers
  use_backend bk_web_static if { path_end .jpg .png .gif .css .js }

  default_backend bk_web

# Dynamic part of the application
backend bk_web
  balance roundrobin
  cookie MYSRV insert indirect nocache

  # If the source IP sent 10 or more http request over the defined period, 
  # flag the IP as abuser on the frontend
  acl abuse src_http_req_rate(ft_web) ge 10
  acl flag_abuser src_inc_gpc0(ft_web) ge 0
  # Returns a 403 to the abuser
  http-request deny if abuse flag_abuser

  server srv1 192.168.1.2:80 check cookie srv1 maxconn 100
  server srv2 192.168.1.3:80 check cookie srv2 maxconn 100

# Static objects
backend bk_web_static
  balance roundrobin
  server srv1 192.168.1.2:80 check maxconn 1000
  server srv2 192.168.1.3:80 check maxconn 1000

Links