Apache cdorked backdoor detection

Apache Cdorked.A backdoor

This is a pretty recent attack, using Cpanel to change the Apache httpd binary by a compromised one which embeds a backdoor.

A few articles with more details are available here:
  * http://www.welivesecurity.com/2013/04/26/linuxcdorked-new-apache-backdoor-in-the-wild-serves-blackhole/
  * http://blog.sucuri.net/2013/04/apache-binary-backdoors-on-cpanel-based-servers.html

It seems there are a few ways to detect if your server was compromised:
  1. requests with “GET_BACK;” encoded in the query string may arrive
  2. an unexpected Cookie (SECID in that case) may be sent to the server
  3. the server may answer some unexpected Location headers


The HAProxy configuration below provides a few hints on how you can detect if you’ve been infected by the backdoor and how you can try to protect users using your services

I consider the website hostname is “www.domain.tld” and static content is delivered by “static.domain.tld”.

The configuration below can be added in the Frontend section:

# We want to capture and log the cookies sent by the client
 capture request header Cookie Len 128
# We want to capture and log the Location header sent by the server
 capture response header Location Len 128

# block any request with GET_BACK; string encoded
 http-request deny if { url_sub 4745545f4241434b3b } 
# block any request with a weird cookie
 http-request deny if { cook_cnt(SECID) ge 1 }

# block a response with a Location header for a unknown domain
 rspdeny ^Location:\ http://(www|static).domain.tld.*

Note that with such backdoor, you may have to monitor your logs (detection phase) first to know if you’ve been affected. Then you can update your configuration to block the attack (protection phase) and of course, you should remove the bad apache binary.


Posted in security | Tagged , , | Leave a comment

wordpress CMS brute force protection with HAProxy

Brute force attacks

Brute force is a pretty simple type of attacks: it consists of massively send requests to a URL with different parameter each time. The main purpose is to try to find the right parameter combination.
Usually, brute force is used to discover login/password credentials to enter into a web application.

Fortunately, brute force are easy to detect, and latest HAProxy version have everything they need to protect any web application web form from brute forcing.
In the current article, we’ll apply the configuration on a wordpress CMS (content Management System), which was brute forced around mid of april 2013.

Note that the ALOHA Load-Balancer firmware 5.5 and above also includes the features introduced here.

WordPress login

WordPress provide the login form through the URL: /wp-login.php. Of course, it’s a regular GET.
The user just fill up the form, then the browser sends a POST on the URL /wp-login.php with form data in the body.
So basically, a brute force attacker will forge POST request on /wp-login.php and trying many different form data combination.
Below, an example of a forged request to try the credential admin/admin:


We can clearly see the log and pwd fields.

Blocking a brute force ???? Better sandboxing it smartly!!!!

It would be easy with HAProxy to drop the TCP connection, or to answer HTTP deny (403) status codes when we see somebody is abusing.
Actually, the attacker could use these information to know the maximum request rate he can achieve without being blacklisted.

The current article proposes to send abusers into a sandbox which keeps on delivering a static version of the login form, letting the abuser trying to hack your site, but actually hacking a static page :)
Furthermore, HAProxy will also slowdown the abusers request rate by tarpitting the request during 1s.

Brute force protection with HAProxy / ALOHA Load-Balancer

The configration is split in 2 parts:
  1. in the frontend, we store the list of blocked users
  2. in the backend, we do the brute force detection and we notify the frontend when an abuser is detected

Configuration for Brute force detection in the backend

This configuration stores a hash of 3 elements: HTTP Host header, URL path and source IP.
We’ll enable tracking only when the requests occur on the wordpress login URL (/wp-login.php) and if the method is POST.
Based on this, we can track the number of HTTP request the source IP did over a period of 20s and we can decide if the source IP did more than 5 login tentative during this period of time, then we want to flag this user as an abuser.
Basically 5 tries are allowed per 20s, over this limit, then the 6th try will make the user blocked.

  tcp-request inspect-delay 10s
  tcp-request content accept if HTTP
  # brute force protection
  acl wp_login                path_beg -i /wp-login.php
  stick-table type binary len 20 size 500 store http_req_rate(20s) peers local
  tcp-request content track-sc2  base32+src if METH_POST wp_login
  stick store-request base32+src            if METH_POST wp_login
  acl bruteforce_detection  sc2_http_req_rate gt 5
  acl flag_bruteforce       sc1_inc_gpc0      gt 0
  http-request deny if bruteforce_detection flag_bruteforce

Configuration for blocking abusers in the frontend

The configuration below detects that a user has abused the login page and then redirect him into a sandbox where HAProxy has been configured to serve a wordpress login page.
Which means the attacker will still think he is trying to brute force wordpress, but actually, he will brute force a static page !!!!! It will be impossible for him to know he has been sandboxed…
  * Frontend configuration:

  tcp-request inspect-delay 10s
  tcp-request accept if HTTP
  acl wp_login                 path_beg     -i /wp-login.php
  acl flagged_as_abuser        sc1_get_gpc0 gt 0
  stick-table type binary len 20 size 500 store gpc0 peers local
  tcp-request content track-sc1  base32+src if METH_POST wp_login
  use_backend bk_login_abusers if flagged_as_abuser

  * sandbox backend configuration:

backend bk_login_abusers
  mode http
  log global
  option httplog
  timeout tarpit 1s
  http-request tarpit
  errorfile 500 /etc/haproxy/pages/wp_fake_login.http
  errorfile 503 /etc/haproxy/pages/wp_fake_login.http

  * Errorfile content example is provided at the bottom of this article, in the Apendice section

The protection in action

Below, an extract of HAProxy logs (anonymized) which show the blocking capacity of the configuration above:

 ft_www bk_wordpress/w1 "POST /wp-login.php HTTP/1.1"
 ft_www bk_wordpress/w1 "POST /wp-login.php HTTP/1.1"
 ft_www bk_wordpress/w1 "POST /wp-login.php HTTP/1.1"
 ft_www bk_wordpress/w1 "POST /wp-login.php HTTP/1.1"
 ft_www bk_wordpress/w1 "POST /wp-login.php HTTP/1.1"
 ft_www bk_login_abusers/<NOSRV> "POST /wp-login.php HTTP/1.1"
 ft_www bk_login_abusers/<NOSRV> "POST /wp-login.php HTTP/1.1"
 ft_www bk_login_abusers/<NOSRV> "POST /wp-login.php HTTP/1.1"

5 attempts before being redirected to the sendbox, and still attempting ;)

Let’s have a look at the stick table:

# table: ft_www, type: binary, size:500, used:1
0x24f81e4: key=57FD750958BE12B3000000000000000000000000 use=0 exp=0 gpc0=1

# table: bk_wordpress, type: binary, size:500, used:1
0x24f8740: key=57FD750958BE12B3000000000000000000000000 use=0 exp=0 server_id=1 http_req_rate(20000)=6

Even if the http_req_rate decrease, as long as gpc0 is greater than 0 in the ft_www frontend stick-table, the user will be redirected to the sandbox.



  * Errorfile content, which is the wordpress login page content:
Don’t forget to change the wwww.domain.tld by your own domain, and don’t forget to update the Content-Length header using the following script from our github: errorfile_content_length

HTTP/1.0 200 OK
Server: webserver
Date: Fri, 26 Apr 2013 08:17:37 GMT
Content-Type: text/html; charset=UTF-8
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
Pragma: no-cache
Set-Cookie: wordpress_test_cookie=WP+Cookie+check; path=/
X-Frame-Options: SAMEORIGIN
Connection: close
Content-Length: 3253

<!DOCTYPE html>
        <html xmlns="http://www.w3.org/1999/xhtml" lang="en-US">
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
        <title>Bedis › Log In</title>
        <link rel='stylesheet' id='wp-admin-css'  href='http://www.domain.tld/wp-admin/css/wp-admin.min.css?ver=3.5.1' type='text/css' media='all' />
<link rel='stylesheet' id='buttons-css'  href='http://www.domain.tld/wp-includes/css/buttons.min.css?ver=3.5.1' type='text/css' media='all' />
<link rel='stylesheet' id='colors-fresh-css'  href='http://www.domain.tld/wp-admin/css/colors-fresh.min.css?ver=3.5.1' type='text/css' media='all' />
<link rel="stylesheet" href="http://www.domain.tld/wp-content/themes/notes-blog-core-theme/custom/login.css" type="text/css" media="screen" /><meta name='robots' content='noindex,nofollow' />
<script type="text/javascript">
addLoadEvent = function(func){if(typeof jQuery!="undefined")jQuery(document).ready(func);else if(typeof wpOnload!='function'){wpOnload=func;}else{var oldonload=wpOnload;wpOnload=function(){oldonload();func();}}};
function s(id,pos){g(id).left=pos+'px';}
function g(id){return document.getElementById(id).style;}
function shake(id,a,d){c=a.shift();s(id,c);if(a.length>0){setTimeout(function(){shake(id,a,d);},d);}else{try{g(id).position='static';wp_attempt_focus();}catch(e){}}}
addLoadEvent(function(){ var p=new Array(15,30,15,0,-15,-30,-15,0);p=p.concat(p.concat(p));var i=document.forms[0].id;g(i).position='relative';shake(i,p,20);});
        <body class="login login-action-login wp-core-ui">
        <div id="login">
                <h1><a href="http://www.domain.tld/" title="Bedis Sites">Bedis</a></h1>
        <div id="login_error">  <strong>ERROR</strong>: Invalid username. <a href="http://www.domain.tld/wp-login.php?action=lostpassword" title="Password Lost and Found">Lost your password</a>?<br />

<form name="loginform" id="loginform" action="http://www.domain.tld/wp-login.php" method="post">
                <label for="user_login">Username<br />
                <input type="text" name="log" id="user_login" class="input" value="" size="20" /></label>
                <label for="user_pass">Password<br />
                <input type="password" name="pwd" id="user_pass" class="input" value="" size="20" /></label>
        <p class="forgetmenot"><label for="rememberme"><input name="rememberme" type="checkbox" id="rememberme" value="forever"  /> Remember Me</label></p>
        <p class="submit">
                <input type="submit" name="wp-submit" id="wp-submit" class="button button-primary button-large" value="Log In" />
                <input type="hidden" name="redirect_to" value="http://www.domain.tld/wp-admin/" />
                <input type="hidden" name="testcookie" value="1" />

<p id="nav">
<a href="http://www.domain.tld/wp-login.php?action=lostpassword" title="Password Lost and Found">Lost your password?</a>

<script type="text/javascript">
function wp_attempt_focus(){
setTimeout( function(){ try{
d = document.getElementById('user_login');
if( d.value != '' )
d.value = '';
} catch(e){}
}, 200);

if(typeof wpOnload=='function')wpOnload();

        <p id="backtoblog"><a href="http://www.domain.tld/" title="Are you lost?">← Back to Bedis</a></p>


                <div class="clear"></div>

Posted in HAProxy, security | Tagged , , , , | 4 Comments

Client IP persistence OR source IP hash load-balancing?

Client or Source IP ???

Well, this is roughly the same! Depends on people, environment, products, etc… I may use both of them in this article, but be aware that both of them points to the IP that is being used to get connected on the service whose being load-balanced.

Load-Balancing and Stickiness

Load-Balancing is the ability to spread requests among a server pool which deliver the same service. By definition, it means that any request can be sent to any server in the pool.
Some applications require stickiness between a client and a server: it means all the requests from a client must be sent to the same server. Otherwise, the application session may be broken and that may have a negative impact on the client.

Source IP stickiness

We may have many ways to stick a user to a server, which has already been discussed on this blog (Read load balancing, affinity, persistence, sticky sessions: what you need to know) (and many other article may follow).

That said, sometimes, the only information we can “rely” on to perform stickiness is the client (or source) IP address.
Note this is not optimal because:
  * many clients can be “hidden” behind a single IP address (Firewall, proxy, etc…)
  * a client can change its IP address during the session
  * a client can use multiple IP addresses
  * etc…

Performing source IP affinity

There are two ways of performing source IP affinity:
  1. Using a dedicated load-balancing algorithm: a hash on the source IP
  2. Using a stick table in memory (and a roundrobin load-balancing algorithm)

Actually, the main purpose of this article was to introduce both methods which are quite often misunderstood, and to show pros and cons of each, so people can make the right decision when configuring their Load-Balancer.

Source IP hash load-balancing algorithm

This algorithm is deterministic. It means that if no elements involved in the hash computation, then the result will be the same. 2 equipment are able to apply the same hash, hence load-balance the same way, making load-balancer failover transparent.
A hash function is applied on the source IP address of the incoming request. The hash must take into account the number of servers and each server’s weight.
The following events can make the hash change and so may redirect traffic differently over the time:
  * a server in the pool goes down
  * a server in the pool goes up
  * a server weight change

The main issue with source IP hash loadbalancing algorithm, is that each change can redirect EVERYBODY to a different server!!!
That’s why, some good load-balancers have implemented a consistent hashing method which ensure that if a server fails, for example, only the client connected to this server are redirected.
The counterpart of consistent hashing is that it doesn’t provide a perfect hash, and so, in a farm of 4 servers, some may receive more clients than others.
Note that when a failed server comes back, its “sticked” users (determined by the hash) will be redirected to it.
There is no overhead in term of CPU or memory when using such algorithm.

Configuration example in HAProxy or in the ALOHA Load-Balancer:

balance source
hash-type consistent

Source IP persistence using a stick-table

A table in memory is created by the Load-balancer to store the source IP address and the affected server from the pool.
We can rely on any non-deterministic load-balancing algorithm, such as roundrobin or leastconn (it usually depends on the type of application you’re load-balancing).
Once a client is sticked to a server, he’s sticked until the entry in the table expires OR the server fails.
There is an overhead in memory, to store stickiness information. In HAProxy, the overhead is pretty low: 40MB for 1.000.000 IPv4 addresses.
One main advantage of using a stick table is that when a failed server comes back, no existing sessions will be redirected to it. Only new incoming IPs can reach it. So no impact on users.
It is also possible to synchronize tables in memory between multiple HAProxy or ALOHA Load-Balancers, making a LB failover transparent.

Configuration example in HAProxy or in the ALOHA Load-Balancer:

stick-table type ip size 1m expire 1h
stick on src


Posted in Aloha, architecture, layer4, layer7 | Tagged , , | Leave a comment

Microsoft Remote Desktop Services (RDS) Load-Balancing and protection

RDS, RDP, TSE, remoteapp

Whatever you call it, it’s the remote desktop protocol from Microsoft, which has been renamed during the product life.
Basically, it allows users to get connected on remote server and run an application or a full desktop remotely.

It’s more and more fashion nowadays, because it’s easier for IT teams to patch and upgrade a single server hosting many remote desktops than maintaining many laptop individually.

VDI or Virtual Desktop Infrastructure

A Virtual Desktop Infrastructure is an architecture purposely built to export desktops to end users. It can use different type of product, such as Citrix or VMWare View.
Of course, you could build yours using Microsoft RDS as well as the ALOHA Load-Balancer to improve scalability and security of such platform.


In a previous article, we already demonstrate how good the ALOHA Load-Balancer is to smartly load-balance Microsoft RDS.
Let’s start again from the same platform (I know, I’m a lazy engineer!):
rdp infrastructure

Windows Broker

When using the ALOHA Load-Balancer, you don’t need a windows broker anymore: the ALOHA is able to balance a new user to the less loaded server, as well as resume a broken session.
When an ALOHA outage occurs and a failover from the master to the slave happens, then all the sessions are resumed: the ALOHAs can share session affinity within a cluster, ensuring nobody will notice the issue.

Making an RDS platforms more secured

It’s easy to load-balance RDS protocol using the mstshash cookie, since microsoft described his protocol: it allows advanced persistence. Basically, each user owns a dedicated mstshash cookie value.

That said, being able to detect weird behavior or service abusers would be much better.

Actually, the ALOHA Load-Balancer allows you to monitor each source IP address, and, much better, each mstshash cookie value.
For any object, we can monitor the connection rate over a period of time, the number of current connection, the bytes in and out (over a period of time), etc…

Now, it’s up to you to protect your VDI infrastructure using the ALOHA Load-Balancer. In example:
  * counting the number of active connections per user (mstshash cookie)
  * counting the number of connection tries over the last 5 minutes per user

The above will allow you to detect the following problems / misbehavior on a VDI platform:
  * a user being abnormally connected too many times
  * a user trying to get connected to quickly, or maybe somebody is trying to discover someone else’s password ;)
  * etc…


The configuration below explains how to create a highly-available VDI infrastructure with advanced persistence and improved security:

# for High-Availability purpose
peers aloha
  peer aloha1
  peer aloha2

# RDP / TSE configuration
frontend ft_rdp
  bind name rdp
  mode tcp
  timeout client 1h
  option tcplog
  option log global
  option tcpka
  # Protection 1
  # wait up to 5s for the mstshash cookie, and reject the client if none
  tcp-request inspect-delay 5s
  tcp-request content accept if RDP_COOKIE

  default_backend bk_rdp

backend bk_rdp
  mode tcp
  balance roundrobin
  # Options
  timeout server 1h
  timeout connect 4s
  option redispatch
  option tcpka
  option tcplog
  option log global
  # sticky persistence + monitoring based on mstshash Cookie
  #    established connection
  #    connection tries over the last minute
  stick-table type string len 32 size 10k expire 1d peers aloha store conn_cur,conn_rate(1m)
  stick on rdp_cookie(mstshash)

  # Protection 2
  # Each user is supposed to get a single active connection at a time, block the second one
  tcp-request content reject if { sc1_conn_cur ge 2 }

  # Protection 3
  # if a user tried to get connected at least 10 times over the last minute, 
  # it could be a brute force
  tcp-request content reject if { sc1_conn_rate ge 10 }

  # Server farm
  server tse1 weight 10 check inter 2s rise 2 fall 3
  server tse2 weight 10 check inter 2s rise 2 fall 3
  server tse3 weight 10 check inter 2s rise 2 fall 3
  server tse4 weight 10 check inter 2s rise 2 fall 3

Troubleshouting RDS connections issues with the ALOHA

The command below, when run from your ALOHA Load-Balancer CLI, will print out (and refresh every 1 second) the content of the stick-table with the following information:
  * table name, table size and number of current entries in the table
  * mstshash cookie value as sent by RDS client: field key
  * affected server id: field server_id
  * monitored counter and values, fields conn_rate and conn_cur

while true ; do clear ; echo show table bk_rdp | socat /tmp/haproxy.sock - ; sleep 1 ; done

stick-table output in a normal case

Below, an example of the output when everything goes well:

# table: bk_rdp, type: string, size:20480, used:2
0x912f84: key=Administrator use=1 exp=0 server_id=0 conn_rate(60000)=1 conn_cur=1
0x91ba14: key=XLC\\Admin use=1 exp=0 server_id=1 conn_rate(60000)=1 conn_cur=1

stick-table output tracking a RDS abuser

Below, an example of the output when somebody tries many connections to discover someone’s password:

# table: bk_rdp, type: string, size:20480, used:2
0x912f84: key=Administrator use=1 exp=0 server_id=0 conn_rate(60000)=1 conn_cur=1
0x91ba14: key=XLC\\Admin use=1 exp=0 server_id=1 conn_rate(60000)=1 conn_cur=1
0x91ca12: key=XLC\\user1 use=1 exp=0 server_id=2 conn_rate(60000)=15 conn_cur=0

NOTE: in this case, the client has been blocked and the user user1 from the domain XLC can’t get connected for 1 minute (conn_rate monitoring period). I agree, this type of protection can lead to DOS your users…
We can easily configure the load-balancer to track source IP connection rate and is a single source IP tries to connect too often on our RDS farm, then we can block it before he can access it. (this type of attack would succeed only if the user is not only connected on the VDI platform)
Combining both source IP and mstshash cookie allows us to improve security on the VDI platform.

Securing a RDS platform combining both mstshash cookie and source IP address

The configuration below improve the protection by combining source IP address and RDS cookie.
The purpose is to block an abuser at a lower layer (IP) to avoid him making the Load-Balancer to blacklist regular user, even if this is temporary.

# for High-Availability purpose
peers aloha
  peer aloha1
  peer aloha2

# RDP / TSE configuration
frontend ft_rdp
  mode tcp
  bind name rdp
  timeout client 1h
  option tcpka
  log global
  option tcplog
  default_backend bk_rdp

backend bk_rdp
  mode tcp
  balance roundrobin

  # Options
  timeout server 1h
  timeout connect 4s
  option redispatch
  option tcpka
  log global
  option tcplog

  # sticky persistence + monitoring based on mstshash Cookie
  #    established connection
  #    connection tries over the last minute
  stick-table type string len 32 size 10k expire 1d peers aloha store conn_cur,conn_rate(1m)
  stick on rdp_cookie(mstshash)

  # Protection 1
  # wait up to 5s for the mstshash cookie, and reject the client if none
  tcp-request inspect-delay 5s
  tcp-request content accept if RDP_COOKIE
  tcp-request content track-sc1 rdp_cookie(mstshash) table bk_rdp
  tcp-request content track-sc2 src                  table sourceip

# IP layer protection
  #  Protection 2
  #   each single IP can have up to 2 connections on the VDI infrastructure
  tcp-request content reject if { sc2_conn_cur ge 2 }

  #  Protection 3
  #   each single IP can try up to 5 connections in a single minute
  tcp-request content reject if { sc2_conn_rate ge 10 }

# RDS / TSE application layer protection
  #  Protection 4
  #   Each user is supposed to get a single active connection at a time, block the second one
  tcp-request content reject if { sc1_conn_cur ge 2 }

  #  Protection 5
  #   if a user tried to get connected at least 10 times over the last minute, 
  #   it could be a brute force
  tcp-request content reject if { sc1_conn_rate ge 10 }

  # Server farm
  server tse1 weight 10 check inter 2s rise 2 fall 3
  server tse2 weight 10 check inter 2s rise 2 fall 3
  server tse3 weight 10 check inter 2s rise 2 fall 3
  server tse4 weight 10 check inter 2s rise 2 fall 3

backend sourceip
 stick-table type ip size 20k store conn_cur,conn_rate(1m)

Note that if you have remote sites using your VDI infrastructure, it is also possible to manage a source IP white list. And the same for the mstshash cookies: you can allow some users (like Administrator) to get connected multiple time on the VDI infrastructure.

Source IP and mstshash cookie combined protection

To observe the content of both tables (and still refreshed every 1s), just run the command below on your ALOHA CLI:

while true ; do clear ; echo show table bk_rdp | socat /tmp/haproxy.sock - ; echo "=====" ; echo show table sourceip | socat /tmp/haproxy.sock - ; sleep 1 ; done

And the result:

# table: bk_rdp, type: string, size:20480, used:2
0x23f1424: key=Administrator use=1 exp=0 server_id=0 conn_rate(60000)=0 conn_cur=1
0x2402904: key=XLC\\Admin use=1 exp=0 server_id=1 conn_rate(60000)=1 conn_cur=1

# table: sourceip, type: ip, size:20480, used:2
0x23f9f70: key= use=2 exp=0 conn_rate(60000)=2 conn_cur=2
0x23f1500: key= use=1 exp=0 conn_rate(60000)=0 conn_cur=1

Important Note about Microsoft RDS clients

As Mathew Levett stated on LoadBalancer.org’s blog (here is the article: http://blog.loadbalancer.org/microsoft-drops-support-for-mstshash-cookies/), latest version of Microsoft RDS clients don’t send the full user’s information into the mstshash cookie. Worst, sometime, the client don’t even send the cookie!!!!

In the example we saw above, there was 2 users connected on the infrastructure: the Administrator user was connected from a Linux client using xfreerdp tool, and we saw the cookie properly in the stick table. Second user, Administrator, from the Windows domain XLC, as his credential truncated in the stick table (Value: XLC\\Admin), it was getting connected using Windows 2012 Server Enterprise RDS client.

As lb.org said, hopefully Microsoft will fix this issue. Or maybe there are some alternative RDS client for Windows…

To bypass this issue, we can take advantage of the ALOHA: we can do a persistence based on the source IP address when there is no mstshash cookies in the client connection information.
Just add the line below, right after the stick on rdp_cookie(mstshash):

  stick on src

With such configuration, the ALOHA Load-Balancer will stick on mstshash cookie, if present, and will failover on source IP address otherwise.


Virtualisation Desktop Infrastructure is a very fashion way of providing access to end users to a company’s applications.
If you don’t use the right product to build your RDS / TSE / RemoteApp infrastructure, you may be in trouble at some point.

The ALOHA Load-Balancer, thanks to its flexible configuration has many advantages

You can freely give a try to the ALOHA Load-Balancer, it is available from our website for download, allowing up to 10 connections with no time limitation: Download the ALOHA Load-Balancer


Posted in Aloha, architecture, security, Virtual Desktop Infrastructure | Tagged , , , | 4 Comments

Configure syslog-ng to log readable HTTP URL from HAProxy

This tips is provided by Exosec.
Exosec provides a very good monitoring product called POM, based on Nagios with very strong value added such as very simple administration, application monitoring, etc…
For some of their project, they use either HAProxy or the ALOHA Load-Balancer (heh, what else???) and they export log entries into syslog-ng for storage and later analysis.

HAProxy’s log

HAProxy’s logs are very powerfull since they provide many information about the request and the status of the platform at the moment of the request.
For a readable HAProxy’s log description, please consult the ALOHA memo dedicated to HAProxy’s HTTP log line description.

One of the weakness of the log line is that it logs only the path and the query string of each URL. No server name neither protocol information.
Well, HAProxy allows us to log the Host header, which is fine and there is a tild ‘~’ after the frontend name when the connection is made over SSL.

Using syslog-ng flexible configuration, we can re-order things to make haproxy log the URL exactly like it was sent by the client, like:


HAProxy configuration

Note: this is a very minimalistic configuration, not recommended in production :)

 log local2

frontend ft_http
 bind ssl crt /etc/haproxy/haproxy.pem
 option http-server-close
 mode http
 log global
 option httplog
 # Mandatory to build the URL:
 capture request header Host       len 32
 # Optional, just for statistics:
 capture request header User-Agent len 200
 default_backend bk_http

backend bk_http
 option http-server-close
 mode http
 log global
 option httplog

 server srv1

Syslog-ng configuration

The configuration below will reproduce an HAProxy log line, but will replace the URL part by something more readable.

Note: if you capture a different number of HTTP headers in HAProxy (current example contains 2 captured headers), you may have to update the parser p_haproxy_headers_req and the destination d_haproxy_full.

source s_loopback { syslog(ip( port(514) transport("udp")); };

destination d_haproxy_full {
          group(adm) perm(0640) dir_perm(0750) template_escape(no)

filter f_haproxy { program("haproxy"); };
filter f_frontend_ssl { match("~ "); };

rewrite r_set_frontend_protocol {
  set("http", value("HAPROXY.FRONTEND_PROTOCOL") condition(filter(f_haproxy)));
  set("https", value("HAPROXY.FRONTEND_PROTOCOL") condition(filter(f_frontend_ssl)));

parser p_haproxy {
  csv-parser(columns("HAPROXY.CLIENT_IPPORT", "HAPROXY.DATE",
                     "HAPROXY.FRONTEND", "HAPROXY.BACKEND",
                     "HAPROXY.TIME", "HAPROXY.STATUS_CODE",
                     "HAPROXY.BYTES_READ", "HAPROXY.COOKIE_REQ",
                     "HAPROXY.RUN_STATE", "HAPROXY.QUEUE_STATE",
                     "HAPROXY.HEADERS_REQ", "HAPROXY.REQUEST")
             delimiters(" ")

parser p_haproxy_request {
  csv-parser(columns("HAPROXY.METHOD", "HAPROXY.URL",
             delimiters(" ")

parser p_haproxy_headers_req {
  csv-parser(columns("HAPROXY.HOST", "HAPROXY.USER_AGENT")

log {

This configuration is downloadable from HAProxy Technologies github: https://raw.github.com/exceliance/haproxy/master/logs/syslog-ng_full_http_url.conf

And the result would look like below at the end of the logged line:

[...] "GET http://test.domain.tld/blah HTTP/1.1"
[...] "GET https://test.domain.tld/blah HTTP/1.1"


Posted in HAProxy, optimization | Tagged , | Leave a comment

IIS 6.0 appsession cookie and PCI compliance


You’re using HAProxy or the ALOHA Load-Balancer to load-balance IIS 6.0 web applications and you want them to pass successfully PCI compliance test.
One of the pre-requisite is to force the cookie to be “HttpOnly”, in order to tell the browser to use this cookie for HTTP requests only, and “protect” it from local javascript access (to steal session information).
Unfortunately, II 6.0 is not able to setup such cookies. That’s why HAProxy can be used to update the cookie on the fly, when setup by the application server.

Rewriting appsession Cookie with HAProxy

Place the configuration line below in your backend configuration:

rspirep ^Set-Cookie:\ (appsession.*)    Set-Cookie:\ \1;\ HttpOnly

Now, you’re application is “more” secured… Well, at least, you can successfully pass the PCI compliancy tests!


Posted in Aloha, HAProxy, layer7, security | Tagged , , , | 1 Comment

SSL offloading impact on web applications

SSL Offloading

Nowadays, it is common (and convenient) to use the Load-Balancer SSL capabilities to cypher/uncypher traffic from clients to the web application platform.
Performing SSL at the Load-Balancer Layer is called “SSL offloading“, because you offload this process from your application servers.

Note that SSL offloading is also “marketingly” called SSL acceleration. The only acceleration performed is moving SSL processing from application server, which have an other heavy task to perform, to an other devices in the network…
So no acceleration at all, despite some people claming they do accelerate SSL in a dedicated ASIC, they just offload

The diagram below shows what happens on an architecture with SSL offloading. We can clearly see that the traffic is encrypted up to the Load-Balancer, then it’s in clear between the Load-Balancer and the application server:
ssl offloading diagram

Benefits of SSL offloading

Offloading SSL from the application servers to the Load-Balancers have many benefits:
  * offload an heavy task from your application servers, letting them to focus on the application itself
  * save resources on your application servers
  * the Load-Balancers have access to clear HTTP traffic and can perform advanced features such as reverse-proxying, Cookie persistence, traffic regulation, etc…
  * When using an ALOHA Load-Balancer (or HAProxy), there are much more features available on the SSL stack than on any web application server.

It seems there should only be positive impact when offloading SSL.

Counterparts of SSL offloading

As we saw on the diagram above, the load-balancer will hide three important information from the client connection:
  1. protocol scheme: the client was using HTTPS while the connection on the application server is made over HTTP
  2. connection type: was it cyphered or not? Actually, this is linked to the protocol scheme
  3. destination port: on the load-balancer, the port is usually 443 and on the application server should be 80 (this may change)

Most web application will use 301/302 responses with a Location header to redirect the client to a page (most of the time after a login or a POST) as well as requiring an application cookie.

So basically, the worst impact is that your application server may have difficulties to know the client connection information and may not be able to perform right responses: it can break totally the application, preventing it to work.
Fortunately, the ALOHA Load-Balancer or HAProxy can help in such situation!

Basically, the application could return such responses to the client:

Location: http://www.domain.com/login.jsp

And the client will leave the secured connection…

Tracking issues with the load-balancer

In order to know if your application supports well SSL offloading, you can configure your ALOHA Load-Balancer to log some server responses.
Add the configuration below in your application frontend section:

capture response header Location   len 32
capture response header Set-Cookie len 32

Now, in the traffic log, you’ll see clearly if the application is setting up response for HTTP or HTTPS.

NOTE: some other headers may be needed, depending on your application.

Web Applications and SSL offloading

Here we are :)

There are 3 answers possible for such situation, detailed below.

Client connection information provided in HTTP header by the Load-Balancer

First of all, the Load-Balancer can provide client side connection information to the application server through HTTP header.
The configuration below shows how to insert a header called X-Forwared-Proto containing the scheme used by the client.
To be added in your backend section.

http-request set-header X-Forwarded-Proto https if  { ssl_fc }
http-request set-header X-Forwarded-Proto http  if !{ ssl_fc }

Now, the ALOHA Load-Balancer will insert the following header when the connection is made over SSL:

X-Forwarded-Proto: https

and when performed over clear HTTP:

X-Forwarded-Proto: http

It’s up to your application to use this information to answer the right responses: this may require some code update.

HTTP header rewriting on the fly

An other way is to rewrite the response setup by the server on the fly.
The following configuration line would match the Location header and translate it from http to https if needed.
Rewriting rspirep http://www.domain.com:80/url:

rspirep ^Location:\ http://(.*):80(.*)  Location:\ https://\1:443\2   if  { ssl_fc }

NOTE: above example applies on HTTP redirection (Location header), but can be applied on Set-cookie as well (or any other header your application may use).
NOTE2: you can only rewrite the response HTTP headers, not in the body. So this is not compatible with applications setting up hard links in the HTML content.

SSL Bridging

In some cases, the application is not compatible at all with SSL offloading (even with the tricks above) and we must use a ciphered connection to the server but we still may require to perform cookie based persistence, content switching, etc…
This is called SSL bridging, or can also be called a man in the middle.

In the ALOHA, there is nothing to do, just do SSL offloading as usual and add the keyword ssl on your server directive.

Using this method, you can choose a light cipher and a light key between the Load-Balancer and the server and still use the Load-Balancer advanced SSL feature with a stronger key and a stronger cipher.
The application server won’t notice anything and the Load-Balancer can still perform Layer 7 processing.


Posted in Aloha, architecture, HAProxy, ssl | Tagged , , , | 8 Comments