Category Archives: security

Using HAProxy with the Proxy Protocol to Better Secure Your Database

The Proxy protocol is a widely used invention of our CTO at HAProxy Technologies, Willy Tarreau, to solve the problem of TCP connection parameters being lost when relaying TCP connections through proxies. Its primary purpose is to chain proxies and reverse-proxies without losing client information, and it’s used and supported by AWS ELB, Apache, NGINX, Varnish, Citrix, and many more (here’s a longer list of Proxy protocol supported technologies).

Why use the Proxy protocol? Well, when you lose client information like IP address when relaying connections through proxies, this tends to prevent you from being able to implement some pretty basic IP-level security and logging.

So you may not really know who you’re letting in to access your data:

Intruder-Revealed

In this post, I’ll show you how to use the Proxy protocol with HAProxy to enhance the security of your database.

Background

Let’s take a step back and explain a bit (Proxy protocol experts can skip this section). A proxy uses its IP stack to connect to remote servers, but this process normally loses the initial TCP connection data, including the source IP address, destination IP address and port number.

Proxy-protocol-figure-1

Traditional workarounds to this problem include Tproxy, which requires you to compile your kernel and set your proxy as your server’s default gateway. Furthermore, Tproxy can’t pass IP packets through firewalls that use NAT. You can also use HTTP X-Forwarded-For headers to retain TCP connection data, although this approach only works for HTTP. Many of these workarounds also require either a specific protocol or architectural changes, which prevents you from scaling up.

The Proxy protocol is protocol independent, meaning that it works with any layer-7 protocol. It doesn’t require infrastructure changes, works with NAT firewalls, and is scalable. The Proxy protocol’s only technical requirement is that both of the connection’s endpoints must be compatible with the Proxy protocol. These endpoints could include proxies, reverse-proxies, load-balancers, application servers and WAFs.

Proxy-protocol-figure-2

The Database Security Problem

An HAProxy server running without the Proxy protocol can create a couple security problems when you grant users access to the MySQL servers behind it.

First, Slow query logs and “show full processlist” commands on a MySQL server behind a HAProxy server that isn’t running the Proxy protocol won’t show the correct IP addresses of the clients, making it more difficult to identify hosts which are sending unoptimized, incorrect or queries injected via SQLi. Consider the following entry from a slow query log:


# Time: 2017-03-09T21:56:07.640875Z
# User@Host: test[test] @ centos7vert [192.168.122.64] Id: 11
# Schema: Last_errno: 0 Killed: 0
# Query_time: 7.026677 Lock_time: 0.025969 Rows_sent: 0 Rows_examined: 1 Rows_affected: 0
# Bytes_sent: 104
SET timestamp=1489096567;
SELECT * FROM test.test WHERE sleep(7);

The above example shows that a host on or behind the HAProxy server at 192.168.122.64 has issued a query that is suspicious or has performance issues, but we can’t identify the host. You could review the HAProxy logs for activity that occurred around the indicated time to identify the host, but this approach is usually impractical due to the delay between the timestamps of the HAProxy and MySQL logs.

The second security issue you may face from seeing the incorrect IP address is that MySQL grants can no longer allow just one client to use a given username if it is allowed to access MySQL through the HAProxy server via its firewall/acl’s, as to MySQL all IP’s look the same.  With the Proxy protocol you can maintain better privilege segregation between databases even if an attacker manages to get the password for a MySQL user dedicated for another client.

Using the Proxy protocol for a more secure database

Now here’s the good part: how to install, configure and use the Proxy protocol with a MySQL database. Note: you can use either HAProxy Community Edition or HAProxy Enterprise Edition with these instructions.

1. Percona installation

The instructions on the Percona website provide the details for installing Percona Server.

2. HAProxy configuration

Add a section to the HAProxy configuration file like the following:

frontend fe_mysqld
bind *:3306
mode tcp
log global
option tcplog
use_backend be_mysqld
backend be_mysqld
mode tcp
option mysql-check user haproxy post-41
server percona_server 192.168.122.185:3306 check

Add a grant to the Percona server so it will allow health checks with a MySQL command as follows:

mysql> CREATE USER 'haproxy'@'192.168.122.64';

Restart HAProxy after adding the grant statements to the Percona server.

3. New grant statements

A HAProxy server that doesn’t use the Proxy protocol requires you to add grant statements just for the IP address of the HAProxy server. The Proxy protocol requires you to add grants for servers that will be behind an HAProxy server as if the proxy wasn’t there. For example, assume that you want the host at IP address 192.168.122.1 to connect to the database. The following MySQL statement will add said grant:

mysql> CREATE USER 'test'@'192.168.122.1';

The HAProxy server will use these grants instead of the existing grants once the Proxy protocol has been enabled on the HAProxy and Percona servers.

4. Percona configuration

In the mysqld section of /etc/mysql/percona-server.conf.d/mysqld.cnf add the following line:

proxy_protocol_networks="192.168.122.64"

The above statement will accept a single IP address, multiple IP addresses separated by commas or a masked IP address like 192.168.122.0/24. An asterisk (“*”) for the IP address will cause the server to accept the Proxy protocol from any host, although this isn’t recommended for security reasons. These IPs must be trusted to send correct Proxy protocol information before adding them to the list.
Note that Percona won’t accept the Proxy protocol from 127.0.0.1 even if it is in proxy_protocol_networks; so if you are running HAProxy on the same server that you are running Percona on this issue may rear its ugly head.

Add an entry for each HAProxy server that will handle traffic to a Percona server. You must enable the Proxy protocol for the Percona and HAProxy servers or disable it for both at the same time. If the Proxy protocol configurations of the HAProxy and Percona servers don’t match, the connections will fail with a “packets received out of order” error. You may also want to configure the firewall on the Percona server to only accept connections from a HAProxy server, which ensures that clients must use a HAProxy server to access the Percona server.

The status of the Percona server will be marked as “down” in HAProxy after you enable the proxy_protocol_networks statement on the Percona server, provided you didn’t manually disable it during transmission (and are following this guide in order and haven’t added send-proxy yet). This problem occurs because Percona now requires the Proxy protocol but HAProxy isn’t yet sending it. You’ll fix this in the next step. Health checks will mark the node as down until send-proxy is added (and regular connections through it will encounter the same error).

5. Send-proxy

Append the following bolded argument to the server line (of the Percona server) in the backend section of the HAProxy configuration:

server percona_server 192.168.122.185:3306 check send-proxy

The above statement tells HAProxy to send the Proxy protocol packet for both health checks and normal connections.

If you are chaining multiple HAProxy servers together you may wish to use “send-proxy-v2” instead of “send-proxy”.  Its a newer version of the protocol that is binary rather than text so is faster to parse.  The downside is that servers need to support v2 of the protocol (Percona server does, so you can just put send-proxy-v2 in and it will work as expected) and that debugging its parsing is a bit more difficult from a packet capture.

With this added HAProxy will also send the Proxy protocol for health checks, but the result won’t be used by Percona as it won’t accept 127.0.0.1; as such your existing grants for health checks will continue to be used.

6. Rest easier with better security in place

Proxy protocol makes a Percona server more secure and easier to manage as shown by the now-correct bold text in the following Percona slow log entry:

# Time: 2017-03-09T22:41:20.195387Z
# User@Host: test[test] @ [192.168.122.1] Id: 540
# Schema: Last_errno: 0 Killed: 0
# Query_time: 7.039918 Lock_time: 0.038505 Rows_sent: 0 Rows_examined: 1 Rows_affected: 0
# Bytes_sent: 104
SET timestamp=1489099280;
SELECT * FROM test.test WHERE sleep(7);

The previous output is produced by the same query (and from the same node) that was shown in The Database Security Problem section, but we can now see that the client’s actual IP address is 192.168.122.1.

The IP addresses in the process list table are also more useful with the Proxy protocol as can be seen here:
mysql-processlist

You can also run grants on the MySQL server with the client’s real IP address instead of just the HAProxy server’s IP address. This change is primarily useful to better understand the items in the grant tables during later changes/audits and to ensure that users can only access select user accounts from select IP addresses.

So you go from letting anyone in who seems to have a legitimate credentials (with possible malicious intent):

Intruder-Accepted

To keeping out any users not coming from authorized IP addresses:

Intruder-Denied

Congratulations on your new level of data security!

Additional Resources

Want more information? Here’s another guide to using the Proxy protocol with a Percona server.

What is a slow POST Attack and how to turn HAProxy into your first line of Defense?

One of the biggest security challenges that companies face in today’s modern climate is the POST attack. Unlike a more traditional “Denial-of-Service” attack, POST attacks target a servers logical resources – making them particularly powerful when executed.

What is a slow POST Attack?


In a POST attack, an attacker begins by sending a legitimate HTTP POST header to a Web server, exactly as they would under normal circumstances. The header specifies the exact size of the message body that will then follow. However, that message body is then sent at an alarmingly low rate – sometimes as slow as 1 byte per approximately two minutes. Because the entire message is technically correct and complete, the targeted server attempts to obey all specified rules – which as you would expect, can take quite awhile. The issue is that if an attacker were to establish hundreds or even thousands of these POST attacks simultaneously, it will quickly use up all server resources and make legitimate connections impossible.

How HAProxy can protect against slow POST attack?


Because POST attacks can be incredibly powerful, it’s always important to have a tool in place to identify these types of issues when they’re still in their nascent stages to prevent them from becoming much larger, more serious issues down the road. Because HAProxy was designed as an application delivery controller to manage Web application high availability and performance, it is already in an ideal position to stop these types of POST attacks in their tracks.

HAProxy Configuration Example

Because of HAProxy‘s structure and configuration flexibility, many professionals and consumers alike often use it as a security tool. Case in point: by using the following configuration example, you can easily help protect your servers against POST attacks to prevent attackers from clogging resources and ultimately harming the well-being of not only your equipment but your entire organization at the same time.

frontend ft_myapp
 [...]
 option http-buffer-request
 timeout http-request 10s

As you can see, with just a few simple modifications, HAProxy can quickly and effortlessly remove POST attacks from the list of things you have to worry about on a daily basis with regards to your mission-critical business applications and API.
The option http-buffer-request instructs HAProxy to wait for the whole DATA before forwarding it to a server and the timeout http-request 10s option tells how much time HAProxy let to a client to send the whole POST.

Thanks to its functionality as a security tool, a reverse proxy and more in addition to its intended functionality as a load balancer, it’s easy to see why HAProxy is used by some of the largest sites on the Internet including Reddit, Tumblr, GitHub and more on a daily basis.

This function is available in the following versions of HAProxy:

Related links

Links

Serving ECC and RSA certificates on same IP with HAproxy

ECC and RSA certificates and HTTPS

To keep this practical, we will not go into theory of ECC or RSA certificates. Let’s just mention that ECC certificates can provide as much security as RSA with much lower key size, meaning much lower computation requirements on the server side. Sadly, many clients do not support ciphers based on ECC, so to maintain compatibility as well as provide good performance we need to be able to detect which type of certificate is supported by the client to be able to serve it correctly.

The above is usually achieved with analyzing the cipher suites sent by the client in the ClientHello message at the start of the SSL handshake, but we’ve opted for a much simpler approach that works very well with all modern browsers (clients).

Prerequisites

First you will need to obtain both RSA and ECC certificates for your web site. Depending on the registrar you are using, check their documentation. After you have been issued with the certificates, make sure you download the appropriate intermediate certificates and create the bundle files for HAproxy to read.

To be able to use the sample fetch required, you will need at least HAproxy 1.6-dev3 (not yet released as of writing) or you can clone latest HAproxy from the git repository. Feature was introduced in commit 5fc7d7e.

Configuration

We will use chaining in order to achieve desired functionality. You can use abstract sockets on Linux to get even more performance, but note the drawbacks that can be found in HAproxy documentation.

 frontend ssl-relay
 mode tcp
 bind 0.0.0.0:443
 use_backend ssl-ecc if { req.ssl_ec_ext 1 }
 default_backend ssl-rsa

 backend ssl-ecc
 mode tcp
 server ecc unix@/var/run/haproxy_ssl_ecc.sock send-proxy-v2

 backend ssl-rsa
 mode tcp
 server rsa unix@/var/run/haproxy_ssl_rsa.sock send-proxy-v2

 listen all-ssl
 bind unix@/var/run/haproxy_ssl_ecc.sock accept-proxy ssl crt /usr/local/haproxy/ecc.www.foo.com.pem user nobody
 bind unix@/var/run/haproxy_ssl_rsa.sock accept-proxy ssl crt /usr/local/haproxy/www.foo.com.pem user nobody
 mode http
 server backend_1 192.168.1.1:8000 check

The whole configuration revolves around the newly implemented sample fetch: req.ssl_ec_ext. What this fetch does is that it detects the presence of Supported Elliptic Curves Extension inside the ClientHello message. This extension is defined in RFC4492 and according to it, it SHOULD be sent with every ClientHello message by the client supporting ECC. We have observed that all modern clients send it correctly.

If the extension is detected, the client is sent through a unix socket to the frontend that will serve an ECC certificate. If not, a regular RSA certificate will be served.

Benchmark

We will provide full HAproxy benchmarks in the near future, but for the sake of comparison, let us view the difference present on an E5-2680v3 CPU and OpenSSL 1.0.2.

256bit ECDSA:
sign verify sign/s verify/s
0.0000s 0.0001s 24453.3 9866.9

2048bit RSA:
sign verify sign/s verify/s
0.000682s 0.000028s 1466.4 35225.1

As you can see, looking at the sign/s we are getting over 15 times the performance with ECDSA256 compared to RSA2048.

Packetshield: quand votre load-balancer vous protège contre les DDOS!

Les attaques par DDOS


Il y a quelques temps, nous avions publié sur ce blog, un article expliquant comment utiliser un load-balancer pour se protéger contre les attaques de type DDOS applicatif: http://blog.haproxy.com/2012/02/27/use-a-load-balancer-as-a-first-row-of-defense-against-ddos/.

Malheureusement, les attaques sont aujourd’hui sur plusieurs vecteurs, notamment réseau et applicatif.
Leur but est toujours le même: trouver un point faible dans l’architecture et le saturer:

  1. un firewall, nombre de sessions ou paquets par secondes
  2. un load-balancer, nombre sessions établies
  3. un serveur applicatif, nombre de requêtes par secondes
  4. etc…

Dans l’article précédent, nous avions vu comment protéger les serveurs d’applications en utilisant le load-balancer, ce qui correspond à la partie applicative de l’architecture.

Pour protéger la tête de l’architecture, le firewall, il faut en général investir beaucoup d’argent dans des équipements coûteux.
Par ailleurs, dans certains cas, lorsque l’ALOHA était connecté en direct sur Internet, il ne savait se protéger que modérément contre les attaques “réseau”.

C’est pourquoi, HAProxy Technologies a développé sa propre solution de protection contre les DDOS de niveau réseau. Cet outil s’appelle Packetshield.

Les attaques de niveau réseau sont assez simples à mettre en oeuvre pour l’attaquant, mais peuvent être dévastatrices pour la cible: lorsqu’un équipement est saturé, il bloque tout le trafic.
Un équipement peut se saturer de différentes manières:

  • un trop grand nombre de paquets par secondes
  • un trop grand nombre se sessions ouvertes (TCP et/ou UDP)
  • un trop gros volume de trafic entrant

(liste non exhaustive)

NOTE: pour ce dernier cas, il faut passer par un service externe de “nettoyage” du flux, qui renverra ensuite un trafic légitime vers votre architecture.

Packetshield: Protection contre les DDOS réseau

L’ALOHA, équipé de packetshield, se place en point d’entrée de l’infrastructure.
Packetshield va agir comme un Firewall stateful en analysant les flux réseaux le traversant. Seuls le trafic considéré comme légitime sera autorisé à passer.
Différents mécanismes rentrent en jeu lors de la protection:

  • analyse des paquets: les paquets non valides sont automatiquement bloqués
  • filtrage de flux: l’administrateur peut définir ce qui est autorisé ou pas
  • analyse des flux autorisés: Packetshield fait le trie pour savoir ce qui est légitime ou pas

Contrairement aux solutions existantes qui fonctionnent dans un mode stateless et qui génèrent beaucoup de faux positifs, Packetshield garantie qu’aucune session valide ne sera bloquée.

Disponibilité de Packetshield

Packetshield est disponible dans l’ALOHA en version appliance physique depuis le firmware 7.0.
La protection de l’interface gigabit (Max 1 million de paquets par secondes) est inclu gratuitement dans l’ALOHA.
Nos clients existants n’ont qu’à mettre à jour le firmware de leur appliance pour en bénéficier. Tout nouveau client achetant l’une de nos appliances bénéficiera lui aussi de cette protection, sans surcoût.

Packetshield est aussi disponible en version 10G (14 millions de paquets traités par secondes) sur nos toutes appliances dernier modèle ALB5100, peut importe la licence choisie, du 8K au 64K.

Mode de déploiement


De part son fonctionnement stateful, Packetshield est utilisé de manière optimal dans les déploiement suivant:

Packetshield peut fonctionner en mode stateless, sans ses fonctionnalités avancé. Dans ce mode, Packetshield est compatible avec les type de déploiement suivants:

Pour plus d’information

La documentation sur la configuration de Packetshield est disponible à cette adresse: http://haproxy.com/doc/aloha/7.0/packet_flood_protection/.

Pour plus d’information, un petit mail à contact (at) haproxy (point) com.

Liens

HAProxy and sslv3 poodle vulnerability

SSLv3 poodle vulnerability

Yesterday, Google security researchers have disclosed a new vulnerability on SSL protocol.
Fortunately, this vulnerability is only on an old version of the SSL protocol: SSLv3 (15 years old protocol).
An attacker can force a browser to downgrade the protocol version used to cipher traffic to SSLv3 in order to exploit the POODLE vulnerability and access to data in clear.

Some reading about SSLv3 Poodle vulnerability:
* http://googleonlinesecurity.blogspot.fr/2014/10/this-poodle-bites-exploiting-ssl-30.html
* https://www.imperialviolet.org/2014/10/14/poodle.html
* https://www.poodletest.com/

Today’s article is going to explain how to use HAProxy to simply prevent using SSLv3 or to prevent those users to reach your applications and print them a message.

Disable SSLv3 in HAProxy

In SSL offloading mode

In this mode, HAProxy is the SSL endpoint of the connection.
It’s a simple keyword on the frontend bind directive:

  bind 10.0.0.1:443 ssl crt /pat/to/cert.pem no-sslv3

In SSL forward mode


In this mode, HAProxy forwards the SSL traffic to the server without deciphering it.
We must setup an ACL to match the SSL protocol version, then we can refuse the connection. This must be added in a **frontend** section:

  bind 10.0.0.1:443
  tcp-request inspect-delay 2s
  acl sslv3 req.ssl_ver 3
  tcp-request content reject if sslv3

Communicate a message to users

Denying sslv3 is a good way, but a better one would to educate as well users who are using this protocol.
The configuration below shows how to redirect a user to a specific page when they want to use your application over an SSLv3 connection. Of course, HAProxy must allow itself SSLv3:

frontend ft_www
  bind 10.0.0.1:443 ssl crt /pat/to/cert.pem
  acl sslv3 ssl_fc_protocol SSLv3
# first rule after all your 'http-request deny' and
# before all the redirect, rewrite, etc....
  http-request allow if sslv3
[...]
# first content switching rule
  use_backend bk_sslv3 if sslv3

backend bk_sslv3
  mode http
  errorfile 503 /etc/haproxy/pages/sslv3.http

And the content of the file /etc/haproxy/pages/sslv3.http:

HTTP/1.0 200 OK
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<head>
<title>SSLv3 spotted</title>
</head>
<body><h1>SSLv3 spotted</h1></body>
SSLv3 forbidden for your safety:<BR>
http://googleonlinesecurity.blogspot.fr/2014/10/this-poodle-bites-exploiting-ssl-30.html<BR>
<BR>
If you want to browse this website, you should upgrade your browser.
</html>

Links

Mitigating the shellshock vulnerability with HAProxy

Bash Shellshock vulnerability (CVE-2014-6271 and CVE-2014-7169)

Last week, a vulnerability in bash has been discovered. It is possible, under some circumstances, to inject code into a bash shell script.
It could be very dangerous if bash is used to process request sent remotely.
For now, you’re safe if no bash scripts are called by services with remote accesses.

Some reading about bash shellshock vulnerability:
  * https://access.redhat.com/announcements/1210053
  * https://access.redhat.com/articles/1212303

Today’s article is going to explain how to use HAProxy to protect your application from bash shellshock vulnerability if you’re in the case where you have to be protected.

Diagram

The diagram is pretty simple. Our purpose will to detect any purposely built requests and to prevent them to reach the server:

+-------------------------------------------------+
|                                                 |
| +----------+     +---------+     +------------+ |
| |          |     |         |     |            | |
| | Attacker | +-> | HAProxy | +-> | Vulnerable | |
| |          |     |         |     | server     | |
| +----------+     +---------+     |            | |
|                                  +------------+ |
|                                                 |
+-------------------------------------------------+

Configuration

Place the configuration sniplet into your HAProxy frontend configuration:

  reqdeny  ^[^:]+:s*(s*)s+{
  reqdeny  ^[^:]+:s+.*?(<<[^<;]+){5,}

Of course, your frontend must be in http mode and HAProxy must have been compiled with USE_PCRE option.

HAProxy will return a 403 if a request matches the shellshock attack.

Note: greeting to Thomas for providing the tip on HAProxy’s mailing list

Links

How to protect application cookies while offloading SSL

SSL offloading

SSL offloading or acceleration is often seen as a huge benefit for applications. People usually forget that it may have impacts on the application itself. Some times ago, I wrote a blog article which lists these impacts and propose some solutions, using HAProxy.

One thing I forgot to mention at that time was Cookies.
You don’t want your clients to send their cookies (understand their identity) in clear through the Internet.
This is today’s article purpose.

Actually, there is a cookie attribute called Secure which can be emit by a server. When this attribute is set, the client SHOULD not send the cookie over a clear HTTP connection.

SSL offloading Diagram


Simple SSL offloading diagram:

|--------|              |---------|           |--------|
| client |  ==HTTPS==>  | HAProxy | --HTTP--> | Server |
|--------|              |---------|           |--------|

The client uses HTTPs to get connected on HAProxy, HAProxy gets connected to the application server through HTTP.

Even if HAProxy can forward client connection mode information to the application server, the application server may not protect its cookie…
Fortunately, we can use HAProxy for this purpose.

Howto make HAProxy to protect application cookie when SSL offloading is enabled

That’s the question.

The response is as simple as the configuration below:

acl https          ssl_fc
acl secured_cookie res.hdr(Set-Cookie),lower -m sub secure
rspirep ^(set-cookie:.*) \1;\ Secure if https !secured_cookie

The configuration above sets up the Secure attribute if it has not been setup by the application server while the client was browsing the application over a ciphered connection.

Related Links

Links