Category Archives: Aloha

ALOHA Pocket is coming…

Well, this project is not really a secret anymore and people start to ask about it, so let me present the beast :

front_smThis is the ALOHA Pocket. Probably the smallest load balancer you have ever seen from any vendor. It is a full-featured ALOHA with layer 4/7, SSL, VRRP, the complete web interface with templates, the logs… It consumes less than a watt (0.75W to be precise) and is powered over USB.  It can run for about ten hours from a single 2200mAh battery. Still it achieves more than a thousand connections per second and forward 70 Mbps between the two ports. Yes, this is more than what some applications we’ve seen in field deliver on huge servers consuming 1000 times this power and running with 4000 times its amount of RAM. This is made possible thanks to our highly optimized, lightweight products which are so energy efficient and need so little resource that they can run on almost anything (and of course, they are magnified when running on powerful hardware).

Obviously nobody wants to run their production on this, it would not look serious! But we found that this is the ideal format to bring your machine everywhere, for demos, for tests, to develop in the train, or even just to tease friends. And it’s so cool that I have several of them  on my desk and others in my bag and am using them all the day for various tests. And while using it I found that it was so much more convenient to use than a VM when explaining high availability to someone that we realized that it’s the format of choice for students discovering load balancing and high availability. Another nice thing is that since it has two ports, it’s perfect for plugging between your PC and the LAN to observe the HTTP communications between your browser and the application you’re developing.

So we decided to prepare one hundred of them that we’ll offer to students and interns working on a load balancing project, in exchange for their promise to blog about their project’s progress.  If they need we can even send them a cluster of two.  And who knows, maybe among these, someone will have a great idea and develop a worldwide successful project, and then we’ll be very proud to have provided the initial spark that made this possible. And if it helps students get a career around load balancing, we’ll be quite proud to transmit this passion as well!

We still have a few things to complete before it can go wild, such as a bit of documentation to explain how to start with it. But if you think you’re going to work on a load balancing project or are joining a company as an intern and will be doing some stuff with web servers, this can be the perfect way to discover this new amazing world to design solutions which resist to real failures caused by pulling off a cable and not just the clean “power down” button pressed in a VM. Start thinking about it to reserve one (or a pair) when we launch it in the upcoming weeks. Conversely if you absolutely want one, you just have to find a load balancing project to work on 🙂

In any case, don’t wait too much to think about your project, because the stock is limited, so if there is too much demand, we’ll have to selective on the projects we’re going to support for  the last ones.

Stay tuned!

Packetshield: quand votre load-balancer vous protège contre les DDOS!

Les attaques par DDOS


Il y a quelques temps, nous avions publié sur ce blog, un article expliquant comment utiliser un load-balancer pour se protéger contre les attaques de type DDOS applicatif: http://blog.haproxy.com/2012/02/27/use-a-load-balancer-as-a-first-row-of-defense-against-ddos/.

Malheureusement, les attaques sont aujourd’hui sur plusieurs vecteurs, notamment réseau et applicatif.
Leur but est toujours le même: trouver un point faible dans l’architecture et le saturer:

  1. un firewall, nombre de sessions ou paquets par secondes
  2. un load-balancer, nombre sessions établies
  3. un serveur applicatif, nombre de requêtes par secondes
  4. etc…

Dans l’article précédent, nous avions vu comment protéger les serveurs d’applications en utilisant le load-balancer, ce qui correspond à la partie applicative de l’architecture.

Pour protéger la tête de l’architecture, le firewall, il faut en général investir beaucoup d’argent dans des équipements coûteux.
Par ailleurs, dans certains cas, lorsque l’ALOHA était connecté en direct sur Internet, il ne savait se protéger que modérément contre les attaques “réseau”.

C’est pourquoi, HAProxy Technologies a développé sa propre solution de protection contre les DDOS de niveau réseau. Cet outil s’appelle Packetshield.

Les attaques de niveau réseau sont assez simples à mettre en oeuvre pour l’attaquant, mais peuvent être dévastatrices pour la cible: lorsqu’un équipement est saturé, il bloque tout le trafic.
Un équipement peut se saturer de différentes manières:

  • un trop grand nombre de paquets par secondes
  • un trop grand nombre se sessions ouvertes (TCP et/ou UDP)
  • un trop gros volume de trafic entrant

(liste non exhaustive)

NOTE: pour ce dernier cas, il faut passer par un service externe de “nettoyage” du flux, qui renverra ensuite un trafic légitime vers votre architecture.

Packetshield: Protection contre les DDOS réseau

L’ALOHA, équipé de packetshield, se place en point d’entrée de l’infrastructure.
Packetshield va agir comme un Firewall stateful en analysant les flux réseaux le traversant. Seuls le trafic considéré comme légitime sera autorisé à passer.
Différents mécanismes rentrent en jeu lors de la protection:

  • analyse des paquets: les paquets non valides sont automatiquement bloqués
  • filtrage de flux: l’administrateur peut définir ce qui est autorisé ou pas
  • analyse des flux autorisés: Packetshield fait le trie pour savoir ce qui est légitime ou pas

Contrairement aux solutions existantes qui fonctionnent dans un mode stateless et qui génèrent beaucoup de faux positifs, Packetshield garantie qu’aucune session valide ne sera bloquée.

Disponibilité de Packetshield

Packetshield est disponible dans l’ALOHA en version appliance physique depuis le firmware 7.0.
La protection de l’interface gigabit (Max 1 million de paquets par secondes) est inclu gratuitement dans l’ALOHA.
Nos clients existants n’ont qu’à mettre à jour le firmware de leur appliance pour en bénéficier. Tout nouveau client achetant l’une de nos appliances bénéficiera lui aussi de cette protection, sans surcoût.

Packetshield est aussi disponible en version 10G (14 millions de paquets traités par secondes) sur nos toutes appliances dernier modèle ALB5100, peut importe la licence choisie, du 8K au 64K.

Mode de déploiement


De part son fonctionnement stateful, Packetshield est utilisé de manière optimal dans les déploiement suivant:

Packetshield peut fonctionner en mode stateless, sans ses fonctionnalités avancé. Dans ce mode, Packetshield est compatible avec les type de déploiement suivants:

Pour plus d’information

La documentation sur la configuration de Packetshield est disponible à cette adresse: http://haproxy.com/doc/aloha/7.0/packet_flood_protection/.

Pour plus d’information, un petit mail à contact (at) haproxy (point) com.

Liens

Microsoft Remote Desktop Services (RDS) Load-Balancing

Microsoft Remote Desktop services (RDS)

Remote Desktop Services, formerly Terminal Services, is a technology from Microsoft that allows users to access remotely to a session-based desktop, virtual machine-based desktop or applications hosted in a datacenter from their corporate network or from the internet.

Multiple RDS servers can be used in a farm. Hence we need to balance the load against them.
To achieve this purpose, we have different ways:
* using a connection broker
* using a load-balancer with the connection broker
* using a load-balancer without the connection broker

Of course, our load-balancer of choice is HAProxy!
In this blog article, we’re going to focus only on the case where a load-balancer is used.

The main issue when load-balancing multiple Remote Desktop Services servers is to ensure a user the continuity of his session in case of a network outage.

Current article will focus on session high availability for an optimal end user experience.

HAProxy with a connection broker

The connection broker, formerly Session broker, main purpose is to reconnect a user to his existing session. Since Windows 2008, the connection broker also have some load-balancing mechanism.

So, why using a load-balancer if the connection broker can do load-balance?


Answer is simple: security. Since HAProxy is a Reverse-Proxy, it breaks the TCP connection between the client and the server. HAProxy can be deployed in DMZ to give access to users coming from internet to a RDS farm deployed in the VLAN dedicated to servers.

HAProxy configuration


Note: this configuration works for the ALOHA 6.0 and above and HAPEE (HAProxy Enterprise Edition) 1.5 and above.

frontend ft_rdp
  mode tcp
  bind 192.168.13.128:3389 name rdp
  timeout client 1h
  log global
  option tcplog
  tcp-request inspect-delay 2s
  tcp-request content accept if RDP_COOKIE
  default_backend bk_rdp

backend bk_rdp
  mode tcp
  balance leastconn
  persist rdp-cookie
  timeout server 1h
  timeout connect 4s
  log global
  option tcplog
  option tcp-check
  tcp-check connect port 3389 ssl
  default-server inter 3s rise 2 fall 3
  server srv01 192.168.13.13:3389 weight 10 check
  server srv02 192.168.13.14:3389 weight 10 check

HAProxy without a connection broker

HAProxy can be used on its own to perform session load-balancing and resumption. For this purpose, it needs a stick-table where the user-server association is stored.
A peers section is added to the configuration. So we can share session persistence information between a cluster of ALOHAs or HAPEE servers.

peers aloha
 peer aloha1 192.168.13.1:1023
 peer aloha2 192.168.13.2:1023

frontend ft_rdp
  mode tcp
  bind 192.168.13.128:3389 name rdp
  timeout client 1h
  log global
  option tcplog
  tcp-request inspect-delay 2s
  tcp-request content accept if RDP_COOKIE
  default_backend bk_rdp

backend bk_rdp
  mode tcp
  balance leastconn
  timeout server 1h
  timeout connect 4s
  log global
  option tcplog
  stick-table type string len 32 size 10k expire 8h peers aloha
  stick on rdp_cookie(mstshash)
  option tcp-check
  tcp-check connect port 3389 ssl
  default-server inter 3s rise 2 fall 3
  server srv01 192.168.13.13:3389 weight 10 check
  server srv02 192.168.13.14:3389 weight 10 check

To know the user-server association, we can simply read the content of the stick-table:

echo show table bk_rdp | socat /var/run/haproxy.stat -
# table: bk_rdp, type: string, size:10240, used:5
0x21c7eac: key=Administrator use=0 exp=83332288 server_id=1
0x21c7eac: key=test-001 use=0 exp=83332288 server_id=2

We can easily read the login used by the user, the expiation date (in milliseconds) and the server ID used for the session.

Links

Web application name to backend mapping in HAProxy

Synopsis

Let’s take a web application platform where many HTTP Host header points to.
Of course, this platform hosts many backends and HAProxy is used to perform content switching based on the Host header to route HTTP traffic to each backend.

HAProxy map


HAProxy 1.5 introduced a cool feature: converters. One converter type is map.
Long story made short: a map allows to map a data in input to an other one on output.

A map is stored in a flat file which is loaded by HAProxy on startup. It is composed by 2 columns, on the left the input string, on the right the output one:

in out

Basically, if you call the map above and give it the in strings, it will return out.

Mapping

Now, the interesting part of the article 🙂

As stated in introduction, we want to map hundreds of Host headers to tens of backends.

The old way of mapping: acl and use_backend rules

Before the map, we had to use acls and use_backend rules.

like below:

frontend ft_allapps
 [...]
 use_backend bk_app1 if { hdr(Host) -i app1.domain1.com app1.domain2.com }
 use_backend bk_app2 if { hdr(Host) -i app2.domain1.com app2.domain2.com }
 default_backend bk_default

Add one statement per use_backend rule.

This works nicely for a few backends and a few domain names. But this type of configuration is hardly scallable…

The new way of mapping: one map and one use_backend rule

Now we can use map to achieve the same purpose.

First, let’s create a map file called domain2backend.map, with the following content: on the left, the domain name, on the right, the backend name:

#domainname  backendname
app1.domain1.com bk_app1
app1.domain2.com bk_app1
app2.domain1.com bk_app2
app2.domain2.com bk_app2

And now, HAProxy configuration:

frontend ft_allapps
 [...]
 use_backend %[req.hdr(host),lower,map_dom(/etc/hapee-1.5/domain2backend.map,bk_default)]

Here is what HAProxy will do:

  1. req.hdr(host) ==> fetch the Host header from the HTTP request
  2. lower ==> convert the string into lowercase
  3. map_dom(/etc/hapee-1.5/domain2backend.map) ==> look for the lowercase Host header in the map and return the backend name if found. If not found, the name of a default backend is returned
  4. route traffic to the backend name returned by the map

Now, adding a new content switching rule means just add one new line in the map content (and reload HAProxy). No regexes, map data is stored in a tree, so processing time is very low compared to matching many string in many ACLs for many use_backend rules.

simple is beautiful!!!

HAProxy map content auto update


If you are an HAPEE user (and soon available for the ALOHA), you can use the lb-update content to download the content of the map automatically.
Add the following statement in your configuration:

dynamic-update
 update id domain2backend.map url https://10.0.0.1/domain2backend.map delay 60s timeout 5s retries 3 map

Links

binary health check with HAProxy 1.5: php-fpm/fastcgi probe example

Application layer health checking

Health checking is the ability to probe a server to ensure the service is up and running.
This is one of the root feature of any load-balancer.

One can probe servers and services at different layer of the OSI model:
  * ARP check (not available in HAProxy)
  * ICMP (ping) check (not available in HAProxy)
  * TCP (handshake) check
  * Application (HTTP, MySql, SMTP, POP, etc…) check

The most representative of the application status the check is, the best.

This means that the best way to check a service is to “speak” the protocol itself.
Unfortunately, it is impossible to write one check per protocol, there are too many protocols and some of them are proprietary and/or binary.
That’s why HAProxy 1.5 now embeds a new health checking method called “tcp-check“. It is very simple and basic “send/expect” probing method where HAProxy can send arbitrary strings and match string or regex comparison on server responses.
Many send and expect can be executed in a row to determine the status of a server.

I’ve already explained how to check redis server and how to balance traffic only to the redis master server of a cluster.

Today’s article introduces a binary protocol widely deployed: fastcgi used by php-fpm.

fastcgi binary ping/pong health check

fastcgi is a binary protocol. It means data on the network are not readable by humans, like HTTP or SMTP.
php-fpm relies on this protocol to treat PHP code. It is common to use HAProxy to load-balance many php-fpm servers for resiliency and scalability.

php-fpm Configuration

Enable a dedicated url to probe and setup the response in your php-fpm configuration:

ping.path = /ping
ping.response = pong

Which means HAProxy has to send a fastcgi request on /ping and expects the server to response with “pong“.

HAProxy health checking for php-fpm

Add the following tcp-check sequence in your php-fpm backend to probe the /ping url and ensure the server answers a “pong“.
The comment at the end of the line describe the php-cgi protocol fields.

option tcp-check
 # FCGI_BEGIN_REQUEST
 tcp-check send-binary   01 # version
 tcp-check send-binary   01 # FCGI_BEGIN_REQUEST
 tcp-check send-binary 0001 # request id
 tcp-check send-binary 0008 # content length
 tcp-check send-binary   00 # padding length
 tcp-check send-binary   00 #
 tcp-check send-binary 0001 # FCGI responder
 tcp-check send-binary 0000 # flags
 tcp-check send-binary 0000 #
 tcp-check send-binary 0000 #
 # FCGI_PARAMS
 tcp-check send-binary   01 # version
 tcp-check send-binary   04 # FCGI_PARAMS
 tcp-check send-binary 0001 # request id
 tcp-check send-binary 0045 # content length
 tcp-check send-binary   03 # padding length: padding for content % 8 = 0
 tcp-check send-binary   00 #
 tcp-check send-binary 0e03524551554553545f4d4554484f44474554 # REQUEST_METHOD = GET
 tcp-check send-binary 0b055343524950545f4e414d452f70696e67   # SCRIPT_NAME = /ping
 tcp-check send-binary 0f055343524950545f46494c454e414d452f70696e67 # SCRIPT_FILENAME = /ping
 tcp-check send-binary 040455534552524F4F54 # USER = ROOT
 tcp-check send-binary 000000 # padding
 # FCGI_PARAMS
 tcp-check send-binary   01 # version
 tcp-check send-binary   04 # FCGI_PARAMS
 tcp-check send-binary 0001 # request id
 tcp-check send-binary 0000 # content length
 tcp-check send-binary   00 # padding length: padding for content % 8 = 0
 tcp-check send-binary   00 #

 tcp-check expect binary 706f6e67 # pong

Note that the whole send string could be written on a single line.

Any protocol???

If some day, you have to do the same type of configuration on a protocol nobody knows, simply capture network traffic of a “hello” sequence using tcpdump.
Then send the tcp payload cpatured using the tcp-check send command and configure the appropriate expect.
And it will work!

Links

Asymmetric routing, multiple default gateways on Linux with HAProxy

Why we may need multiple default gateways?

Nowadays, Application Delivery controllers (aka Load-Balancers) become the entry point for all the applications hosted in a company or administration.
That said, many different type of population could access the applications:
  * internal users from the LAN
  * partners through MPLS or VPNs
  * external users from internet

On the other side, applications could be hosted on different VLANs in the architecture:
  * internal LAN
  * external DMZ

The diagram below shows the “big picture” of this type of architecture:
multiple_default_gateways

Routing in the Linux network stack

I’m not going to deeply explain how it works, sorry… It would deserve a complete blog post 🙂
That said, any device connected on an IP network needs an IP address to be able to talk to other devices in its LAN. It also needs a default gateway to be able to reach devices which are located outside its LAN.
A Linux kernel can use a single default gateway at a time, but thanks to the metric you can configure many default gateways.
When needed, the Linux Kernel will parse the default gateway table and will use the one with the lowest metric. By default, when no metric is configured, the kernel attributes a metric 0.
Each metric must be unique in your Kernel IP stack.

How HAProxy can help in such situation??


Users access applications through a HAProxy bind. The bind can be hosted on any IP address available or not (play with your sysctl for this purpose) on the server.
By default, the traffic comes in HAProxy through this bind and HAProxy let the kernel choose the most appropriate default gateway to forward the answer to the client. As we’ve seen above, the most appropriate default gateway from the kernel point of view is the one with the lowest metric usually 0.

That said, HAProxy is smart enough to tell the kernel which network interface to use to forward the response to the client. Just add the statement interface ethX (where X is the id of the interface you want to use) on HAProxy bind line.
With this parameter, HAProxy can force the kernel to use the default gateway associated to the network interface ethX if it exists, otherwise, the interface with the lowest metric will be used.

Security concern


From a security point of view, some security manager would say that it is absolutely unsecure to plug a device in multiple DMZ or VLANs. They are right. But usually, this type of company’s business is very important and they can affoard one load-balancer per DMZ or LAN.
That said, there is no security breach with the setup introduced here. HAProxy is a reverse-proxy and so you don’t need to allow ip_forward between all interfaces for this solution to work.
I mean that nobody could use the loadbalancer as a default gateway to reach an other subnet bypassing the firewall…
Then only traffic allowed to pass through is the one load-balanced!

Configuration

The configuration below applies to the ALOHA Loadbalancer. Just update the content to match your Linux distribution configuration syntax.
The configuration is also related to the diagram above.

Network configuration


In your ALOHA, go in the Services tab, then edit the Network configuration.
To keep it simple, I’m not going to add any VRRP configuration.

service network eth0
    ########## eth0.
    auto on
    ip   address 10.0.0.2/24
    ip   route   default 10.0.0.1

service network eth1
    ########## eth1.
    auto on
    ip   address 10.0.1.2/24
    ip   route   default 10.0.1.1 metric 1

service network eth2
    ########## eth2.
    auto on
    ip   address 10.0.2.2/24
    ip   route   default 10.0.2.1 metric 2

service network eth3
    ########## eth3.
    auto on
    ip   address 10.0.3.2/24
    ip   route   default 10.0.3.1 metric 3

service network eth4
    ########## eth4.
    auto on
    ip   address 10.0.4.2/24
    ip   route   default 10.0.4.1 metric 4

The routing table from the ALOHA looks like:

default via 10.0.0.1 dev eth0
default via 10.0.1.1 dev eth1  metric 1
default via 10.0.2.1 dev eth2  metric 2
default via 10.0.3.1 dev eth3  metric 3
default via 10.0.4.1 dev eth4  metric 4

HAProxy configuration for Corporate website or ADFS proxies


These services are used by internet users only.

frontend ft_www
 bind 10.0.0.2:80
[...]

no need to specify any interface here, since the traffic comes from internet, HAProxy can let the kernel to use the default gateway which points in that direction (here eth0).

HAProxy configuration for Exchange 2010 or 2013


This service is used by both internal and internet users.

frontend ft_exchange
 bind 10.0.0.3:443
 bind 10.0.2.3:443 interface eth2
[...]

The responses to internet users will go through eth0 while the one for internal LAN users will use the default gateway configured on eth2 10.0.2.1.

HAProxy configuration for Sharepoint 2010 or 2013


This service is used by MPLS/VPN users and internal users.

frontend ft_exchange
 bind 10.0.1.4:443 interface eth1
 bind 10.0.2.4:443 interface eth2
[...]

The responses to MPLS/VPN users will go through eth1 default gateway 10.0.1.1 while the one for internal LAN users will use the default gateway configured on eth2 10.0.2.1.

Links

Emulating Active/passing application clustering with HAProxy

Synopsis

HAProxy is a Load-Balancer, this is a fact. It is used to route traffic to servers to primarily ensure applications reliability.

Most of the time, the sessions are locally stored in a server. Which means that if you want to split client traffic on multiple servers, you have to ensure each user can be redirected to the server which manages his session (if the server is available, of course). HAProxy can do this in many ways: we call it persistence.
Thanks to persistence, we usually says that any application can be load-balanced… Which is true in 99% of the cases. In very rare cases, the application can’t be load-balanced. I mean that there might be a lock somewhere in the code or for some other good reasons…

In such case, to ensure high-availability, we build “active/passive” clusters, where a node can be active at a time.
HAProxy can be use in different ways to emulate an active/passive clustering mode, and this is the purpose of today’s article.

Bear in mind that by “active/passive”, I mean that 100% of the users must be forwarded to the same server. And if a fail over occurs, they must follow it in the mean time!

Diagram

Let’s use one HAProxy with a couple of servers, s1 and s2.
When starting up, s1 is master and s2 is used as backup:

  -------------
  |  HAProxy  |
  -------------
   |         `
   |active    ` backup
   |           `
 ------       ------
 | s1 |       | s2 |
 ------       ------

Configuration

Automatic failover and failback

The configuration below makes HAProxy to use s1 when available, otherwise fail over to s2 if available:

defaults
 mode http
 option http-server-close
 timeout client 20s
 timeout server 20s
 timeout connect 4s

frontend ft_app
 bind 10.0.0.100:80 name app
 default_backend bk_app

backend bk_app
 server s1 10.0.0.1:80 check
 server s2 10.0.0.2:80 check backup

The most important keyword above is “backup” on s2 configuration line.
Unfortunately, as soon as s1 comes back, then all the traffic will fail back to it again, which can be acceptable for web applications, but not for active/passive

Automatic failover without failback

The configuration below makes HAProxy to use s1 when available, otherwise fail over to s2 if available.
When a failover has occured, no failback will be processed automatically, thanks to the stick table:

peers LB
 peer LB1 10.0.0.98:1234
 peer LB2 10.0.0.99:1234

defaults
 mode http
 option http-server-close
 timeout client 20s
 timeout server 20s
 timeout connect 4s

frontend ft_app
 bind 10.0.0.100:80 name app
 default_backend bk_app

backend bk_app
 stick-table type ip size 1 nopurge peers LB
 stick on dst
 server s1 10.0.0.1:80 check
 server s2 10.0.0.2:80 check backup

The stick table will maintain persistence based on destination IP address (10.0.0.100 in this case):

show table bk_app
# table: bk_app, type: ip, size:20480, used:1
0x869154: key=10.0.0.100 use=0 exp=0 server_id=1

With such configuration, you can trigger a fail back by disabling s2 during a few second period.

Links