Category Archives: layer4

Packetshield: quand votre load-balancer vous protège contre les DDOS!

Les attaques par DDOS


Il y a quelques temps, nous avions publié sur ce blog, un article expliquant comment utiliser un load-balancer pour se protéger contre les attaques de type DDOS applicatif: http://blog.haproxy.com/2012/02/27/use-a-load-balancer-as-a-first-row-of-defense-against-ddos/.

Malheureusement, les attaques sont aujourd’hui sur plusieurs vecteurs, notamment réseau et applicatif.
Leur but est toujours le même: trouver un point faible dans l’architecture et le saturer:

  1. un firewall, nombre de sessions ou paquets par secondes
  2. un load-balancer, nombre sessions établies
  3. un serveur applicatif, nombre de requêtes par secondes
  4. etc…

Dans l’article précédent, nous avions vu comment protéger les serveurs d’applications en utilisant le load-balancer, ce qui correspond à la partie applicative de l’architecture.

Pour protéger la tête de l’architecture, le firewall, il faut en général investir beaucoup d’argent dans des équipements coûteux.
Par ailleurs, dans certains cas, lorsque l’ALOHA était connecté en direct sur Internet, il ne savait se protéger que modérément contre les attaques “réseau”.

C’est pourquoi, HAProxy Technologies a développé sa propre solution de protection contre les DDOS de niveau réseau. Cet outil s’appelle Packetshield.

Les attaques de niveau réseau sont assez simples à mettre en oeuvre pour l’attaquant, mais peuvent être dévastatrices pour la cible: lorsqu’un équipement est saturé, il bloque tout le trafic.
Un équipement peut se saturer de différentes manières:

  • un trop grand nombre de paquets par secondes
  • un trop grand nombre se sessions ouvertes (TCP et/ou UDP)
  • un trop gros volume de trafic entrant

(liste non exhaustive)

NOTE: pour ce dernier cas, il faut passer par un service externe de “nettoyage” du flux, qui renverra ensuite un trafic légitime vers votre architecture.

Packetshield: Protection contre les DDOS réseau

L’ALOHA, équipé de packetshield, se place en point d’entrée de l’infrastructure.
Packetshield va agir comme un Firewall stateful en analysant les flux réseaux le traversant. Seuls le trafic considéré comme légitime sera autorisé à passer.
Différents mécanismes rentrent en jeu lors de la protection:

  • analyse des paquets: les paquets non valides sont automatiquement bloqués
  • filtrage de flux: l’administrateur peut définir ce qui est autorisé ou pas
  • analyse des flux autorisés: Packetshield fait le trie pour savoir ce qui est légitime ou pas

Contrairement aux solutions existantes qui fonctionnent dans un mode stateless et qui génèrent beaucoup de faux positifs, Packetshield garantie qu’aucune session valide ne sera bloquée.

Disponibilité de Packetshield

Packetshield est disponible dans l’ALOHA en version appliance physique depuis le firmware 7.0.
La protection de l’interface gigabit (Max 1 million de paquets par secondes) est inclu gratuitement dans l’ALOHA.
Nos clients existants n’ont qu’à mettre à jour le firmware de leur appliance pour en bénéficier. Tout nouveau client achetant l’une de nos appliances bénéficiera lui aussi de cette protection, sans surcoût.

Packetshield est aussi disponible en version 10G (14 millions de paquets traités par secondes) sur nos toutes appliances dernier modèle ALB5100, peut importe la licence choisie, du 8K au 64K.

Mode de déploiement


De part son fonctionnement stateful, Packetshield est utilisé de manière optimal dans les déploiement suivant:

Packetshield peut fonctionner en mode stateless, sans ses fonctionnalités avancé. Dans ce mode, Packetshield est compatible avec les type de déploiement suivants:

Pour plus d’information

La documentation sur la configuration de Packetshield est disponible à cette adresse: http://haproxy.com/doc/aloha/7.0/packet_flood_protection/.

Pour plus d’information, un petit mail à contact (at) haproxy (point) com.

Liens

Client IP persistence OR source IP hash load-balancing?

Client or Source IP ???


Well, this is roughly the same! Depends on people, environment, products, etc… I may use both of them in this article, but be aware that both of them points to the IP that is being used to get connected on the service whose being load-balanced.

Load-Balancing and Stickiness


Load-Balancing is the ability to spread requests among a server pool which deliver the same service. By definition, it means that any request can be sent to any server in the pool.
Some applications require stickiness between a client and a server: it means all the requests from a client must be sent to the same server. Otherwise, the application session may be broken and that may have a negative impact on the client.

Source IP stickiness


We may have many ways to stick a user to a server, which has already been discussed on this blog (Read load balancing, affinity, persistence, sticky sessions: what you need to know) (and many other article may follow).

That said, sometimes, the only information we can “rely” on to perform stickiness is the client (or source) IP address.
Note this is not optimal because:
  * many clients can be “hidden” behind a single IP address (Firewall, proxy, etc…)
  * a client can change its IP address during the session
  * a client can use multiple IP addresses
  * etc…

Performing source IP affinity


There are two ways of performing source IP affinity:
  1. Using a dedicated load-balancing algorithm: a hash on the source IP
  2. Using a stick table in memory (and a roundrobin load-balancing algorithm)

Actually, the main purpose of this article was to introduce both methods which are quite often misunderstood, and to show pros and cons of each, so people can make the right decision when configuring their Load-Balancer.

Source IP hash load-balancing algorithm


This algorithm is deterministic. It means that if no elements involved in the hash computation, then the result will be the same. 2 equipment are able to apply the same hash, hence load-balance the same way, making load-balancer failover transparent.
A hash function is applied on the source IP address of the incoming request. The hash must take into account the number of servers and each server’s weight.
The following events can make the hash change and so may redirect traffic differently over the time:
  * a server in the pool goes down
  * a server in the pool goes up
  * a server weight change

The main issue with source IP hash loadbalancing algorithm, is that each change can redirect EVERYBODY to a different server!!!
That’s why, some good load-balancers have implemented a consistent hashing method which ensure that if a server fails, for example, only the client connected to this server are redirected.
The counterpart of consistent hashing is that it doesn’t provide a perfect hash, and so, in a farm of 4 servers, some may receive more clients than others.
Note that when a failed server comes back, its “sticked” users (determined by the hash) will be redirected to it.
There is no overhead in term of CPU or memory when using such algorithm.

Configuration example in HAProxy or in the ALOHA Load-Balancer:

balance source
hash-type consistent

Source IP persistence using a stick-table


A table in memory is created by the Load-balancer to store the source IP address and the affected server from the pool.
We can rely on any non-deterministic load-balancing algorithm, such as roundrobin or leastconn (it usually depends on the type of application you’re load-balancing).
Once a client is sticked to a server, he’s sticked until the entry in the table expires OR the server fails.
There is an overhead in memory, to store stickiness information. In HAProxy, the overhead is pretty low: 40MB for 1.000.000 IPv4 addresses.
One main advantage of using a stick table is that when a failed server comes back, no existing sessions will be redirected to it. Only new incoming IPs can reach it. So no impact on users.
It is also possible to synchronize tables in memory between multiple HAProxy or ALOHA Load-Balancers, making a LB failover transparent.

Configuration example in HAProxy or in the ALOHA Load-Balancer:

stick-table type ip size 1m expire 1h
stick on src

Links

Microsoft Exchange 2013 architectures

Introduction to Microsoft Exchange 2013

There are 2 types of server in Exchange 2013:
  * Mailbox server
  * Client Access server

Definitions from Microsoft Technet website:
  * The Client Access server provides authentication, limited redirection, and proxy services, and offers all the usual client access protocols: HTTP, POP and IMAP, and SMTP. The Client Access server, a thin and stateless server, doesn’t do any data rendering. There’s never anything queued or stored on the Client Access server.
  * The Mailbox server includes all the traditional server components found in Exchange 2010: the Client Access protocols, the Transport service, the Mailbox databases, and Unified Messaging (the Client Access server redirects SIP traffic generated from incoming calls to the Mailbox server). The Mailbox server handles all activity for the active mailboxes on that server.

High availability in Exchange 2013


Mailbox server high availability is achieved by the creation of a DAG: Database Availability Group. All the redundancy is achieved within the DAG and the Client Access server will retrieve the user session through the DAG automatically when a fail over occurs.

Client Access servers must be Load-Balanced to achieve high-availability. There are many ways to load-balance them:
  * DNS round-robin (Com’on, we’re in 2013, stop mentioning this option !!!!)
  * NLB (Microsoft don’t recommend it)
  * Load-Balancers (also known as Hardware Load-Balancer or HLB): this solution is recommended since it can bring smart load-balancing and advanced health checking.

Performance in Exchange 2013


The key performance in Exchange 2013 is obviously the Mailbox server. If this one slows down, all the users are affected. So be careful when designing your Exchange platform and try to create multiple DAGs with one single master per DAG. If one Mailbox server fails or is in maintenance, then you’ll be in degraded mode (which does not mean degraded performance) where one Mailbox server would be master on 2 DAGs. The disks IO may also be important here 😉

Concerning the Client Access server, it’s easy to disable temporarily one of them if it’s slowing down and route connections to the other Client Access server(s).

Exchange 2013 Architecture


Your architecture should meet your requirements:
  * if you don’t need high-availability and performance, then a single server with both Client Access and Mailbox role is sufficient
  * if you need high-availability and have a moderate number of users, a couple of servers, each one hosting both Client Access and Mailbox role is enough. A DAG to ensure high-availability of the Mailbox servers and a Load-Balancer to ensure high-availability of the Client Access servers.
  * if you need high-availability and performance for a huge amount of connections, then multiple Mailbox server, multiple DAGs, multiple Client Access servers and a couple of Load-Balancers.

Note that the path from the first architecture to the third one is doable with almost no effort, thanks to Exchange 2013.

Client Access server Load-Balancing in Exchange 2013


Using a Load-Balancer to balance user connections to the Client Access servers allows multiple type of architectures. The current article mainly focus on these architectures: we do provide load-balancers 😉
If you need help from Exchange architects, we have partners who can help you with the installation, setup and tuning of the whole architecture.

Load-Balancing Client Access servers in Exchange 2013

First of all, bear in mind that Client Access servers in Exchange 2013 are stateless. It is very important from a load-balancing point of view because it makes the configuration much more simple and scalable.
Basically, I can see 3 main types of architectures for Client Access servers load-balancing in Exchange 2013:
  1. Reverse-proxy mode (also known as source NAT)
  2. Layer 4 NAT mode (desitnation NAT)
  3. Layer 4 DSR mode (Direct Server Return, also known as Gateway)

Each type of architecture has pros and cons and may have impact on your global infrastructure. In the next chapter, I’ll provide details for each of them.

To illustrate the article, I’ll use a simple Exchange 2013 configuration:
  * 2 Exchange 2013 servers with hosting both Client Access and Mailbox role
  * 1 ALOHA Load-Balancer
  * 1 client (laptop)
  * all the Exchange services hosted on a single hostname (IE mail.domain.com) (could work as well with an hostname per service)

Reverse-proxy mode (also known as source NAT)


Diagram
The diagram below shows how things work with a Load-balancer in Reverse-proxy:
  1. the client establishes a connection onto the Load-Balancer
  2. the Load-Balancer chooses a Client Access server and establishes the connection on it
  3. client and CAS server discuss through the Load-Balancer
Basically, the Load-Balancer breaks the TCP connection between the client and the server: there are 2 TCP connections established.

exchange_2013_reverse_proxy

Advantages of Reverse-proxy mode
  * not intrusive: no infrastructure changes neither server changes
  * ability to perform SSL offloading and layer 7 persistence
  * client, Client Access servers and load-balancers can be located anywhere in the infrastructure (same subnet or different subnet)
  * can be used in a DMZ
  * A single interface is required on the Load-Balancer

Limitations of Reverse-proxy mode
  * the servers don’t see the client IPs (only the LB one)
  * maximum 65K connections on the Client Access farm (without tricking the configuration)

Configuration
To perform this type of configuration, you can use the LB Admin tab and do it through the GUI or just copy/paste the following lines into the LB Layer 7 tab:

######## Default values for all entries till next defaults section
defaults
  option  dontlognull             # Do not log connections with no requests
  option  redispatch              # Try another server in case of connection failure
  option  contstats               # Enable continuous traffic statistics updates
  retries 3                       # Try to connect up to 3 times in case of failure 
  backlog 10000                   # Size of SYN backlog queue
  mode tcp                        #alctl: protocol analyser
  log global                      #alctl: log activation
  option tcplog                   #alctl: log format
  timeout client  300s            #alctl: client inactivity timeout
  timeout server  300s            #alctl: server inactivity timeout
  timeout connect   5s            # 5 seconds max to connect or to stay in queue
  default-server inter 3s rise 2 fall 3   #alctl: default check parameters

frontend ft_exchange
  bind 10.0.0.9:443 name https    #alctl: listener https configuration.
  maxconn 10000                   #alctl: max connections (dep. on ALOHA capacity)
  default_backend bk_exchange     #alctl: default farm to use

backend bk_exchange
  balance roundrobin              #alctl: load balancing algorithm
  server exchange1 10.0.0.13:443 check
  server exchange2 10.0.0.14:443 check

(update IPs for your infrastructure and update the DNS name with the IP configured on the frontend bind line.)

Layer 4 NAT mode (desitnation NAT)


Diagram
The diagram below shows how things work with a Load-balancer in Layer 4 NAT mode:
  1. the client establishes a connection to the Client Access server through the Load-Balancer
  2. client and CAS server discuss through the Load-Balancer
Basically, the Load-Balancer acts as a router between the client and the server: a single TCP connection is established directly between the client and the Client Access server, and the LB just forward packets between both of them.
exchange_2013_layer4_nat

Advantages of Layer 4 NAT mode
  * no connection limit
  * Client Access servers can see the client IP address

Limitations of Layer 4 NAT mode
  * infrastructure intrusive: the server default gateway must be the load-balancer
  * clients and Client Access servers can’t be in the same subnet
  * no SSL acceleration, no advanced persistence
  * Must use 2 interfaces on the Load-Balancer (or VLAN interface)

Configuration
To perform this type of configuration, you can use the LB Admin tab and do it through the GUI or just copy/paste the following lines into the LB Layer 4 tab:

director exchange 10.0.1.9:443 TCP
  balance roundrobin                               #alctl: load balancing algorythm
  mode nat                                         #alctl: forwarding mode
  check interval 10 timeout 2                      #alctl: check parameters
  option tcpcheck                                  #alctl: adv check parameters
  server exchange1 10.0.0.13:443 weight 10 check   #alctl: server exchange1
  server exchange2 10.0.0.14:443 weight 10 check   #alctl: server exchange2

(update IPs for your infrastructure and update the DNS name with the IP configured on the frontend bind line.)

Layer 4 DSR mode (Direct Server Return, also known as Gateway)


Diagram
The diagram below shows how things work with a Load-balancer in Layer 4 DSR mode:
  1. the client establishes a connection to the Client Access server through the Load-Balancer
  2. the Client Access server acknowledges the connection directly to the client, bypassing the Load-Balancer on the way back. The Client Access server must have the Load-Balancer Virtual IP configured on a loopback interface.
  3. client and CAS server keep on discussing the same way: request through the load-balancer and responses directly from server to client.
Basically, the Load-Balancer acts as a router between the client and the server: a single TCP connection is established directly between the client and the Client Access server.
The Client and the server can be in the same subnet, like in the diagram, or can be in two different subnet. In the second case, the server would use its default gateway (no need to forward the traffic back through the load-balancer).
exchange_2013_layer4_dsr

Advantages of Layer 4 DSR mode
  * no connection limit
  * Client Access servers can see the client IP address
  * clients and Client Access servers can be in the same subnet
  * A single interface is required on the Load-Balancer

Limitations of Layer 4 DSR mode
  * infrastructure intrusive: the Load-Balancer Virtual IP must be configured on the Client Access server (Loopback).
  * no SSL acceleration, no advanced persistence

Configuration
To perform this type of configuration, you can use the LB Admin tab and do it though the web form or just copy/paste the following lines into the LB Layer 4 tab:

director exchange 10.0.0.9:443 TCP
  balance roundrobin                               #alctl: load balancing algorythm
  mode gateway                                     #alctl: forwarding mode
  check interval 10 timeout 2                      #alctl: check parameters
  option tcpcheck                                  #alctl: adv check parameters
  server exchange1 10.0.0.13:443 weight 10 check   #alctl: server exchange1
  server exchange2 10.0.0.14:443 weight 10 check   #alctl: server exchange2

Related links

Links

Layer 4 load balancing tunnel mode

The tunnel mode looks like the Direct Server Return mode, except that traffic between the load-balancer and the server can be routed.

The load-balancer encapsulates the request in an IP tunnel to the server.
The server recover the client request from the loadbalancer, process it and forward the response directly to the client.

TCP connection overview

layer4_tunnel_tcp_connection
The loadbalancer takes client requests then encapsulate them into an IP tunnel to forward them to the server.

Data flow

layer4_tunnel_data_flow
The client traffic between the server and the load-balancer is tunneled and can be routed between both of them.
The server will answer directly to the client.

Pros and cons

Pros

  • backends from multiple datacenters can be used
  • load-balancer network bandwith is not a bottleneck anymore
  • total output bandwith is the sum of each backend bandwith

Cons

  • requires patched backend to be able to tunnel IP traffic
  • no layer 7 advanced features are available

When use this architecture?

  • when the only way to reach backends is routing.
  • where no intelligence is required
  • when output capacity of the load-balancer could be the bottleneck

Links

layer 4 load balancing Direct Server Return mode

Direct server return is usually shortened to DSR.

In DSR mode, the load-balancer routes packets to the backends without changing anything in it but the destination MAC address.
The backends process the requests and answer directly to the clients, without passing through the load-balancer.

The backends must have the service IP configured on a loopback to be able to accept the requests.

TCP connection overview

layer4_dsr_tcp_connection
As usual when performing layer 4 load-balancing, the TCP connection is established directly between the client and the backend.
Note that the the requests pass through the load-balancer while the responses not.

Data flow

layer4_dsr_data_flow
As explained above, the load-balancer sees only the requests and just change the destination MAC address of the packets.
The backends answers directly to the client using the service IP configured on a loopback interface.

Pros and cons

Pros

  • very fast load-balancing mode
  • load-balancer network bandwith is not a bottleneck anymore
  • total output bandwith is the sum of each backend bandwith
  • less intrusive than the layer 4 load-balancing NAT mode

Cons

  • the service VIP must be configured on a loopback interface on each backend and must not answer to ARP requests
  • no layer 7 advanced features are available

When use this architecture?

  • where response time matters
  • where no intelligence is required
  • when output capacity of the load-balancer could be the bottleneck

Links

Layer 4 load balancing NAT mode

NAT stands for Network Address Translation.

In the NAT mode, the load-balancer will route traffic between user and server by changing destination IP address of the packets.

TCP connection overview

TCP connection is established between the client and the server.
The loadbalancer just ensures a client is always forwarded to the same server.
layer4_nat_tcp_connection

Data flow

As shown below, the clients get connected to the service VIP.
The load balancer chooses a server in the pool then forwards packets to it by changing destination IP address.
layer4_nat_data_flow

Pros and cons

Pros

  • fast load balancing
  • easy to deploy

Cons

  • infrastructure intrusive: need to change the default gateway of the servers
  • The server default gateway must use the load balancer, in order to do reverse NAT operation.
  • output bandwith is limitated by loadbalancer output capacity

When use this architecture?

  • where response time matters
  • where no intelligence is required
  • when output capacity of the load-balancer won’t be a bottleneck in a near future
  • when nothing but the default gateway of the servers can be changed

Links