Category Archives: HAProxy

Use GeoIP database within HAProxy

Introduction

Sometimes we need to know the country of the user using the application, for different purposes:

  • Automatically select the most appropriate language
  • Send a 302 to redirect the user to the closest POP from his location
  • Allow only a single country to browse the site, for legal reason
  • Block some countries we don’t do business with and which are the source of most web attacks

IP based location


To achieve this purpose, the most “reliable” information we have from a user is his IP address.
Well, not as reliable as we could hope:

  • not “reliable” because it’s easy to use a proxy installed in a foreign country to fake your IP address…
  • not “reliable” because GeoIP databases are not accurate.
  • not “reliable” because GeoIP databases rely on information provided by the ISP.
  • not “reliable” because any subnets can be routed from anywhere on earth.

When an ISP does a request for a new subnet to its local RIR, it has to tell the country where it will be used.
This country is supplied as a code composed by two letters, normalized by ISO 3166.

You can use whois tool can be used to know the country code of a IP address:

whois 1.1.1.1
[...]
country:        AU
[...]

geolocation definition

Well, it’s quite easy to understand: Geolocation is the process to link a third party to a geographicl location. In easier words: know the country of a client IP address.
On Internet, such base is called GeoIP.

geolocation databases


There are a few GeoIP databases available on the Internet, most of them uses IP ranges to link an IP address to its country code.
An IP range is simply a couple of IP addresses representing the first and the last IP address of a range.
NOTE: It might correspond to a real subnet, but in most cases, it doesn’t ;).

In example:

"1.1.2.0","1.1.63.255","16843264","16859135","CN","China"

What’s the issue with HAProxy then???


HAProxy can only use CIDR notation, with real subnets.
It means we’ll have to turn the IP ranges into CIDR notation. This is not a easy task since you must split the ip range in multiple subnets…
Once done, we’ll be able to configure HAProxy to use them in ACLs and do anything an ACL can do.

For example, the range above should be translated to the following subnets:

1.1.2.0/23 "CN"
1.1.4.0/22 "CN"
1.1.8.0/21 "CN"
1.1.16.0/20 "CN"
1.1.32.0/19 "CN"

Now, you can understand why GeoIP databases uses IP ranges: it takes fewer lines ūüôā

iprange tool

To ease this job, Willy released a tool called iprange in HAProxy sources contrib directory.
You can see it here, in HAProxy’s git: http://www.haproxy.org/git/?p=haproxy.git;a=tree;f=contrib/iprange
It can be used to extract CIDR subnets from an IP range.

iprange installation


Just downlaod both Makefile and iprange.c then run make:

make
gcc -s -O3 -o iprange iprange.c

too complicated ūüôā

iprange usage


iprange take a single incoming format, composed by 3 columns separated by commas:

  1. first IP
  2. Last IP
  3. Country code

For example:

"1.1.2.0","1.1.63.255","CN"

NOTE: in the example below, we’ll work with Maxmind Country code lite database.

The database looks like:

$ head GeoIPCountryWhois.csv
"1.0.0.0","1.0.0.255","16777216","16777471","AU","Australia"
"1.0.1.0","1.0.3.255","16777472","16778239","CN","China"
"1.0.4.0","1.0.7.255","16778240","16779263","AU","Australia"
"1.0.8.0","1.0.15.255","16779264","16781311","CN","China"
"1.0.16.0","1.0.31.255","16781312","16785407","JP","Japan"
"1.0.32.0","1.0.63.255","16785408","16793599","CN","China"
"1.0.64.0","1.0.127.255","16793600","16809983","JP","Japan"
"1.0.128.0","1.0.255.255","16809984","16842751","TH","Thailand"
"1.1.0.0","1.1.0.255","16842752","16843007","CN","China"
"1.1.1.0","1.1.1.255","16843008","16843263","AU","Australia"

In order to make it compatible with iprange tool, use cut:

$ cut -d, -f1,2,5 GeoIPCountryWhois.csv | head
"1.0.0.0","1.0.0.255","AU"
"1.0.1.0","1.0.3.255","CN"
"1.0.4.0","1.0.7.255","AU"
"1.0.8.0","1.0.15.255","CN"
"1.0.16.0","1.0.31.255","JP"
"1.0.32.0","1.0.63.255","CN"
"1.0.64.0","1.0.127.255","JP"
"1.0.128.0","1.0.255.255","TH"
"1.1.0.0","1.1.0.255","CN"
"1.1.1.0","1.1.1.255","AU"

Now, you can use it with iprange:

$ cut -d, -f1,2,5 GeoIPCountryWhois.csv | head | ./iprange 
1.0.0.0/24 "AU"
1.0.1.0/24 "CN"
1.0.2.0/23 "CN"
1.0.4.0/22 "AU"
1.0.8.0/21 "CN"
1.0.16.0/20 "JP"
1.0.32.0/19 "CN"
1.0.64.0/18 "JP"
1.0.128.0/17 "TH"
1.1.0.0/24 "CN"
1.1.1.0/24 "AU"

Country codes and HAProxy ACLs


Now we’re ready to turn IP ranges into subnets associated to a country code.
Still need to be able to use it in HAProxy.
The easiest is to write all the subnets concerning a country code in a single file.

$ cut -d, -f1,2,5 GeoIPCountryWhois.csv | ./iprange | sed 's/"//g' 
| awk -F' ' '{ print $1 >> $2".subnets" }'

And the result is nice:

$ ls *.subnets
A1.subnets  AX.subnets  BW.subnets  CX.subnets  FJ.subnets  GR.subnets  IR.subnets  LA.subnets  ML.subnets  NF.subnets  PR.subnets  SI.subnets  TK.subnets  VE.subnets
A2.subnets  AZ.subnets  BY.subnets  CY.subnets  FK.subnets  GS.subnets  IS.subnets  LB.subnets  MM.subnets  NG.subnets  PS.subnets  SJ.subnets  TL.subnets  VG.subnets
AD.subnets  BA.subnets  BZ.subnets  CZ.subnets  FM.subnets  GT.subnets  IT.subnets  LC.subnets  MN.subnets  NI.subnets  PT.subnets  SK.subnets  TM.subnets  VI.subnets
AE.subnets  BB.subnets  CA.subnets  DE.subnets  FO.subnets  GU.subnets  JE.subnets  LI.subnets  MO.subnets  NL.subnets  PW.subnets  SL.subnets  TN.subnets  VN.subnets
AF.subnets  BD.subnets  CC.subnets  DJ.subnets  FR.subnets  GW.subnets  JM.subnets  LK.subnets  MP.subnets  NO.subnets  PY.subnets  SM.subnets  TO.subnets  VU.subnets
AG.subnets  BE.subnets  CD.subnets  DK.subnets  GA.subnets  GY.subnets  JO.subnets  LR.subnets  MQ.subnets  NP.subnets  QA.subnets  SN.subnets  TR.subnets  WF.subnets
AI.subnets  BF.subnets  CF.subnets  DM.subnets  GB.subnets  HK.subnets  JP.subnets  LS.subnets  MR.subnets  NR.subnets  RE.subnets  SO.subnets  TT.subnets  WS.subnets
AL.subnets  BG.subnets  CG.subnets  DO.subnets  GD.subnets  HN.subnets  KE.subnets  LT.subnets  MS.subnets  NU.subnets  RO.subnets  SR.subnets  TV.subnets  YE.subnets
AM.subnets  BH.subnets  CH.subnets  DZ.subnets  GE.subnets  HR.subnets  KG.subnets  LU.subnets  MT.subnets  NZ.subnets  RS.subnets  ST.subnets  TW.subnets  YT.subnets
AN.subnets  BI.subnets  CI.subnets  EC.subnets  GF.subnets  HT.subnets  KH.subnets  LV.subnets  MU.subnets  OM.subnets  RU.subnets  SV.subnets  TZ.subnets  ZA.subnets
AO.subnets  BJ.subnets  CK.subnets  EE.subnets  GG.subnets  HU.subnets  KI.subnets  LY.subnets  MV.subnets  PA.subnets  RW.subnets  SY.subnets  UA.subnets  ZM.subnets
AP.subnets  BM.subnets  CL.subnets  EG.subnets  GH.subnets  ID.subnets  KM.subnets  MA.subnets  MW.subnets  PE.subnets  SA.subnets  SZ.subnets  UG.subnets  ZW.subnets
AQ.subnets  BN.subnets  CM.subnets  EH.subnets  GI.subnets  IE.subnets  KN.subnets  MC.subnets  MX.subnets  PF.subnets  SB.subnets  TC.subnets  UM.subnets
AR.subnets  BO.subnets  CN.subnets  ER.subnets  GL.subnets  IL.subnets  KP.subnets  MD.subnets  MY.subnets  PG.subnets  SC.subnets  TD.subnets  US.subnets
AS.subnets  BR.subnets  CO.subnets  ES.subnets  GM.subnets  IM.subnets  KR.subnets  ME.subnets  MZ.subnets  PH.subnets  SD.subnets  TF.subnets  UY.subnets
AT.subnets  BS.subnets  CR.subnets  ET.subnets  GN.subnets  IN.subnets  KW.subnets  MG.subnets  NA.subnets  PK.subnets  SE.subnets  TG.subnets  UZ.subnets
AU.subnets  BT.subnets  CU.subnets  EU.subnets  GP.subnets  IO.subnets  KY.subnets  MH.subnets  NC.subnets  PL.subnets  SG.subnets  TH.subnets  VA.subnets
AW.subnets  BV.subnets  CV.subnets  FI.subnets  GQ.subnets  IQ.subnets  KZ.subnets  MK.subnets  NE.subnets  PM.subnets  SH.subnets  TJ.subnets  VC.subnets

Which makes subnets available for 246 countries !!!!!

In example, the subnets associated to AUstralia are:

$ cat AU.subnets 
1.0.0.0/24
1.0.4.0/22
1.1.1.0/24
[...]

The bash loop below prepares the ACLs in a file called haproxy.cfg:

$ for f in `ls *.subnets` ; do echo $f | 
awk -F'.' '{ print "acl "$1" src -f "$0 >> "haproxy.cfg" }' ; done
$ head haproxy.cfg 
acl src A1 -f A1.subnets
acl src A2 -f A2.subnets
acl src AD -f AD.subnets
acl src AE -f AE.subnets
acl src AF -f AF.subnets
acl src AG -f AG.subnets
acl src AI -f AI.subnets
acl src AL -f AL.subnets
acl src AM -f AM.subnets
acl src AN -f AN.subnets

That makes a lot of countries ūüôā

Continent codes and HAProxy ACLs


Fortunately, we can summarizes it to continent. Copy and paste into a file the country code and continent relation from maxmind website: http://www.maxmind.com/app/country_continent.

The script below will create files with named with the continent name and containing the country codes related to it:

$ for c in `fgrep -v '-' country_continents.txt | sort -t',' -k 2` ; 
do echo $c | awk -F',' '{ print $1 >> $2".continent" }' ; done

We have now 7 new files:

$ ls *.continent
AF.continent  AN.continent  AS.continent  EU.continent  
NA.continent  OC.continent  SA.continent

Let’s have a look for countries in South America:

$ cat SA.continent 
AR
BO
BR
CL
CO
EC
FK
GF
GY
PE
PY
SR
UY
VE

Let’s aggregate subnets for each country in a continent into a single file:

$ for f in `ls *.continent` ; do for c in $(cat $f) ; 
do cat ${c}.subnets >> ${f%%.*}.subnets ; done ;  done

Now we can generate the HAProxy configuration file to use them:

$ for c in AF AN AS EU NA OC SA ; do 
echo acl $c src -f $c.subnets >> "haproxy.conf" ; done

Usage in HAProxy


Coming soon, an article giving some examples on how to use the files generated below to improve performance and security of your platforms.

Links

Efficient SMTP relay infrastructure with Postfix and load-balancers

Scalable architecture

In order to make your architecture scalable, you may often want to use a load-balancer or an application delivery controller.
When using one of them (or a reverse-proxy), the client information is almost all the time hidden. Or if you want to get them, it requires huge modifications in the architecture.

Unfortunately, for troubleshooting and security purpose it would be interesting to know the client information (mainly source IP address)…
That’s where the proxy protocol comes in.

The proxy protocol???

As explained in a previous article“preserve source ip address despite reverse proxies”, the proxy protocol was developped to maintain client information when chaining proxies and reverse-proxies.
Two main advantages when using it:

  • you can provide a downstream proxy or server (aka next hop) the client information (for now, mainly IP and port source)
  • you can use servers in multiple datacenter without a complex network architecture (just need to provide routing for a TCP connection)

Why simply not using TPROXY (transparent proxy) mode?


TPROXY allows a load-balancer or reverse-proxy to open the TCP connection to the server using the client IP address.
One of the drawback of TPROXY is that the default gateway of the application servers must be the load-balancer.
Or you must do policy based routing on your network which could be painfull.

Why Postfix and HAProxy?

HAProxy was the first software to implement the proxy protocol.
Note that you’ll have to use HAProxy 1.5 branch or patched HAProxy 1.4.
< advertisement >
An other solution would to use Aloha load-balancer which does everything for you in a box (from the OS to HAPrxoxy) with all the nice features you could expect. ūüėČ
< /advertisement >

Lately, Postfix implemented it. It is available in Postfix 2.10.
It is the first application server first application server to ship with it: THANKS and CONGRATULATION!!!!
Hopefully other MTAs will implement it soon. It is simple and brings so many improvements to an architecture.

SMTP, spam and securtiy


In SMTP, it is really important to know the client IP, since we use it most of the time through RBL to fight spam.
For security purpose as well: we may want to allow only some hosts to use our SMTP relays and block any other clients.
Without the proxy protocol, the load-balancer will hide the client IP with its own IP. You would have to maintain whitelists into the load-balancer (which is doable). Thanks to proxy protocol, Postscreen would be aware of the client IP, it means you could maintain lists directly into the MTA.

HAProxy and Postfix connection flow

The diagram below shows the protocols and the process in place in this kind of architecture:

           smtp              proxy-protocol
                             + smtp
(INTERNET) ---> 25 (HAPROXY)      --->      srv1:10024 (Postscreen
                                                       / smtpd)
                                  --->      srv2:10024 (Postscreen
                                                       / smtpd)

Note that the default gateway of the MTA servers is not anymore the load-balancer.
Both servers migt be in the same LAN or datacenter. Any type of architecture is now doable.

Configuration

HAProxy

frontend ft_smtp
  bind 0.0.0.0:25
  mode tcp
  no option http-server-close
  timeout client 1m
  log global
  option tcplog
  default_backend bk_postfix

backend bk_postfix
  mode tcp
  no option http-server-close
  log global
  option tcplog
  timeout server 1m
  timeout connect 5s
  server postfix 127.0.0.1:10024 send-proxy

Postfix

Note: I installed postfix in /opt/postfix directory

main.cf

queue_directory = /opt/postfix/var/spool/postfix
command_directory = /opt/postfix/usr/sbin
daemon_directory = /opt/postfix/usr/libexec/postfix
data_directory = /opt/postfix/var/lib/postfix
mail_owner = postfix
unknown_local_recipient_reject_code = 550
inet_interfaces = localhost
sendmail_path = /opt/postfix/usr/sbin/sendmail
newaliases_path = /opt/postfix/usr/bin/newaliases
mailq_path = /opt/postfix/usr/bin/mailq
setgid_group = postdrop
html_directory = no
manpage_directory = /opt/postfix/usr/local/man
sample_directory = /opt/postfix/etc/postfix
readme_directory = no
inet_protocols = ipv4
postscreen_upstream_proxy_protocol = haproxy

master.cf

10024     inet  n       -       n       -       1       postscreen
smtpd     pass  -       -       n       -       -       smtpd

See the results in Postfix logs

No proxy protocol

Jun 30 01:18:14 sd-33932 postfix/postscreen[2375]: 
       CONNECT from [127.0.0.1]:52841 to [127.0.0.1]:10024
Jun 30 01:18:22 sd-33932 postfix/smtpd[2376]: 
       disconnect from localhost[127.0.0.1]

With proxy protocol

Jun 29 09:13:41 sd-33932 postfix/postscreen[30505]: 
       CONNECT from [<client public IP>]:59338 to [<server IP>]:25
Jun 29 09:13:52 sd-33932 postfix/postscreen[30505]: 
       DISCONNECT [<client public IP>]:59338

Related Links

Links

Enhanced SSL load-balancing with Server Name Indication (SNI) TLS extension

Synopsis

Some time ago, we wrote an article which explained how to load-balance SSL services, maintaining affinity using the SSLID.
The main limitation of this kind of architecture is that you must dedicate a public IP address and port per service.
If you’re hosting web or mail services, you could run out of public IP address quickly.

TLS protocol has been extended in 2003, RFC 3546, by an extension called SNI: Server Name Indication, which allows a client to announce in clear the server name it is contacting.

NOTE: two RFC have obsoleted the one above, the latest one is RFC 6066

The Aloha Load-balancer can use this information to choose a backend or a server.
This allows people to share a single VIP for several services.

Of course, we can use SNI switching with SSLID affinity to build a smart and reliable SSL load-balanced platform.

NOTE: Server Name information is sent with each SSL Handshake, whether you’re establishing a new session or you’re resuming an old one.

SNI is independent from the protocol used at layer 7. So basically, it will work with IMAP, HTTP, SMTP, POP, etc…

Limitation

Bear in mind, that in 2012, not all clients are compatible with SNI.
Concerning web browsers, a few of used in 2012 them are still not compatible with this TLS protocol extension.

We strongly recommend you to read the Wikipedia Server Name Indication page which lists all the limitation of this extension.

  • Only HAProxy nightly snapshots from 8th of April are compatible (with no bug knows) with it.
  • Concerning Aloha, it will be available by Aloha Load-balancer firmware 5.0.2.

Diagram

The picture below shows a platform with a single VIP which host services for 2 applications:
sni_loadbalancing

We can use SNI information to choose a backend, then, inside a backend, we can use SSLID affinity.

Configuration

Choose a backend using SNI TLS extension


The configuration below matches names provided by the SNI extention and choose a farm based on it.
In the farm, it provides SSLID affinity.
If no SNI extention is sent, then we redirect the user to a server farm which can be used to tell the user to upgrade its software.

# Adjust the timeout to your needs
defaults
  timeout client 30s
  timeout server 30s
  timeout connect 5s

# Single VIP with sni content switching
frontend ft_ssl_vip
  bind 10.0.0.10:443
  mode tcp

  tcp-request inspect-delay 5s
  tcp-request content accept if { req_ssl_hello_type 1 }
  
  acl application_1 req_ssl_sni -i application1.domain.com
  acl application_2 req_ssl_sni -i application2.domain.com

  use_backend bk_ssl_application_1 if application_1
  use_backend bk_ssl_application_2 if application_2

  default_backend bk_ssl_default

# Application 1 farm description
backend bk_ssl_application_1
  mode tcp
  balance roundrobin

  # maximum SSL session ID length is 32 bytes.
  stick-table type binary len 32 size 30k expire 30m

  acl clienthello req_ssl_hello_type 1
  acl serverhello rep_ssl_hello_type 2

  # use tcp content accepts to detects ssl client and server hello.
  tcp-request inspect-delay 5s
  tcp-request content accept if clienthello

  # no timeout on response inspect delay by default.
  tcp-response content accept if serverhello

  stick on payload_lv(43,1) if clienthello

  # Learn on response if server hello.
  stick store-response payload_lv(43,1) if serverhello

  option ssl-hello-chk
  server server1 192.168.1.1:443 check
  server server2 192.168.1.2:443 check

# Application 2 farm description
backend bk_ssl_application_2
  mode tcp
  balance roundrobin

  # maximum SSL session ID length is 32 bytes.
  stick-table type binary len 32 size 30k expire 30m

  acl clienthello req_ssl_hello_type 1
  acl serverhello rep_ssl_hello_type 2

  # use tcp content accepts to detects ssl client and server hello.
  tcp-request inspect-delay 5s
  tcp-request content accept if clienthello

  # no timeout on response inspect delay by default.
  tcp-response content accept if serverhello

  stick on payload_lv(43,1) if clienthello

  # Learn on response if server hello.
  stick store-response payload_lv(43,1) if serverhello

  option ssl-hello-chk
  server server1 192.168.2.1:443 check
  server server2 192.168.2.2:443 check

# Sorry backend which should invite the user to update its client
backend bk_ssl_default
  mode tcp
  balance roundrobin
  
  # maximum SSL session ID length is 32 bytes.
  stick-table type binary len 32 size 30k expire 30m

  acl clienthello req_ssl_hello_type 1
  acl serverhello rep_ssl_hello_type 2

  # use tcp content accepts to detects ssl client and server hello.
  tcp-request inspect-delay 5s
  tcp-request content accept if clienthello

  # no timeout on response inspect delay by default.
  tcp-response content accept if serverhello

  stick on payload_lv(43,1) if clienthello

  # Learn on response if server hello.
  stick store-response payload_lv(43,1) if serverhello

  option ssl-hello-chk
  server server1 10.0.0.11:443 check
  server server2 10.0.0.12:443 check

Choose a server using SNI: aka SSL routing


The configuration below matches names provided by the SNI extention and choose a server based on it.
If no SNI is provided or we can’t find the expected name, then the traffic is forwarded to server3 which can be used to tell the user to upgrade its software.

# Adjust the timeout to your needs
defaults
  timeout client 30s
  timeout server 30s
  timeout connect 5s

# Single VIP 
frontend ft_ssl_vip
  bind 10.0.0.10:443
  mode tcp

  tcp-request inspect-delay 5s
  tcp-request content accept if { req_ssl_hello_type 1 }

  default_backend bk_ssl_default

# Using SNI to take routing decision
backend bk_ssl_default
  mode tcp

  acl application_1 req_ssl_sni -i application1.domain.com
  acl application_2 req_ssl_sni -i application2.domain.com

  use-server server1 if application_1
  use-server server2 if application_2
  use-server server3 if !application_1 !application_2

  option ssl-hello-chk
  server server1 10.0.0.11:443 check
  server server2 10.0.0.12:443 check
  server server3 10.0.0.13:443 check

Related Links

Links

load balancing, affinity, persistence, sticky sessions: what you need to know

Synopsis

To ensure high availability and performance of Web applications, it is now common to use a load-balancer.
While some people uses layer 4 load-balancers, it can be sometime recommended to use layer 7 load-balancers to be more efficient with HTTP protocol.

NOTE: To understand better the difference between such load-balancers, please read the Load-Balancing FAQ.

A load-balancer in an infrastructure

The picture below shows how we usually install a load-balancer in an infrastructure:
reverse_proxy
This is a logical diagram. When working at layer 7 (aka Application layer), the load-balancer acts as a reverse proxy.
So, from a physical point of view, it can be plugged anywhere in the architecture:

  • in a DMZ
  • in the server LAN
  • as front of the servers, acting as the default gateway
  • far away in an other separated datacenter

Why does load-balancing web application is a problem????


Well, HTTP is not a connected protocol: it means that the session is totally independent from the TCP connections.
Even worst, an HTTP session can be spread over a few TCP connections…
When there is no load-balancer involved, there won’t be any issues at all, since the single application server will be aware the session information of all users, and whatever the number of client connections, they are all redirected to the unique server.
When using several application servers, then the problem occurs: what happens when a user is sending requests to a server which is not aware of its session?
The user will get back to the login page since the application server can’t access his session: he is considered as a new user.

To avoid this kind of problem, there are several ways:

  • Use a clustered web application server where the session are available for all the servers
  • Sharing user’s session information in a database or a file system on application servers
  • Use IP level information to maintain affinity between a user and a server
  • Use application layer information to maintain persistance between a user and a server

NOTE: you can mix different technc listed above.

Building a web application cluster

Only a few products on the market allow administrators to create a cluster (like Weblogic, tomcat, jboss, etc…).
I’ve never configured any of them, but from Administrators I talk too, it does not seem to be an easy task.
By the way, for Web applications, clustering does not mean scaling. Later, I’ll write an article explaining while even if you’re clustering, you still may need a load-balancer in front of your cluster to build a robust and scalable application.

Sharing user’s session in a database or a file system

This Technic applies to application servers which has no clustering features, or if you don’t want to enable cluster feature from.
It is pretty simple, you choose a way to share your session, usually a file system like NFS or CIFS, or a Database like MySql or SQLServer or a memcached then you configure each application server with proper parameters to share the sessions and to access them if required.
I’m not going to give any details on how to do it here, just google with proper keywords and you’ll get answers very quickly.

IP source affinity to server

An easy way to maintain affinity between a user and a server is to use user’s IP address: this is called Source IP affinity.
There are a lot of issues doing that and I’m not going to detail them right now (TODO++: an other article to write).
The only thing you have to know is that source IP affinity is the latest method to use when you want to “stick” a user to a server.
Well, it’s true that it will solve our issue as long as the user use a single IP address or he never change his IP address during the session.

Application layer persistence

Since a web application server has to identify each users individually, to avoid serving content from a user to an other one, we may use this information, or at least try to reproduce the same behavior in the load-balancer to maintain persistence between a user and a server.
The information we’ll use is the Session Cookie, either set by the load-balancer itself or using one set up by the application server.

What is the difference between Persistence and Affinity

Affinity: this is when we use an information from a layer below the application layer to maintain a client request to a single server
Persistence: this is when we use Application layer information to stick a client to a single server
sticky session: a sticky session is a session maintained by persistence

The main advantage of the persistence over affinity is that it’s much more accurate, but sometimes, Persistence is not doable, so we must rely on affinity.

Using persistence, we mean that we’re 100% sure that a user will get redirected to a single server.
Using affinity, we mean that the user may be redirected to the same server…

What is the interraction with load-balancing???


In load-balancer you can choose between several algorithms to pick up a server from a web farm to forward your client requests to.
Some algorithm are deterministic, which means they can use a client side information to choose the server and always send the owner of this information to the same server. This is where you usually do Affinity ūüėČ IE: “balance source”
Some algorithm are not deterministic, which means they choose the server based on internal information, whatever the client sent. This is where you don’t do any affinity nor persistence ūüôā IE: “balance roundrobin” or “balance leastconn”
I don’t want to go too deep in details here, this can be the purpose of a new article about load-balancing algorithms…

You may be wondering: “we have not yet speak about persistence in this chapter”. That’s right, let’s do it.
As we saw previously, persistence means that the server can be chosen based on application layer information.
This means that persistence is an other way to choose a server from a farm, as load-balancing algorithm does.

Actually, session persistence has precedence over load-balancing algorithm.
Let’s show this on a diagram:

      client request
            |
            V
    HAProxy Frontend
            |
            V
     backend choice
            |
            V
    HAproxy Backend
            |
            V
Does the request contain
persistence information ---------
            |                   |
            | NO                |
            V                   |
    Server choice by            |  YES
load-balancing algorithm        |
            |                   |
            V                   |
   Forwarding request <----------
      to the server

Which means that when doing session persistence in a load balancer, the following happens:

  • the first user’s request comes in without session persistence information
  • the request bypass the session persistence server choice since it has no session persistence information
  • the request pass through the load-balancing algorithm, where a server is chosen and affected to the client
  • the server answers back, setting its own session information
  • depending on its configuration, the load-balancer can either use this session information or setup its own before sending the response back to the client
  • the client sends a second request, now with session information he learnt during the first request
  • the load-balancer choose the server based on the client side information
  • the request DOES NOT PASS THROUGH the load-balancing algorithm
  • the server answers the request

and so on…

At HAProxy Technologies we say that “Persistence is a exception to load-balancing“.
And the demonstration is just above.

Affinity configuration in HAProxy / Aloha load-balancer

The configuration below shows how to do affinity within HAProxy, based on client IP information:

frontend ft_web
  bind 0.0.0.0:80
  default_backend bk_web

backend bk_web
  balance source
  hash-type consistent # optional
  server s1 192.168.10.11:80 check
  server s2 192.168.10.21:80 check

Web application persistence

In order to provide persistence at application layer, we usually use Cookies.
As explained previously, there are two ways to provide persistence using cookies:

  • Let the load-balancer set up a cookie for the session.
  • Using application cookies, such as ASP.NET_SessionId, JSESSIONID, PHPSESSIONID, or any other chosen name

Session cookie setup by the Load-Balancer

The configuration below shows how to configure HAProxy / Aloha load balancer to inject a cookie in the client browser:

frontend ft_web
  bind 0.0.0.0:80
  default_backend bk_web

backend bk_web
  balance roundrobin
  cookie SERVERID insert indirect nocache
  server s1 192.168.10.11:80 check cookie s1
  server s2 192.168.10.21:80 check cookie s2

Two things to notice:

  1. the line “cookie SERVERID insert indirect nocache”:
    This line tells HAProxy to setup a cookie called SERVERID only if the user did not come with such cookie. It is going to append a “Cache-Control: nocache” as well, since this type of traffic is supposed to be personnal and we don’t want any shared cache on the internet to cache it
  2. the statement “cookie XXX” on the server line definition:
    It provides the value of the cookie inserted by HAProxy. When the client comes back, then HAProxy knows directly which server to choose for this client.

So what happens?

  1. At the first response, HAProxy will send the client the following header, if the server chosen by the load-balancing algorithm is s1:
    Set-Cookie: SERVERID=s1
  2. For the second request, the client will send a request containing the header below:
    Cookie: SERVERID=s1

Basically, this kind of configuration is compatible with active/active Aloha load-balancer cluster configuration.

Using application session cookie for persistence

The configuration below shows how to configure HAProxy / Aloha load balancer to use the cookie setup by the application server to maintain affinity between a server and a client:

frontend ft_web
  bind 0.0.0.0:80
  default_backend bk_web

backend bk_web
  balance roundrobin
  cookie JSESSIONID prefix nocache
  server s1 192.168.10.11:80 check cookie s1
  server s2 192.168.10.21:80 check cookie s2

Just replace JSESSIONID by your application cookie. It can be anything, like the default ones from PHP and IIS: PHPSESSID and ASP.NET_SessionId.

So what happens?

  1. At the first response, the server will send the client the following header
    Set-Cookie: JSESSIONID=i12KJF23JKJ1EKJ21213KJ
  2. when passing through HAProxy, the cookie is modified like this:
    Set-Cookie: JSESSIONID=s1~i12KJF23JKJ1EKJ21213KJ


    Note that the Set-Cookie header has been prefixed by the server cookie value (“s1” in this case) and a “~” is used as a separator between this information and the cookie value.

  3. For the second request, the client will send a request containing the header below:
    Cookie: JSESSIONID=s1~i12KJF23JKJ1EKJ21213KJ
  4. HAProxy will clean it up on the fly to set it up back like the origin:
    Cookie: JSESSIONID=i12KJF23JKJ1EKJ21213KJ

Basically, this kind of configuration is compatible with active/active Aloha load-balancer cluster configuration.

What happens when my server goes down???

When doing persistence, if a server goes down, then HAProxy will redispatch the user to an other server.
Since the user will get connected on a new server, then this one may not be aware of the session, so be redirected to the login page.
But this is not a load-balancer problem, this is related to the application server farm.

Links

Use a load-balancer as a first row of defense against DDOS

We‚Äôve seen recently more and more DOS and DDOS attacks. Some of them were very big, requiring thousands of computers…
But in most cases, this kind of attacks are made by a few computers aiming to make a service or website unavailable, either by sending it too many requests or by taking all its available resources, preventing regular users to use the service.
Some attacks targets known vulnerabilities of widely used applications.

In the present article, we’ll explain how to take advantage of an application delivery controller to protect your website and application against DOS, DDOS and vulnerability scans.

Why using a LB for such protection since a firewall and a Web Application Firewall (aka WAF) could already do the job?
Well, the Firewall is not aware of the application layer but would be useful to pretect against SYN flood attacks. That’s why we saw recently application layer firewalls: Web Application Firewalls, also known as WAF.
Well, since the load balancer is in front of the platform, it can be a good partner for the WAF, filtering out 99% of the attacks, which are managed by script kiddies. The WAF can then happily clean up the remaining attacks.
Well, maybe you don’t need a WAF and you want to take advantage of your Aloha and save some money ;).

Note that you need an application layer load-balancer, like Aloha or OpenSource HAProxy to be efficient.

TCP syn flood attacks


The syn flood attacks consist in sending as many TCP syn packets as possible to a single server trying to saturate it or at least, saturating its uplink bandwith.

If you’re using the Aloha load-balancer, you’re already protected against this kind of attacks: the Aloha includes mechanism to protect you.
The TCP syn flood attack mitigation capacity may vary depending on your Aloha box.

It you’re running your own LB based on HAProxy or HAPEE, you should have a look at the sysctl below (edit /etc/sysctl.conf or play with sysctl command):

# Protection SYN flood
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.tcp_max_syn_backlog = 1024 

Note: If the attack is very big and saturates your internet bandwith, the only solution is to ask your internet access provider to null route the attackers IPs on its core network.

Slowloris like attacks


For this kind of attack, the clients will send very slowly their requests to a server: header by header, or even worst character by character, waiting long time between each of them.
The server have to wait until the end of the request to process it and send back its response.
The purpose of this attack is to prevent regular users to use the service, since the attacker would be using all the available resources with very slow queries.

In order to protect your website against this kind of attack, just setup the HAProxy option ‚Äútimeout http-request‚ÄĚ.
You can set it up to 5s, which is long enough.
It tells HAProxy to let five seconds to a client to send its whole HTTP request, otherwise HAProxy would shut the connection with an error.

For example:

# On Aloha, the global section is already setup for you
# and the haproxy stats socket is available at /var/run/haproxy.stats
global
  stats socket ./haproxy.stats level admin

defaults
  option http-server-close
  mode http
  timeout http-request 5s
  timeout connect 5s
  timeout server 10s
  timeout client 30s

listen stats
  bind 0.0.0.0:8880
  stats enable
  stats hide-version
  stats uri     /
  stats realm   HAProxy Statistics
  stats auth    admin:admin

frontend ft_web
  bind 0.0.0.0:8080

  # Spalreadylit static and dynamic traffic since these requests have different impacts on the servers
  use_backend bk_web_static if { path_end .jpg .png .gif .css .js }

  default_backend bk_web

# Dynamic part of the application
backend bk_web
  balance roundrobin
  cookie MYSRV insert indirect nocache
  server srv1 192.168.1.2:80 check cookie srv1 maxconn 100
  server srv2 192.168.1.3:80 check cookie srv2 maxconn 100

# Static objects
backend bk_web_static
  balance roundrobin
  server srv1 192.168.1.2:80 check maxconn 1000
  server srv2 192.168.1.3:80 check maxconn 1000

To test this configuration, simply open a telnet to the frontend port and wait for 5 seconds:

telnet 127.0.0.1 8080
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
HTTP/1.0 408 Request Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<h1>408 Request Time-out</h1>
Your browser didn't send a complete request in time.

Connection closed by foreign host.

Unfair users, AKA abusers


By unfair users, I mean users (or scripts) which have an abnormal behavior on your website:

  • too many connections opened
  • new connection rate too high
  • http request rate too high
  • bandwith usage too high
  • client not respecting RFCs (IE for SMTP)

How does a regular browser works?


Before trying to protect your website from weird behavior, we have to define what a “normal” behavior is!
This paragraphs gives the main lines of how a browser works and there may be some differences between browsers.

So, when one wants to browse a website, we use a browser: Chrome, Firefox, Internet Explorer, Opera are the most famous ones.
After typing the website name in the URL bar, the browser will look like for the IP address of your website.
Then it will establish a tcp connection to the server, downloading the main page, analyze its content and follow its links from the HTML code to get the objects required to build the page: javascript, css, images, etc…
To get the objects, it may open up to 6 or 7 TCP connections per domain name.
Once it has finished to download the objects, it starts aggregating everything then print out the page.

Limiting the number of connections per users


As seen before, a browser opens up 5 to 7 TCP connections to a website when it wants to download objetcs and they are opened quite quickly.
One can consider that somebody having more than 10 connections opened is not a regular user.
The configuration below shows how to do this limitation in the Aloha and HAProxy:

This configuration also applies to any kind of TCP based application.

The most important lines are from 25 to 32.

# On Aloha, the global section is already setup for you
# and the haproxy stats socket is available at /var/run/haproxy.stats
global
  stats socket ./haproxy.stats level admin

defaults
  option http-server-close
  mode http
  timeout http-request 5s
  timeout connect 5s
  timeout server 10s
  timeout client 30s

listen stats
  bind 0.0.0.0:8880
  stats enable
  stats hide-version
  stats uri     /
  stats realm   HAProxy Statistics
  stats auth    admin:admin

frontend ft_web
  bind 0.0.0.0:8080

  # Table definition  
  stick-table type ip size 100k expire 30s store conn_cur

  # Allow clean known IPs to bypass the filter
  tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst }
  # Shut the new connection as long as the client has already 10 opened 
  tcp-request connection reject if { src_conn_cur ge 10 }
  tcp-request connection track-sc1 src

  # Split static and dynamic traffic since these requests have different impacts on the servers
  use_backend bk_web_static if { path_end .jpg .png .gif .css .js }

  default_backend bk_web

# Dynamic part of the application
backend bk_web
  balance roundrobin
  cookie MYSRV insert indirect nocache
  server srv1 192.168.1.2:80 check cookie srv1 maxconn 100
  server srv2 192.168.1.3:80 check cookie srv2 maxconn 100

# Static objects
backend bk_web_static
  balance roundrobin
  server srv1 192.168.1.2:80 check maxconn 1000
  server srv2 192.168.1.3:80 check maxconn 1000
  • NOTE: if several domain name points to your frontend, then you may want to increase the conn_cur limit. (Remember a browser opens its 5 to 7 TCP connections per domain name).
  • NOTE2: if several users are hidden behind the same IP (NAT or proxy), this configuration may have a negative impact for them. You can whitelist these IPs.

Testing the configuration

run an apache bench to open 10 connections and doing request on these connections:

ab -n 50000000 -c 10 http://127.0.0.1:8080/

Watch the table content on the haproxy stats socket:

echo "show table ft_web" | socat unix:./haproxy.stats -
# table: ft_web, type: ip, size:102400, used:1
0x7afa34: key=127.0.0.1 use=10 exp=29994 conn_cur=10

Let’s try to open an eleventh connection using telnet:

telnet 127.0.0.1 8080
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
Connection closed by foreign host.

Basically, opened connections can keep on working, while a new one can’t be established.

Limiting the connection rate per user


In the previous chapter, we’ve seen how to protect ourselves from somebody who wants to open more than X connections at the same time.
Well, this is good, but something which may kill performance would to allow somebody to open and close a lot of tcp connections over a short period of time.
As we’ve seen previously, a browser will open up to 7 TCP connections in a very short period of time (a few seconds). One can consider that somebody having more than 20 connections opened over a period of 3 seconds is not a regular user.
The configuration below shows how to do this limitation in the Aloha and HAProxy:

This configuration also applies to any kind of TCP based application.

The most important lines are from 25 to 32.

# On Aloha, the global section is already setup for you
# and the haproxy stats socket is available at /var/run/haproxy.stats
global
  stats socket ./haproxy.stats level admin

defaults
  option http-server-close
  mode http
  timeout http-request 5s
  timeout connect 5s
  timeout server 10s
  timeout client 30s

listen stats
  bind 0.0.0.0:8880
  stats enable
  stats hide-version
  stats uri     /
  stats realm   HAProxy Statistics
  stats auth    admin:admin

frontend ft_web
  bind 0.0.0.0:8080

  # Table definition  
  stick-table type ip size 100k expire 30s store conn_rate(3s)

  # Allow clean known IPs to bypass the filter
  tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst }
  # Shut the new connection as long as the client has already 10 opened 
  tcp-request connection reject if { src_conn_rate ge 10 }
  tcp-request connection track-sc1 src

  # Split static and dynamic traffic since these requests have different impacts on the servers
  use_backend bk_web_static if { path_end .jpg .png .gif .css .js }

  default_backend bk_web

# Dynamic part of the application
backend bk_web
  balance roundrobin
  cookie MYSRV insert indirect nocache
  server srv1 192.168.1.2:80 check cookie srv1 maxconn 100
  server srv2 192.168.1.3:80 check cookie srv2 maxconn 100

# Static objects
backend bk_web_static
  balance roundrobin
  server srv1 192.168.1.2:80 check maxconn 1000
  server srv2 192.168.1.3:80 check maxconn 1000
  • NOTE2: if several users are hidden behind the same IP (NAT or proxy), this configuration may have a negative impact for them. You can whitelist these IPs.

Testing the configuration


run 10 requests with ApacheBench, everything may be fine:

ab -n 10 -c 1 -r http://127.0.0.1:8080/

Using socat we can watch this traffic in the stick-table:

# table: ft_web, type: ip, size:102400, used:1
0x11faa3c: key=127.0.0.1 use=0 exp=28395 conn_rate(3000)=10

Running a telnet to run a eleventh request and the connections get closed:

telnet 127.0.0.1 8080
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
Connection closed by foreign host.

Limiting the HTTP request rate


Even if in the previous examples, we were using HTTP as the protocol, we based our protection on layer 4 information: number or opening rate of TCP connections.
An attacker could respect the number of connection we would set by emulating the behavior of a regular browser.
Now, let’s go deeper and see what we can do on HTTP protocol.

The configuration below tracks HTTP request rate per user on the backend side, blocking abusers on the frontend side if the backend detects abuse.

# On Aloha, the global section is already setup for you
# and the haproxy stats socket is available at /var/run/haproxy.stats
global
  stats socket ./haproxy.stats level admin

defaults
  option http-server-close
  mode http
  timeout http-request 5s
  timeout connect 5s
  timeout server 10s
  timeout client 30s

listen stats
  bind 0.0.0.0:8880
  stats enable
  stats hide-version
  stats uri     /
  stats realm   HAProxy Statistics
  stats auth    admin:admin

frontend ft_web
  bind 0.0.0.0:8080

  # Use General Purpose Couter (gpc) 0 in SC1 as a global abuse counter
  # Monitors the number of request sent by an IP over a period of 10 seconds
  stick-table type ip size 1m expire 10s store gpc0,http_req_rate(10s)
  tcp-request connection track-sc1 src
  tcp-request connection reject if { src_get_gpc0 gt 0 }

  # Split static and dynamic traffic since these requests have different impacts on the servers
  use_backend bk_web_static if { path_end .jpg .png .gif .css .js }

  default_backend bk_web

# Dynamic part of the application
backend bk_web
  balance roundrobin
  cookie MYSRV insert indirect nocache

  # If the source IP sent 10 or more http request over the defined period, 
  # flag the IP as abuser on the frontend
  acl abuse src_http_req_rate(ft_web) ge 10
  acl flag_abuser src_inc_gpc0(ft_web)
  tcp-request content reject if abuse flag_abuser

  server srv1 192.168.1.2:80 check cookie srv1 maxconn 100
  server srv2 192.168.1.3:80 check cookie srv2 maxconn 100

# Static objects
backend bk_web_static
  balance roundrobin
  server srv1 192.168.1.2:80 check maxconn 1000
  server srv2 192.168.1.3:80 check maxconn 1000
  • NOTE: if several users are hidden behind the same IP (NAT or proxy), this configuration may have a negative impact for them. You can whitelist these IPs.

Testing the configuration

run 10 requests with ApacheBench, everything may be fine:

ab -n 10 -c 1 -r http://127.0.0.1:8080/

Using socat we can watch this traffic in the stick-table:

# table: ft_web, type: ip, size:1048576, used:1
0xbebbb0: key=127.0.0.1 use=0 exp=8169 gpc0=1 http_req_rate(10000)=10

Running a telnet to run a eleventh request and the connections get closed:

telnet 127.0.0.1 8080
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
Connection closed by foreign host.

Detecting vulnerability scans

Vulnerability scans could generate different kind of errors which can be tracked by Aloha and HAProxy:

  • invalid and truncated requests
  • denied or tarpitted requests
  • failed authentications
  • 4xx error pages

HAProxy is able to monitor an error rate per user then can take decision based on it.

# On Aloha, the global section is already setup for you
# and the haproxy stats socket is available at /var/run/haproxy.stats
global
  stats socket ./haproxy.stats level admin

defaults
  option http-server-close
  mode http
  timeout http-request 5s
  timeout connect 5s
  timeout server 10s
  timeout client 30s

listen stats
  bind 0.0.0.0:8880
  stats enable
  stats hide-version
  stats uri     /
  stats realm   HAProxy Statistics
  stats auth    admin:admin

frontend ft_web
  bind 0.0.0.0:8080

  # Use General Purpose Couter 0 in SC1 as a global abuse counter
  # Monitors the number of errors generated by an IP over a period of 10 seconds
  stick-table type ip size 1m expire 10s store gpc0,http_err_rate(10s)
  tcp-request connection track-sc1 src
  tcp-request connection reject if { src_get_gpc0 gt 0 }

  # Split static and dynamic traffic since these requests have different impacts on the servers
  use_backend bk_web_static if { path_end .jpg .png .gif .css .js }

  default_backend bk_web

# Dynamic part of the application
backend bk_web
  balance roundrobin
  cookie MYSRV insert indirect nocache

  # If the source IP generated 10 or more http request over the defined period, 
  # flag the IP as abuser on the frontend
  acl abuse src_http_err_rate(ft_web) ge 10
  acl flag_abuser src_inc_gpc0(ft_web)
  tcp-request content reject if abuse flag_abuser

  server srv1 192.168.1.2:80 check cookie srv1 maxconn 100
  server srv2 192.168.1.3:80 check cookie srv2 maxconn 100

# Static objects
backend bk_web_static
  balance roundrobin
  server srv1 192.168.1.2:80 check maxconn 1000
  server srv2 192.168.1.3:80 check maxconn 1000

Testing the configuration

run an apache bench, pointing it on a purposely wrong URL:

ab -n 10 http://127.0.0.1:8080/dlskfjlkdsjlkfdsj

Watch the table content on the haproxy stats socket:

echo "show table ft_web" | socat unix:./haproxy.stats -
# table: ft_web, type: ip, size:1048576, used:1
0x8a9770: key=127.0.0.1 use=0 exp=5866 gpc0=1 http_err_rate(10000)=11

Let‚Äôs try to run the same ab command and let’s get the error:

apr_socket_recv: Connection reset by peer (104)

which means that HAProxy has blocked the IP address

Notes

  • We could combine configuration example above together to improve protection. This will be described later in an other article
  • The numbers provided in the examples may be different for your application and architecture. Bench your configuration properly before applying in production.

Related articles

Links

Web traffic limitation

Synopsis

For different reason, we may want to limit the number of connections or the number of requests we allow to a web farm.
In example:

  • give more capacity to authenticated users compared to anonymous one
  • limit web farm users per virtualhost
  • protect your website from spiders
  • etc…

Basically, we’ll manage two webfarm, one with as much as capacity as we need, and an other one where we’ll redirect people we want to slow down.
The routing decision can be taken using a header, a cookie, a part of the url, source IP address, etc…

Configuration

The configuration below would do the job.

There are only two webservers in the farm, but we want to slow down some virtual host or old and almost never used applications in order to protect and let more capacity to the regular traffic.

you can play with the inspect-delay time to be more or less aggressive.

frontend www
  bind :80
  mode http
  acl spiderbots hdr_cnt(User-Agent) eq 0
  acl personnal hdr(Host) www.personnalwebsite.tld www.oldname.tld
  acl oldies path_beg /old /foo /bar
  use_backend limited_www if spiderbots or personnal or oldies
  default_backend www

backend www
 mode http
 server be1  192.168.0.1:80 check maxconn 100
 server be1  192.168.0.2:80 check maxconn 100

backend limited_www
 mode http
 acl too_fast be_sess_rate gt 10
 acl too_many be_conn gt 10
 tcp-request inspect-delay 3s
 tcp-request content accept if ! too_fast or ! too_many
 tcp-request content accept if WAIT_END
 server be1  192.168.0.1:80 check maxconn 100
 server be1  192.168.0.2:80 check maxconn 100

Results

Without the example above, an apache bench would be able to go up to 3600 req/s on the regular farm and only 9 req/s on the limited one.

Related articles

Links

What’s new in HAProxy 1.5-dev7

HAProxy 1.5-dev7 has been released recently with a lot of new features (compared to 1.5-dev6).
The present article will try to introduce all of them.
I won’t go through the bug fixes.

performance tuning parameter

maxconnrate
Set the maximum number connection per second an HAProxy process will accept.

Proxy parameter

option http-no-delay
Reduce delay for very rare situation where the application relies on an interactive HTTP payload.
In such situation, delay can have an impact on performance.

errorfile <code> <url>
errorloc <code> <url>
errorloc303 <code> <url>
Can now return an HTTP status code 200 to requests matching the monitor-uri.

Health checking

option redis-check
Redis health check

Stick tables

stick store-response
Use a response pattern to create an entry in a stick table.

Logging

first character
As you may know, the first character of the log flags is used to define the first event that caused the session to terminate.
There are now 2 new flag values:
D: session killed by HAProxy because the server was DOWN and was configured to kill all connections when going DOWN.
K: session killed by an administrator

Unix socket command

clear table <table>
Remove entries from the stick-table <table>.

disable frontend <frontend>
Mark a frontend as temporarily stopped.

enable frontend <frontend>
Resume a frontend which has been stopped temporarily.

set maxconn frontend <frontend> <value>
Dynamically change the maxconn frontend setting

set maxconn global <maxconn>
Dynamically change the global maxconn setting.

set rate-limit connections global <value>
Change the process-wide connection rate limit, which is set by the global maxconnrate setting

show table <name>
Dump contents of stick-table .

shutdown frontend <frontend>
Completely delete a frontend

shutdown session <id>
Immediately terminate a session

Links