Tag Archives: ssl

Mitigating the SSL Beast attack using the ALOHA Load-Balancer / HAProxy

The beast attack on SSL isn’t new, but we have not yet published an article to explain how to mitigate it with the ALOHA or HAProxy.
First of all, to mitigate this attack, you must use the Load-Balancer as the SSL endpoint, then just append the following parameter on your HAProxy SSL frontend:
  * For the ALOHA Load-Balancer:

bind 10.0.0.9:443 name https ssl crt domain ciphers RC4:HIGH:!aNULL:!MD5

  * For HAProxy OpenSource:

bind 10.0.0.9:443 name https ssl crt /path/to/domain.pem ciphers RC4:HIGH:!aNULL:!MD5

As you may have understood, the most important part is the ciphers RC4:HIGH:!aNULL:!MD5 directive which can be used to force the cipher used during the connection and to force it to be strong enough to resist to the attack.

Related Links

Links

HAProxy log customization

Synopsis

One of the strength of HAProxy is its logging system. It is very verbose and provides many information.
HAProxy HTTP log line is briefly explained in an HAProxy Technologies memo. It’s a must have document when you have to analyze HAProxy‘s log lines to troubleshoot an issue.
An other interesting tool is HALog. It is available in HAProxy‘s sources, in the contrib directory. I’ll write later an article about it. In order to have an idea on how to use it, just have a look at HAProxy Technologies howto related to halog and HTTP analyze.

Why customizing HAProxy’s logs ???


There may be several reasons why one want to customize HAProxy’s logs:

  • the default log format is too much complicated
  • there are too many information in the default log format
  • there is not enough information in the default log format
  • third party log anaylzer can hardly understand default HAProxy log format
  • logs generated by HAPorxy must be compliant to an existing format from an existing appliance in the architecture
  • … add your own reason here …

That’s why, at HAProxy Technologies, we felt the need of letting our users to create their own HAProxy log-format.
As for compression in HAProxy, the job was done by Wlallemand.

HAProxy log format customization

Configuration directive

The name of the directive which allows you to generate a home made log format is simply called log-format.

Variables

The log-format directive understand variables.
A variable follows the rules below:

  • it is preceded by a percent character: ‘%
  • it can take arguments in braces ‘{}‘.
  • If multiple arguments, then they are separated by commas ‘,‘ within the braces.
  • Flags may be added or removed by prefixing them with a ‘+‘ or ‘‘ sign.
  • spaces ‘ ‘ must be escaped (It is considered as a separator)

Currently available flags:

  • Q: quote a string
  • X: hexadecimal representation (IPs, Ports, %Ts, %rt, %pid)

Currently available variables:

  +---+------+-----------------------------------------------+-------------+
  | R | var  | field name (8.2.2 and 8.2.3 for description)  | type        |
  +---+------+-----------------------------------------------+-------------+
  |   | %o   | special variable, apply flags on all next var |             |
  +---+------+-----------------------------------------------+-------------+
  |   | %B   | bytes_read                                    | numeric     |
  |   | %Ci  | client_ip                                     | IP          |
  |   | %Cp  | client_port                                   | numeric     |
  |   | %Bi  | backend_source_ip                             | IP          |
  |   | %Bp  | backend_source_port                           | numeric     |
  |   | %Fi  | frontend_ip                                   | IP          |
  |   | %Fp  | frontend_port                                 | numeric     |
  |   | %H   | hostname                                      | string      |
  |   | %ID  | unique-id                                     | string      |
  |   | %Si  | server_IP                                     | IP          |
  |   | %Sp  | server_port                                   | numeric     |
  |   | %T   | gmt_date_time                                 | date        |
  |   | %Tc  | Tc                                            | numeric     |
  | H | %Tq  | Tq                                            | numeric     |
  | H | %Tr  | Tr                                            | numeric     |
  |   | %Ts  | timestamp                                     | numeric     |
  |   | %Tt  | Tt                                            | numeric     |
  |   | %Tw  | Tw                                            | numeric     |
  |   | %ac  | actconn                                       | numeric     |
  |   | %b   | backend_name                                  | string      |
  |   | %bc  | beconn                                        | numeric     |
  |   | %bq  | backend_queue                                 | numeric     |
  | H | %cc  | captured_request_cookie                       | string      |
  | H | %rt  | http_request_counter                          | numeric     |
  | H | %cs  | captured_response_cookie                      | string      |
  |   | %f   | frontend_name                                 | string      |
  |   | %ft  | frontend_name_transport ('~' suffix for SSL)  | string      |
  |   | %fc  | feconn                                        | numeric     |
  | H | %hr  | captured_request_headers default style        | string      |
  | H | %hrl | captured_request_headers CLF style            | string list |
  | H | %hs  | captured_response_headers default style       | string      |
  | H | %hsl | captured_response_headers CLF style           | string list |
  |   | %ms  | accept date milliseconds                      | numeric     |
  |   | %pid | PID                                           | numeric     |
  | H | %r   | http_request                                  | string      |
  |   | %rc  | retries                                       | numeric     |
  |   | %s   | server_name                                   | string      |
  |   | %sc  | srv_conn                                      | numeric     |
  |   | %sq  | srv_queue                                     | numeric     |
  | S | %sslc| ssl_ciphers (ex: AES-SHA)                     | string      |
  | S | %sslv| ssl_version (ex: TLSv1)                       | string      |
  | H | %st  | status_code                                   | numeric     |
  |   | %t   | date_time                                     | date        |
  |   | %ts  | termination_state                             | string      |
  | H | %tsc | termination_state with cookie status          | string      |
  +---+------+-----------------------------------------------+-------------+

    R = Restrictions : H = mode http only ; S = SSL only

Log format examples

Default log format

  • TCP log format
    log-format %Ci:%Cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts 
               %ac/%fc/%bc/%sc/%rc %sq/%bq
    
  • HTTP log format
    log-format %Ci:%Cp [%t] %ft %b/%s %Tq/%Tw/%Tc/%Tr/%Tt %st %B %cc 
               %cs %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r
    
  • CLF log format
    log-format %{+Q}o %{-Q}Ci - - [%T] %r %st %B "" "" %Cp 
               %ms %ft %b %s %Tq %Tw %Tc %Tr %Tt %tsc %ac %fc 
               %bc %sc %rc %sq %bq %cc %cs %hrl %hsl
    

Home made formats

  • Logging HTTP Host header, the URL, the status code, number of bytes read from server and the server response time
    capture request header Host len 32
    log-format %hr %r %st %B %Tr
    
  • SSL log format with: HAProxy path (frontend, backend and server name), client information (source IP and port), SSL information (protocol version and negotiated cypher), connection termination state, including a few strings:
    log-format frontend:%f %b/%s client_ip:%Ci client_port:%Cp SSL_version:%sslv SSL_cypher:%sslc %ts

Links

SSL Client certificate management at application level

HAProxy and SSL

The history of SSL in HAProxy is very short: around one month ago, we announced the ability for HAProxy to offload SSL from the servers.
HAProxy SSL stack comes with some advanced features like TLS extension SNI.


Well, since yesterday afternoon (Tuesday the 2nd), HAProxy can also offload the client certificate management from the server, with some advanced features. This is the purpose of today’s article.
Again, all the dev is provided by HAProxy Technologies.
For the people using the ALOHA Load-Balancer, these features will be included in the next release without GUI integration (which will come later).
Concerning HAProxy, just git clone the latest version or wait for HAProxy-1.5-dev13. When compiling, don’t forget the USE_OPENSSL=yes flag.

Introduction


Why client certificates?


The main purpose of using client-side certificates is to increase the level of authentication.
Since HAProxy is often in front of web platform, it is the right place to do this authentication. That way, it could do all the certificate checking before allowing the user to pass through. Then it can process SSL on behalf of server and apply any standard features.
The main purpose of the article is to introduce the new HAProxy features related to SSL client certificates.

Basically, we’ll see how to protect access to our application with client-side certificates and how to properly redirect users to the right page when there is an issue with their certificates.

SSL Client certificate generation: thanks nginx!


Well, we’ll have to create a CA, a server certificate and clients certificates!
Nathan, a nginx user, has written a very nice and well documented article on how to generate a CA and client certificate here: http://blog.nategood.com/client-side-certificate-authentication-in-ngi.

So I won’t rewrite all the procedure here, just follow Nathan instructions to create your own CA and generate a few client certificates. All you need is available here:
https://github.com/exceliance/haproxy/blob/master/blog/ssl_client_certificate_management_at_application_level/

Other stuff

During this article, we’ll use a few client certificates: client1, client2 and client_expired (whose certificate has … expired!).
On the githbu link above, you’ll find as well how to generate the PEM file required by HAProxy.

Phase 1: Client Certificate mandatory

In the configuration below, only users with a client certificate are allowed to get connected on the application. This is achieved by the keywords “verify required“.

frontend ft_ssltests
 mode http
 bind 192.168.10.1:443 ssl crt ./server.pem ca-file ./ca.crt verify required
 default_backend bk_ssltests

backend ssltests
 mode http
 server s1 192.168.10.101:80 check
 server s2 192.168.10.102:80 check

If the client does not provide any certificate, then HAProxy would shut the connection during the SSL handshake. It’s up the the user’s software to report the right error…
Testing:

  • Connection with a certificate is allowed:
    $ openssl s_client -connect 192.168.10.1:443 -cert ./client1.crt -key ./client1.key
  • Connection without a certificate is refused:
    $ openssl s_client -connect 192.168.10.1:443
    [...]ssl handshake failure[...]
  • Connection with an expired certificate is refused too:
    $ openssl s_client -connect 192.168.10.1:443 -cert ./client_expired.crt -key ./client_expired.key 
    [...]ssl handshake failure[...]

Phase 2: Client Certificate optional

In the configuration below, all users, those with and those without the certificate are allowed to get connected. This is achieved by the keyword “verify optional“.
We can redirect users to different farm, based on the presence of the certificate or not:

frontend ssltests
 mode http
 bind 192.168.10.1:443 ssl crt ./server.pem ca-file ./ca.crt verify optional
 use_backend sharepoint if { ssl_fc_has_crt }     # check if the certificate has been provided and give access to the application
 default_backend webmail

backend sharepoint
 mode http
 server srv1 192.168.10.101:80 check
 server srv2 192.168.10.102:80 check

backend webmail
 mode http
 server srv3 192.168.10.103:80 check
 server srv4 192.168.10.104:80 check
  • If the client does not provide any certificate, then HAProxy route him to the webmail.
  • If the client provides a certificate, then HAProxy routes him to the application (sharepoint in our example)
  • If the client provides an expired certificate, then HAProxy refuses the connection like in the phase 1

Phase 3: Client Certificate optional and managing expired certificates

In the configuration below, all users, those with and those without the certificate are allowed to get connected. This is achieved by the keyword “verify optional“.
The option “crt-ignore-err 10” tells HAProxy to ignore Certificate errors 10 which actually matches the expired certificate.
We can redirect users to different farm, based on the presence of the certificate or not and we can propose a dedicated page for users whose certificate has expired with a procedure on how to renew or ask for a new certificate

frontend ssltests
 mode http
 bind 192.168.10.1:443 ssl crt ./server.pem ca-file ./ca.crt verify optional crt-ignore-err 10
 use_backend static if { ssl_c_verify 10 }  # if the certificate has expired, route the user to a less sensitive server to print an help page
 use_backend sharepoint if { ssl_fc_has_crt }        # check if the certificate has been provided and give access to the application
 default_backend webmail

backend static
 mode http
 option http-server-close
 redirect location /certexpired.html if { ssl_c_verify 10 } ! { path /certexpired.html }
 server srv5 192.168.10.105:80 check
 server srv6 192.168.10.106:80 check

backend sharepoint
 mode http
 server srv1 192.168.10.101:80 check
 server srv2 192.168.10.102:80 check

backend webmail
 mode http
 server srv3 192.168.10.103:80 check
 server srv4 192.168.10.104:80 check
  • If the client does not provide any certificate, then HAProxy route him to the webmail.
  • If the client provides a certificate, then HAProxy routes him to the application (sharepoint in our example)
  • If the client provides an expired certificate, then HAProxy routes him to a static server (non-sensitive) and force the users to show the page which provides the explanation about the expired certificate and how to renew it (it’s up to the admin to write this page).

Phase 4: Client Certificate optional, managing expired certificates and a revocation list

In the configuration below, all users, those with and those without the certificate are allowed to get connected. This is achieved by the keyword “verify optional“.
The option “crt-ignore-err all” tells HAProxy to ignore any client Certificate error.
The option “crl-file ./ca_crl.pem” tells HAProxy to check if the client has not been revoked in the Certificate Revocation List provided in argument.
We can redirect users to different farm, based on the presence of the certificate or not and we can propose a dedicated page for users whose certificate has expired with a procedure on how to renew or ask for a new certificate. We can also present a dedicate page to users whose certificate has been revoked.

frontend ssltests
 mode http
 bind 192.168.10.1:443 ssl crt ./server.pem ca-file ./ca.crt verify optional crt-ignore-err all crl-file ./ca_crl.pem
 use_backend static unless { ssl_c_verify 0 }  # if there is an error with the certificate, then route the user to a less sensitive farm
 use_backend sharepoint if { ssl_fc_has_crt }           # check if the certificate has been provided and give access to the application
 default_backend webmail

backend static
 mode http
 option http-server-close
 redirect location /certexpired.html if { ssl_c_verify 10 } ! { path /certexpired.html } # SSL error 10 means "certificate expired"
 redirect location /certrevoked.html if { ssl_c_verify 23 } ! { path /certrevoked.html } # SSL error 23 means "Certificate revoked"
redirect location /othererrors.html unless { ssl_c_verify 0 } ! { path /othererrors.html }
 server srv5 192.168.10.105:80 check
 server srv6 192.168.10.106:80 check

backend sharepoint
 mode http
 server srv1 192.168.10.101:80 check
 server srv2 192.168.10.102:80 check

backend webmail
 mode http
 server srv3 192.168.10.103:80 check
 server srv4 192.168.10.104:80 check
  • If the client does not provide any certificate, then HAProxy route him to the webmail.
  • If the client provides a certificate, then HAProxy routes him to the application (sharepoint in our example)
  • If the client provides an expired certificate, then HAProxy routes him to a static server (non-sensitive) and force the users to show the page which provides the explanation about the expired certificate and how to renew it (it’s up to the admin to write this page).
  • If the client provides a revoked certificate, then HAProxy routes him to a static server (non-sensitive) and force the users to show the page which provides the explanation about the revoked certificate (it’s up to the admin to write this page).
  • For any other errors related to the client certificate, then HAProxy routes the user to a static server (non-sensitive) and force the users to show a page explaining there has been an error and how to contact the support (it’s up to the admin to write this page).

Phase 5: same as phase 4, but with multiple CAs, Cert error in header and some ACLs

In the configuration below, all users, those with and those without the certificate are allowed to get connected. This is achieved by the keyword “verify optional“.
The option “crt-ignore-err all” tells HAProxy to ignore all client Certificate.
The option “crl-file ./ca_crl.pem” tells HAProxy to check if the client has not been revoked in the Certificate Revocation List provided in argument.
The file ca.pem contains 2 CAs: ca and ca2.
We can redirect users to different farm, based on the presence of the certificate or not and we can propose a dedicated page for users whose certificate has expired with a procedure on how to renew or ask for a new certificate. We can also present a dedicate page to users whose certificate has been revoked.

frontend ssltests
 mode http
 bind 192.168.10.1:443 ssl crt ./server.pem ca-file ./ca.pem verify optional crt-ignore-err all crl-file ./ca_crl.pem
 use_backend static unless { ssl_c_verify 0 }  # if there is an error with the certificate, then route the user to a less sensitive farm
 use_backend sharepoint if { ssl_fc_has_crt }           # check if the certificate has been provided and give access to the application
 default_backend webmail

backend static
 mode http
 option http-server-close
 acl url_expired path /certexpired.html
 acl url_revoked path /certrevoked.html
 acl url_othererrors path /othererrors.html
 acl cert_expired ssl_c_verify 10
 acl cert_revoked ssl_c_verify 23
 reqadd X-Ssl-Error: 10 if cert_expired
 reqadd X-Ssl-Error: 23 if cert_revoked
 reqadd X-Ssl-Error: other if ! cert_expired ! cert_revoked
 redirect location /certexpired.html if cert_expired ! url_expired
 redirect location /certrevoked.html if cert_revoked ! url_revoked
 redirect location /othererrors.html if ! cert_expired ! cert_revoked ! url_othererrors
 server srv5 192.168.10.105:80 check
 server srv6 192.168.10.106:80 check

backend sharepoint
 mode http
 server srv1 192.168.10.101:80 check
 server srv2 192.168.10.102:80 check

backend webmail
 mode http
 server srv3 192.168.10.103:80 check
 server srv4 192.168.10.104:80 check
  • If the client does not provide any certificate, then HAProxy route him to the webmail.
  • If the client provides a certificate, then HAProxy routes him to the application (sharepoint in our example)
  • If the client provides an expired certificate, then HAProxy routes him to a static server (non-sensitive) and force the users to show the page which provides the explanation about the expired certificate and how to renew it (it’s up to the admin to write this page).
  • If the client provides a revoked certificate, then HAProxy routes him to a static server (non-sensitive) and force the users to show the page which provides the explanation about the revoked certificate (it’s up to the admin to write this page).
  • For any other errors related to the client certificate, then HAProxy routes the user to a static server (non-sensitive) and force the users to show a page explaining there has been an error and how to contact the support (it’s up to the admin to write this page).

Coming soon…


Later, we’ll improve HAProxy client certificate management with:

  • client certificate information in HTTP header
  • ACLs to match the client information provided in the certificate and take classic decision (routing, blocking, etc….)
  • Persistence based on the information provided by the certificate (stick tables)
  • Ability to use a client certificate to get connected (and authenticated) on a server

Don’t hesitate to send your us your wishes!

Related links

Links

How to get SSL with HAProxy getting rid of stunnel, stud, nginx or pound

Update: HAProxy can now handle SSL client certificate: SSL Client certificate management at application level

History

HAProxy is well know for its performance as a reverse-proxy and load-balancer and is widely deployed on web platforms where performance matters. It is sometimes even used to replace hardware load-balancers such as F5 appliances.
When the platform requires SSL, it is common to use nginx, Pound or http://www.stunnel.org/index.html. Recently, stud came in the dance with a major advantage over other software: support for HAProxy’s proxy protocol.

At HAProxy Technologies, we build our ALOHA load-balancers using HAProxy and we use stunnel as the SSL offloading software. Our clients wanted some new features on our SSL implementation we could not provide through stunnel.
By the way, you can find our stunnel patches here: http://www.haproxy.com/download/free/patches/stunnel/
An other important thing to notice, is that stunnel does not scale very well: when managing a lot en encrypted connections, stud or nginx are far much better.
That’s why we decided to implement SSL directly in HAProxy. For now, it is still quite basic: SSL offloading with SNI support and wildcard certificates, ability to encrypt traffic to servers.
But at least, the performance are here!
We’ll keep on improving it later with new features, IE: client certificate management and some fun stuff with ACLs: stay tuned!

Remember that the job was done by HAProxy Technologies engineers.

Note that if you’re using the softwares listed above for other purpose than SSL, then you may still use them. For example, nginx performs very well on static content and on dynamic using php-fpm.

SSL offloading diagram

This is pretty simple, as shown on the picture below. The client will get connected on HAProxy using SSL, HAProxy will process SSL and get connected in clear to the server:
ssl offloading diagram

HAproxy installation

cd /usr/src
wget http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev12.tar.gz
tar xzf haproxy-1.5-dev12.tar.gz
cd haproxy-1.5-dev12/
make TARGET=linux2628 USE_STATIC_PCRE=1 USE_OPENSSL=1
sudo make PREFIX=/opt/haproxy-ssl install

HAProxy configuration for SSL offloading

First of all, you have to generate a few keys and a certificates using openssl and concatenate them in a file, the certificate first, then the key.

HAProxy configuration, very basic, for test purpose, and just to let you know which lines are very important:

defaults
 log 127.0.0.1 local0
 option tcplog

frontend ft_test
  mode http
  bind 0.0.0.0:8443 ssl crt ./haproxy.pem crt ./certs/ prefer-server-ciphers
  # other (self described) options are: [ciphers <suite>] [nosslv3] [notlsv1]
  use_backend bk_cert1 if { ssl_fc_sni cert1 } # content switching based on SNI
  use_backend bk_cert2 if { ssl_fc_sni cert2 } # content switching based on SNI
  default_backend bk_www.haproxy.com

backend bk_www.haproxy.com
 mode http
 server srvxlc 127.0.0.1:80

backend bk_cert1
  mode http
  server srv1 127.0.0.1:80

backend bk_cert2
  mode http
  server srv2 127.0.0.1:80

As you can see, HAProxy load one cert haproxy.pem which will be default one, and all the certificates from the certs dir. Actually, I have only 2 for my tests: cert1 and cert2.

Running HAProxy

First, just test the configuration is valid:

/opt/haproxy-ssl/sbin/haproxy -c -f ./hassl.cfg 
[WARNING] 247/110924 (6497) : config : missing timeouts for frontend 'ft_test'.
   | While not properly invalid, you will certainly encounter various problems
   | with such a configuration. To fix this, please ensure that all following
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[WARNING] 247/110924 (6497) : config : missing timeouts for backend 'bk_test'.
   | While not properly invalid, you will certainly encounter various problems
   | with such a configuration. To fix this, please ensure that all following
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
Configuration file is valid

Don’t worry about warnings, I purposely wrote a very basic configuration.

Now, you can run HAProxy:

/opt/haproxy-ssl/sbin/haproxy  -f ./ha.cfg

Testing SSL provided by HAProxy

Check the default certificate server name:

openssl s_client -connect 127.0.0.1:8443 -servername www.haproxy.com
[...]
Certificate chain
 0 s:/CN=www.haproxy.com
   i:/CN=www.haproxy.com
[...]

HAProxy log line:

[...] ft_test bk_www.haproxy.com/srvxlc [...]

Checking cert1, loaded from ./certs/ dir:

openssl s_client -connect 127.0.0.1:8443 -servername cert1
[...]
Certificate chain
 0 s:/CN=cert1
   i:/CN=cert1
[...]

HAProxy log line:

[...] ft_test bk_cert1/srv1 [...]

Checking cert2, loaded from ./certs/ dir:

openssl s_client -connect 127.0.0.1:8443 -servername cert2
[...]
Certificate chain
 0 s:/CN=cert2
   i:/CN=cert2
[...]

HAProxy log line:

[...] ft_test bk_cert2/srv2 [...]

Checking with an unknown servername:

openssl s_client -connect 127.0.0.1:8443 -servername kemp
[...]
Certificate chain
 0 s:/CN=www.haproxy.com
   i:/CN=www.haproxy.com
[...]

HAProxy log line:

[...] ft_test bk_www.haproxy.com/srvxlc [...]

When the name is unknown, the failover is well done on the default certificate.

And voilà !!!
Since it has been released in the 1.5 branch, you can use it in production 🙂

Related articles

Links

HOWTO SSL native in HAProxy

IMPORTANT NOTE: this article has been outdated since HAProxy-1.5-dev12 has been released (10th of September). For more information about SSL inside HAProxy. please read:


How to get SSL with HAProxy getting rid of stunnel, stud, nginx or pound

Synopsis

Since yesterday night (FR time), HAProxy can support SSL offloading. It can even crypt traffic to a downstream server.
We’ll see later all the fun we could have with these nice features and the goodness it could bring in term of architecture. Today, I’ll just focus on how to install and configure HAProxy to offload SSL processing from your servers.

It’s important to notice that in order to be able to manage SSL connections, a huge rework of connection management has been done in HAProxy. Despite the long time spent on testing, there might still remain some bugs.
So we ask anybody who tests the procedure below to report bugs to HAProxy mailing list.

Note as well that the job was done by HAProxy Technologies engineers, who already improved stunnel and stud.

SSL offloading diagram

This is pretty simple, as shown on the picture below. The client will get connected on HAProxy using SSL, HAProxy will process SSL and get connected in clear to the server:
ssl offloading diagram

HAproxy installation

cd /usr/src
wget http://haproxy.1wt.eu/download/1.5/src/snapshot/haproxy-ss-20120905.tar.gz
tar xzf haproxy-ss-20120905.tar.gz
cd haproxy-ss-20120905/
make TARGET=linux2628 USE_STATIC_PCRE=1 USE_OPENSSL=1
sudo make PREFIX=/opt/haproxy-ssl install

HAProxy configuration for SSL offloading


First of all, you have to generate a key and a certificate using openssl and concatenate them in a file, the certificate first, then the key.
Here is mine, just copy/paste it in a file for your tests:

-----BEGIN CERTIFICATE-----
MIIBrzCCARgCCQCfMsCGwq31yzANBgkqhkiG9w0BAQUFADAcMRowGAYDVQQDExF3
d3cuZXhjZWxpYW5jZS5mcjAeFw0xMjA5MDQwODU3MzNaFw0xMzA5MDQwODU3MzNa
MBwxGjAYBgNVBAMTEXd3dy5leGNlbGlhbmNlLmZyMIGfMA0GCSqGSIb3DQEBAQUA
A4GNADCBiQKBgQDFxSTUwX5RD4AL2Ya5t5PAaNjcwPa3Km40uaPKSHlU8AMydxC1
wB4L0k3Ms9uh98R+kIJS+TxdfDaYxk/GdDYI1CMm4TM+BLHGAVA2DeNf2hBhBRKb
TAgxCxXwORJQSB/B+1r0/ZiQ2ig5Jzr8xGHz+tBsHYZ+t+RmjZPQFjnlewIDAQAB
MA0GCSqGSIb3DQEBBQUAA4GBABqVuloGWHReSGLY1yAs20uhJ3j/9SvtoueyFBag
z5jX4BNO/4yhpKEpCGmzYtjr7us3v/s0mKoIVvAgah778rCZW3kF1Y6xR6TYqZna
1ryKB50/MJg9PC4LNL+sAu+WSslOf6+6Ru5N3JjhIZST8edJsGDi6/5HTKoqyvkp
wOMn
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
MIICXgIBAAKBgQDFxSTUwX5RD4AL2Ya5t5PAaNjcwPa3Km40uaPKSHlU8AMydxC1
wB4L0k3Ms9uh98R+kIJS+TxdfDaYxk/GdDYI1CMm4TM+BLHGAVA2DeNf2hBhBRKb
TAgxCxXwORJQSB/B+1r0/ZiQ2ig5Jzr8xGHz+tBsHYZ+t+RmjZPQFjnlewIDAQAB
AoGBALUeVhuuVLOB4X94qGSe1eZpXunUol2esy0AMhtIAi4iXJsz5Y69sgabg/qL
YQJVOZO7Xk8EyB7JaerB+z9BIFWbZwS9HirqR/sKjjbhu/rAQDgjVWw2Y9sjPhEr
CEAvqmQskT4mY+RW4qz2k8pe4HKq8NAFwbe8iNP7AySP3K4BAkEA4ZPBagtlJzrU
7Tw4BvQJhBmvNYEFviMScipHBlpwzfW+79xvZhTxtsSBHAM9KLbqO33VmJ3C/L/t
xukW8SO6ewJBAOBxU0TfS0EzcRQJ4sn78G6hTjjLwJM2q4xuSwLQDVaWwtXDI6HE
jb7HePaGBGnOrlXxEOFQZCVdDaLhX0zcEQECQQDHcvc+phioGRKPOAFp1HhdfsA2
FIBZX3U90DfAXFMFKFXMiyFMJxSZPyHQ/OQkjaaJN3eWW1c+Vw0MJKgOSkLlAkEA
h8xpqoFEgkXCxHIa00VpuzZEIt89PJVWhJhzMFd7yolbh4UTeRx4+xasHNUHtJFG
MF+0a+99OJIt3wBn7hQ1AQJACScT3p6zJ4llm59xTPeOYpSXyllR4GMilsGIRNzT
RGYxcvqR775RkAgE+5DHmAkswX7TBaxcO6+C1+LJEwFRxw==
-----END RSA PRIVATE KEY-----

Now, HAProxy configuration, very basic, for test purpose, and just to let you know which lines are very important:

frontend ft_test
  mode http
  bind 0.0.0.0:8443 ssl crt ./haproxy.pem  # basic conf require only 1 keyword
  # other (self described) options are: [ciphers <suite>] [nosslv3] [notlsv1]
  default_backend bk_test

backend bk_test
  mode http
  server srv1 127.0.0.1:80

Running HAProxy


First, just test the configuration is valid:

/opt/haproxy-ssl/sbin/haproxy -c -f ./ha.cfg 
[WARNING] 247/110924 (6497) : config : missing timeouts for frontend 'ft_test'.
   | While not properly invalid, you will certainly encounter various problems
   | with such a configuration. To fix this, please ensure that all following
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[WARNING] 247/110924 (6497) : config : missing timeouts for backend 'bk_test'.
   | While not properly invalid, you will certainly encounter various problems
   | with such a configuration. To fix this, please ensure that all following
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
Configuration file is valid

Don’t worry about warnings, I purposely wrote a very basic configuration.

Now, you can run HAProxy:

/opt/haproxy-ssl/sbin/haproxy  -f ./ha.cfg

Testing SSL provided by HAProxy


Use curl, with “–insecure” option if your certificate is self-signed, like mine:

curl --noproxy * -D - --insecure https://127.0.0.1:8443/index.html
HTTP/1.1 200 OK
Date: Tue, 04 Sep 2012 09:13:55 GMT
Server: Apache/2.2.16 (Debian)
Last-Modified: Tue, 04 Sep 2012 09:10:01 GMT
ETag: "a35d1-e-4c8dc9f7d6c40"
Accept-Ranges: bytes
Content-Length: 14
Vary: Accept-Encoding
Content-Type: text/html

Welcome page.

Check SSL parameters with openssl in client mode:

openssl s_client -connect 127.0.0.1:8443
CONNECTED(00000003)
depth=0 /CN=www.exceliance.fr
verify error:num=18:self signed certificate
verify return:1
depth=0 /CN=www.exceliance.fr
verify return:1
---
Certificate chain
 0 s:/CN=www.exceliance.fr
   i:/CN=www.exceliance.fr
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIBrzCCARgCCQCfMsCGwq31yzANBgkqhkiG9w0BAQUFADAcMRowGAYDVQQDExF3
d3cuZXhjZWxpYW5jZS5mcjAeFw0xMjA5MDQwODU3MzNaFw0xMzA5MDQwODU3MzNa
MBwxGjAYBgNVBAMTEXd3dy5leGNlbGlhbmNlLmZyMIGfMA0GCSqGSIb3DQEBAQUA
A4GNADCBiQKBgQDFxSTUwX5RD4AL2Ya5t5PAaNjcwPa3Km40uaPKSHlU8AMydxC1
wB4L0k3Ms9uh98R+kIJS+TxdfDaYxk/GdDYI1CMm4TM+BLHGAVA2DeNf2hBhBRKb
TAgxCxXwORJQSB/B+1r0/ZiQ2ig5Jzr8xGHz+tBsHYZ+t+RmjZPQFjnlewIDAQAB
MA0GCSqGSIb3DQEBBQUAA4GBABqVuloGWHReSGLY1yAs20uhJ3j/9SvtoueyFBag
z5jX4BNO/4yhpKEpCGmzYtjr7us3v/s0mKoIVvAgah778rCZW3kF1Y6xR6TYqZna
1ryKB50/MJg9PC4LNL+sAu+WSslOf6+6Ru5N3JjhIZST8edJsGDi6/5HTKoqyvkp
wOMn
-----END CERTIFICATE-----
subject=/CN=www.exceliance.fr
issuer=/CN=www.exceliance.fr
---
No client certificate CA names sent
---
SSL handshake has read 604 bytes and written 319 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES256-SHA
    Session-ID: CF9B7BFF64DE0B332CE9A76896EC1C59C941340D6913612286113FA1F7E09E88
    Session-ID-ctx: 
    Master-Key: C6893078E49626DAF329C61774BA5A35E0264818E0D76542F25BB958584B835154402E02F9B722DD94C56B14EBB14D46
    Key-Arg   : None
    Start Time: 1346750742
    Timeout   : 300 (sec)
    Verify return code: 18 (self signed certificate)
---
GET / HTTP/1.0

HTTP/1.1 200 OK
Date: Tue, 04 Sep 2012 09:26:44 GMT
Server: Apache/2.2.16 (Debian)
Last-Modified: Tue, 04 Sep 2012 09:10:01 GMT
ETag: "a35d1-e-4c8dc9f7d6c40"
Accept-Ranges: bytes
Content-Length: 14
Vary: Accept-Encoding
Connection: close
Content-Type: text/html

Welcome page.
closed

Related articles

Links

Enhanced SSL load-balancing with Server Name Indication (SNI) TLS extension

Synopsis

Some time ago, we wrote an article which explained how to load-balance SSL services, maintaining affinity using the SSLID.
The main limitation of this kind of architecture is that you must dedicate a public IP address and port per service.
If you’re hosting web or mail services, you could run out of public IP address quickly.

TLS protocol has been extended in 2003, RFC 3546, by an extension called SNI: Server Name Indication, which allows a client to announce in clear the server name it is contacting.

NOTE: two RFC have obsoleted the one above, the latest one is RFC 6066

The Aloha Load-balancer can use this information to choose a backend or a server.
This allows people to share a single VIP for several services.

Of course, we can use SNI switching with SSLID affinity to build a smart and reliable SSL load-balanced platform.

NOTE: Server Name information is sent with each SSL Handshake, whether you’re establishing a new session or you’re resuming an old one.

SNI is independent from the protocol used at layer 7. So basically, it will work with IMAP, HTTP, SMTP, POP, etc…

Limitation

Bear in mind, that in 2012, not all clients are compatible with SNI.
Concerning web browsers, a few of used in 2012 them are still not compatible with this TLS protocol extension.

We strongly recommend you to read the Wikipedia Server Name Indication page which lists all the limitation of this extension.

  • Only HAProxy nightly snapshots from 8th of April are compatible (with no bug knows) with it.
  • Concerning Aloha, it will be available by Aloha Load-balancer firmware 5.0.2.

Diagram

The picture below shows a platform with a single VIP which host services for 2 applications:
sni_loadbalancing

We can use SNI information to choose a backend, then, inside a backend, we can use SSLID affinity.

Configuration

Choose a backend using SNI TLS extension


The configuration below matches names provided by the SNI extention and choose a farm based on it.
In the farm, it provides SSLID affinity.
If no SNI extention is sent, then we redirect the user to a server farm which can be used to tell the user to upgrade its software.

# Adjust the timeout to your needs
defaults
  timeout client 30s
  timeout server 30s
  timeout connect 5s

# Single VIP with sni content switching
frontend ft_ssl_vip
  bind 10.0.0.10:443
  mode tcp

  tcp-request inspect-delay 5s
  tcp-request content accept if { req_ssl_hello_type 1 }
  
  acl application_1 req_ssl_sni -i application1.domain.com
  acl application_2 req_ssl_sni -i application2.domain.com

  use_backend bk_ssl_application_1 if application_1
  use_backend bk_ssl_application_2 if application_2

  default_backend bk_ssl_default

# Application 1 farm description
backend bk_ssl_application_1
  mode tcp
  balance roundrobin

  # maximum SSL session ID length is 32 bytes.
  stick-table type binary len 32 size 30k expire 30m

  acl clienthello req_ssl_hello_type 1
  acl serverhello rep_ssl_hello_type 2

  # use tcp content accepts to detects ssl client and server hello.
  tcp-request inspect-delay 5s
  tcp-request content accept if clienthello

  # no timeout on response inspect delay by default.
  tcp-response content accept if serverhello

  stick on payload_lv(43,1) if clienthello

  # Learn on response if server hello.
  stick store-response payload_lv(43,1) if serverhello

  option ssl-hello-chk
  server server1 192.168.1.1:443 check
  server server2 192.168.1.2:443 check

# Application 2 farm description
backend bk_ssl_application_2
  mode tcp
  balance roundrobin

  # maximum SSL session ID length is 32 bytes.
  stick-table type binary len 32 size 30k expire 30m

  acl clienthello req_ssl_hello_type 1
  acl serverhello rep_ssl_hello_type 2

  # use tcp content accepts to detects ssl client and server hello.
  tcp-request inspect-delay 5s
  tcp-request content accept if clienthello

  # no timeout on response inspect delay by default.
  tcp-response content accept if serverhello

  stick on payload_lv(43,1) if clienthello

  # Learn on response if server hello.
  stick store-response payload_lv(43,1) if serverhello

  option ssl-hello-chk
  server server1 192.168.2.1:443 check
  server server2 192.168.2.2:443 check

# Sorry backend which should invite the user to update its client
backend bk_ssl_default
  mode tcp
  balance roundrobin
  
  # maximum SSL session ID length is 32 bytes.
  stick-table type binary len 32 size 30k expire 30m

  acl clienthello req_ssl_hello_type 1
  acl serverhello rep_ssl_hello_type 2

  # use tcp content accepts to detects ssl client and server hello.
  tcp-request inspect-delay 5s
  tcp-request content accept if clienthello

  # no timeout on response inspect delay by default.
  tcp-response content accept if serverhello

  stick on payload_lv(43,1) if clienthello

  # Learn on response if server hello.
  stick store-response payload_lv(43,1) if serverhello

  option ssl-hello-chk
  server server1 10.0.0.11:443 check
  server server2 10.0.0.12:443 check

Choose a server using SNI: aka SSL routing


The configuration below matches names provided by the SNI extention and choose a server based on it.
If no SNI is provided or we can’t find the expected name, then the traffic is forwarded to server3 which can be used to tell the user to upgrade its software.

# Adjust the timeout to your needs
defaults
  timeout client 30s
  timeout server 30s
  timeout connect 5s

# Single VIP 
frontend ft_ssl_vip
  bind 10.0.0.10:443
  mode tcp

  tcp-request inspect-delay 5s
  tcp-request content accept if { req_ssl_hello_type 1 }

  default_backend bk_ssl_default

# Using SNI to take routing decision
backend bk_ssl_default
  mode tcp

  acl application_1 req_ssl_sni -i application1.domain.com
  acl application_2 req_ssl_sni -i application2.domain.com

  use-server server1 if application_1
  use-server server2 if application_2
  use-server server3 if !application_1 !application_2

  option ssl-hello-chk
  server server1 10.0.0.11:443 check
  server server2 10.0.0.12:443 check
  server server3 10.0.0.13:443 check

Related Links

Links

Scaling out SSL

Synopsis

We’ve seen recently how we could scale up SSL performance.
But what about scaling out SSL performance?
Well, thanks to Aloha and HAProxy, it’s easy to manage smartly a farm of SSL accelerator servers, using persistence based on the SSL Session ID.
This way of load-balancing is smart, but in case of SSL accelerator failure, other servers in the farm would have a CPU overhead to generate SSL Session IDs for sessions re-balanced by the Aloha.

After a talk with (the famous) emericbr, HAProxy Technologies dev team leader, he decided to write a patch for stud to add a new feature: sharing SSL session between different stud processes.
That way, in case of SSL accelerator failure, the servers getting re-balanced sessions would not have to generate a new SSL session.

Emericbr’s patch is available here: https://github.com/bumptech/stud/pull/50
At the end of this article, you’ll learn how to use it.

Stud SSL Session shared caching

Description

As we’ve seen in our article on SSL performance, a good way to improve performance on SSL is to use a SSL Session ID cache.

The idea here, is to use this cache as well as sending updates into a shared cache one can consult to get the SSL Session ID and the data associated to it.

As a consequence, there are 2 levels of cache:

      * Level 1: local process cache, with the currently used SSL session
      * Level 2: shared cache, with the SSL session from all local cache

Way of working

The protocol understand 3 types of request:

      * New: When a process generates a new session, it updates its local cache then the shared cache
      * Get: When a client tries to resume a session and the process receiving it is not aware of it, then the process tries to get it from the shared cache
      * Del: When a session has expired or there is a bad SSL ID sent by a client, then the process will delete the session from the shared cache

Who does what?

Stud has a Father/Son architecture.
The Father starts up then starts up Sons. The Sons bind the TCP external ports, load the certificate and process the SSL requests.
Each son manages its local cache and send the updates to the shared cache. The Father manages the shared cache, receiving the changes and maintaining it up to date.

How are the updates exchanged?

Updates are sent either on Unicast or Multicast, on a specified UDP port.
Updates are compatible both IPv4 and IPv6.
Each packet are signed by an encrypted signature using the SSL certificate, to avoid cache poisoning.

What does a packet look like?


SSL Session ID ASN-1 of SSL Session structure Timestamp Signature
[32 bytes] [max 512 bytes] [4 bytes] [20 bytes]

Note: the SSL Session ID field is padded with 0 if required

Diagram

Let’s show this on a nice picture where each potato represents each process memory area.
stud_shared_cache
Here, the son on host 1 got a new SSL connection to process, since he could not find it in its cache and in the shared cache, he generated the asymmetric key, then push it to his shared cache and the father on host 2 which updates the shared cache for this host.
That way, if this user is routed to any stud son process, he would not have to compute again its asymmetric key.

Let’s try Stud shared cache

Installation:

git clone https://github.com/EmericBr/stud.git
cd stud
wget http://1wt.eu/tools/ebtree/ebtree-6.0.6.tar.gz
tar xvzf ebtree-6.0.6.tar.gz
ln -s ebtree-6.0.6 ebtree
make USE_SHARED_CACHE=1

Generate a key and a certificate, add them in a single file.

Now you can run stud:

sudo ./stud -n 2 -C 10000 -U 10.0.3.20,8888 -P 10.0.0.17 -f 10.0.3.20,443 -b 10.0.0.3,80 cert.pem

and run a test:

curl --noproxy * --insecure -D - https://10.0.3.20:443/

And you can watch the synchronization packets:

$ sudo tcpdump -n -i any port 8888
[sudo] password for bassmann: 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes

17:47:10.557362 IP 10.0.3.20.8888 > 10.0.0.17.8888: UDP, length 176
17:49:04.592522 IP 10.0.3.20.8888 > 10.0.0.17.8888: UDP, length 176
17:49:05.476032 IP 10.0.3.20.8888 > 10.0.0.17.8888: UDP, length 176

Related links