Tag Archives: ssl

What’s new in HAProxy 1.6

[ANNOUNCE] HAProxy 1.6.0 released


Yesterday, 13th of October, Willy has announced the release of HAProxy 1.6.0, after 16 months of development!
First good news is that release cycle goes a bit faster and we aim to carry on making it as fast.
A total of 1156 commits from 59 people were committed since the release of 1.5.0, 16 months ago.

Please find here the official announce: [ANNOUNCE] haproxy-1.6.0 now released!.

In his mail, Willy detailed all the features that have been added to this release. The purpose of this blog article is to highlight a few of them, providing the benefits and some configuration examples.

NOTE: Most of the features below were already backported and integrated into our HAProxy Enterprise and ALOHA products.

HAProxy Enterprise is our Open Source version of HAProxy based on HAProxy community stable branch where we backport many features from dev branch and we package it to make the most stable, reliable, advanced and secured version of HAProxy. It also comes with third party software to fill the gap between a simple HAProxy process and a load-balancer (VRRP, syslog, SNMP, Route Health Injection, etc…). Cherry on the cake, we provide “enterprise” support on top of it.

NOTE 2: the list of new features introduced here is not exhaustive. Example are proposed in a quick and dirty way to teach you how to start with the feature. Don’t run those examples in production 🙂

It’s 2015, let’s use QUOTE in configuration file


Those who uses HAProxy for a long time will be happy to know that ‘\ ‘ (backslash-space) sequence is an old painful souvenir with 1.6 🙂
We can now write:

reqirep "^Host: www.(.*)" "Host: foobar\1"

or

option httpchk GET / "HTTP/1.1\r\nHost: www.domain.com\r\nConnection: close"

Lua Scripting


Maybe the biggest change that occurred is the integration of Lua.
Quote from Lua’s website: “Lua is a powerful, fast, lightweight, embeddable scripting language.“.

Basically, everyone has now the ability to extend HAProxy by writing and running their own Lua scripts. No need to write C code, maintain patches, etc…
If some Lua snipplets are very popular, we may write the equivalent feature in C and make it available in HAProxy mainline.

One of the biggest challenge Thierry faced when integrating Lua is to give it the ability to propose non-blocking processing of Lua code and non-blocking network socket management.

HAProxy requires Lua 5.3 or above.

With Lua, we can add new functions to the following HAProxy elements:

  • Service
  • Action
  • Sample-fetch
  • Converter

Compiling HAProxy and Lua


Installing LUA 5.3

cd /usr/src
curl -R -O http://www.lua.org/ftp/lua-5.3.0.tar.gz
tar zxf lua-5.3.0.tar.gz
cd lua-5.3.0
make linux
sudo make INSTALL_TOP=/opt/lua53 install

LUA 5.3 library and include files are now installed in /opt/lua53.

Compiling HAProxy with Lua support

make TARGET=linux2628 USE_OPENSSL=1 USE_PCRE=1 USE_LUA=1 LUA_LIB=/opt/lua53/lib/ LUA_INC=/opt/lua53/include/

HAProxy/Lua simple Hello world example


A simple Hello world! in Lua could be written like this:

  • The lua code in a file called hello_world.lua:
    core.register_service("hello_world", "tcp", function(applet)
       applet:send("hello world\n")
    end)
  • The haproxy configuration
    global
       lua-load hello_world.lua
    
    listen proxy
       bind 127.0.0.1:10001
       tcp-request content use-service lua.hello_world
    

More with Lua


Please read the doc, ask your questions on HAProxy‘s ML (no registration needed): haproxy@formilux.org.
Of course, we’ll write more articles later on this blog about Lua integration.

Captures


HAProxy‘s running context is very important when writing configuration. In HAProxy, each context is isolated. IE: you can’t use a request header when processing the response.
With HAProxy 1.6, this is now possible: you can declare capture slots, store data in it and use it at any time during a session.

defaults 
 mode http

frontend f_myapp
 bind :9001
 declare capture request len 32 # id=0 to store Host header
 declare capture request len 64 # id=1 to store User-Agent header
 http-request capture req.hdr(Host) id 0
 http-request capture req.hdr(User-Agent) id 1
 default_backend b_myapp

backend b_myapp
 http-response set-header Your-Host %[capture.req.hdr(0)]
 http-response set-header Your-User-Agent %[capture.req.hdr(1)]
 server s1 10.0.0.3:4444 check

Two new headers are inserted in the response:
Your-Host: 127.0.0.1:9001
Your-User-Agent: curl/7.44.0

Multiprocess, peers and stick-tables


In 1.5, we introduced “peers” to synchronize stick-table content between HAProxy servers. This feature was not compatible with multi-process mode.
We can now, in 1.6, synchronize content of the table it is stick to one process. It allows creating configurations for massive SSL processing pointing to a single backend sticked on a single process where we can use stick tables and synchronize its content.

peers article
 peer itchy 127.0.0.1:1023

global
 pidfile /tmp/haproxy.pid
 nbproc 3

defaults
 mode http

frontend f_scalessl
 bind-process 1,2
 bind :9001 ssl crt /home/bassmann/haproxy/ssl/server.pem
 default_backend bk_lo

backend bk_lo
 bind-process 1,2
 server f_myapp unix@/tmp/f_myapp send-proxy-v2

frontend f_myapp
 bind-process 3
 bind unix@/tmp/f_myapp accept-proxy
 default_backend b_myapp

backend b_myapp
 bind-process 3
 stick-table type ip size 10k peers article
 stick on src
 server s1 10.0.0.3:4444 check

Log


log-tag


It is now possible to position a syslog tag per process, frontend or backend. Purpose is to ease the job of syslog servers when classifying logs.
If no log-tag is provided, the default value is the program name.

Example applied to the configuration snipplet right above:

frontend f_scalessl
 log-tag SSL
[...]

frontend f_myapp
 log-tag CLEAR
[...]

New log format variables


New log format variables have appeared:

  • %HM: HTTP method (ex: POST)
  • %HP: HTTP request URI without query string (path)
  • %HQ: HTTP request URI query string (ex: ?bar=baz)
  • %HU: HTTP request URI (ex: /foo?bar=baz)
  • %HV: HTTP version (ex: HTTP/1.0)

Server IP resolution using DNS at runtime


In 1.5 and before, HAProxy performed DNS resolution when parsing configuration, in a synchronous mode and using the glibc (hence /etc/resolv.conf file).
Now, HAProxy can perform DNS resolution at runtime, in an asynchronous way and update server IP on the fly. This is very convenient in environment like Docker or Amazon Web Service where server IPs can be changed at any time.
Configuration example applied to docker. A dnsmasq is used as an interface between /etc/hosts file (where docker stores server IPs) and HAProxy:

resolvers docker
 nameserver dnsmasq 127.0.0.1:53

defaults
 mode http
 log global
 option httplog

frontend f_myapp
 bind :80
 default_backend b_myapp

backend b_myapp
 server s1 nginx1:80 check resolvers docker resolve-prefer ipv4

Then, let’s restart s1 with the command “docker restart nginx1” and let’s have a look at the magic in the logs:
(...) haproxy[15]: b_myapp/nginx1 changed its IP from 172.16.0.4 to 172.16.0.6 by docker/dnsmasq.

HTTP rules


New HTTP rules have appeared:

  • http-request: capture, set-method, set-uri, set-map, set-var, track-scX, sc-in-gpc0, sc-inc-gpt0, silent-drop
  • http-response: capture, set-map, set-var, sc-inc-gpc0, sc-set-gpt0, silent-drop, redirect

Variables


We often used HTTP header fields to store temporary data in HAProxy. With 1.6, we can now define variables.
A variable is available for a scope: session, transaction (request or response), request, response.
A variable name is prefixed by its scope (sess, txn, req, res), a dot ‘.’ and a tag composed by ‘a-z’, ‘A-Z’, ‘0-9’ and ‘_’.

Let’s rewrite the capture example using variables

global
 # variables memory consumption, in bytes
 tune.vars.global-max-size 1048576
 tune.vars.reqres-max-size     512
 tune.vars.sess-max-size      2048
 tune.vars.txn-max-size        256

defaults
 mode http

frontend f_myapp
 bind :9001
 http-request set-var(txn.host) req.hdr(Host)
 http-request set-var(txn.ua) req.hdr(User-Agent)
 default_backend b_myapp

backend b_myapp
 http-response set-header Your-Host %[var(txn.host)]
 http-response set-header Your-User-Agent %[var(txn.ua)]
 server s1 10.0.0.3:4444 check

mailers


Now, HAProxy can send emails when server states change (mainly goes DOWN), so your sysadmins/devos won’t sleep anymore :). It used to be able to log only before.

mailers mymailers
 mailer smtp1 192.168.0.1:587 
 mailer smtp2 192.168.0.2:587 
 
backend mybackend 
 mode tcp 
 balance roundrobin
 email-alert mailers mymailers
 email-alert from test1@horms.org
 email-alert to test2@horms.org
 server srv1 192.168.0.30:80
 server srv2 192.168.0.31:80

Processing of HTTP request body


Until 1.5 included, HAProxy could process only HTTP request headers. It can now access request body.
Simply enable the statement below in your frontend or backend to give HAProxy this ability:

option http-buffer-request

Protection against slow-POST attacks


slow-POST attacks are like the slowlorys one, except that HTTP header are sent quicly, but request body are sent very slowly.
Once enabled, the timeout http-request parameters also apply to the POSTED data.

Fetch methods


A few new fetch methods now exists to play with the body: req.body, req.body_param, req.body_len, req.body_size, etc…

Short example to detect “SELECT *” string in a request POST body, and of course how to deny it:

defaults
 mode http

frontend f_mywaf
 bind :9001
 option http-buffer-request
 http-request deny if { req.body -m reg "SELECT \*" }
 default_backend b_myapp

backend b_myapp
 server s1 10.0.0.3:4444 check

New converters


1.5 introduced the converters but only a very few of them were available.
1.6 adds many of them. The list is too long, but let’s give the most important ones: json, in_table, field, reg_sub, table_* (to access counters from stick-tables), etc…

Device Identification


Through our company, we have some customer who want us to integrate into HAProxy the ability to detect device type and characteristics and report it to the backend server.
We got a couple of contributions from 2 companies experts in this domain: 51 degrees and deviceatlas.
You can now load those libraries in HAProxy in order to fully qualify a client capabilities and set up some headers your application server can rely on to adapt content delivered to the client or let the varnish cache server use it to cache multiple flavor of the same object based on client capabilities.

More on this blog later on how to integrate each product.

Seamless server states


Prior 1.6, when being reloaded, HAProxy considers all the servers are UP until the first check is performed.
Since 1.6, we can dump server states into a flat file right before performing the reload and let the new process know where the states are stored. That way, the old and new processes owns exactly the same server states (hence seamless).
The following information are reported:

  • server IP address when resolved by DNS
  • operational state (UP/DOWN/…)
  • administrative state (MAINT/DRAIN/…)
  • Weight (including slowstart relative weight)
  • health check status
  • rise / fall current counter
  • check state (ENABLED/PAUSED/…)
  • agent check state (ENABLED/PAUSED/…)

The state could be applied globally (all server found) or per backend.
Example:
Simple HAProxy configuration:

global
 stats socket /tmp/socket
 server-state-file /tmp/server_state

backend bk
 load-server-state-from-file global
 server s1 10.0.0.3:4444 check weight 11
 server s2 10.0.0.4:4444 check weight 12

Before reloading HAProxy, we save the server states using the following command:

socat /tmp/socket - <<< "show servers state" > /tmp/server_state

Here is the content of /tmp/server_state file:

1
# <field names skipped for the blog article>
1 bk 1 s1 10.0.0.3 2 0 11 11 4 6 3 4 6 0 0
1 bk 2 s2 10.0.0.4 2 0 12 12 4 6 3 4 6 0 0

Now, let’s proceed with reload as usual.
Of course, the best option is to export the server states using the init script.

External check


HAProxy can run a script to perform complicated health checks.
Just be aware about the security concerns when enabling this feature!!

Configuration:

global
 external-check

backend b_myapp
 external-check path "/usr/bin:/bin"
 external-check command /bin/true
 server s1 10.0.0.3:4444 check

TLS / SSL


NOTE: some of the features introduced here may need a recent openssl library.

Detection of ECDSA-able clients


This has already been documented by Nenad on this blog: Serving ECC and RSA certificates on same IP with HAProxy

SSL certificate forgery on the fly


Since 1.6, HAProxy can forge SSL certificate on the fly!
Yes, you can use HAProxy with your company’s CA to inspect content.

Support of Certificate Transparency (RFC6962) TLS extension


When loading PEM files, HAProxy also checks for the presence of file at the same path suffixed by “.sctl”. If such file is found, support for Certificate Transparency (RFC6962) TLS extension is enabled.
The file must contain a valid Signed Certificate Timestamp List, as described in RFC. File is parsed to check basic syntax, but no signatures are verified.

TLS Tickets key load through stats socket


A new stats socket command is available to update TLS ticket keys at runtime. The new key is used for encryption/decryption while the old ones are used for decryption only.

Server side SNI


Many application servers now takes benefit from Server Name Indication TLS extension.
Example:

backend b_myapp_ssl
 mode http
 server s1 10.0.0.3:4444 check ssl sni req.hdr(Host)

Peers v2


The peers protocol, used to synchronized data from stick-tables between two HAProxy servers can now synchronise more than just the sample and the server-id.
It can synchronize all the tracked counters. Note that each node push its local counter to a peer. So this must be used for safe reload and server failover only.
Don’t expect to see 10 HAProxy servers to sync and aggregate counters in real time.

That said, this protocol has been extended to support different data type, so we may see more features soon relying on it 😉

HTTP connection sharing


On the road to HTTP/2, HAProxy must be able to support connection pooling.
And on the road to the connection pools, we have the ability to share a server side connection between multiple clients.
The server side connection may be used by multiple clients, until the owner (the client side connection) of this session dies. Then new connections may be established.

The new keyword is http-reuse and have different level of sharing connections:

  • never: no connections are shared
  • safe: first request of each client is sent over its own connection, subsequent request may used an other connection. It works like a regular HTTP keepalive
  • aggressive: send request to connections that has proved reliably support connection reuse (no quick connection close after a response has been sent)
  • always: send request to established connections, whatever happens. If the server was closing the connection in the mean time, the request is lost and the client must resend it.

Get rid of 408 in logs


Simply use the new option

option http-ignore-probes

.

Serving ECC and RSA certificates on same IP with HAproxy

ECC and RSA certificates and HTTPS

To keep this practical, we will not go into theory of ECC or RSA certificates. Let’s just mention that ECC certificates can provide as much security as RSA with much lower key size, meaning much lower computation requirements on the server side. Sadly, many clients do not support ciphers based on ECC, so to maintain compatibility as well as provide good performance we need to be able to detect which type of certificate is supported by the client to be able to serve it correctly.

The above is usually achieved with analyzing the cipher suites sent by the client in the ClientHello message at the start of the SSL handshake, but we’ve opted for a much simpler approach that works very well with all modern browsers (clients).

Prerequisites

First you will need to obtain both RSA and ECC certificates for your web site. Depending on the registrar you are using, check their documentation. After you have been issued with the certificates, make sure you download the appropriate intermediate certificates and create the bundle files for HAproxy to read.

To be able to use the sample fetch required, you will need at least HAproxy 1.6-dev3 (not yet released as of writing) or you can clone latest HAproxy from the git repository. Feature was introduced in commit 5fc7d7e.

Configuration

We will use chaining in order to achieve desired functionality. You can use abstract sockets on Linux to get even more performance, but note the drawbacks that can be found in HAproxy documentation.

 frontend ssl-relay
 mode tcp
 bind 0.0.0.0:443
 use_backend ssl-ecc if { req.ssl_ec_ext 1 }
 default_backend ssl-rsa

 backend ssl-ecc
 mode tcp
 server ecc unix@/var/run/haproxy_ssl_ecc.sock send-proxy-v2

 backend ssl-rsa
 mode tcp
 server rsa unix@/var/run/haproxy_ssl_rsa.sock send-proxy-v2

 listen all-ssl
 bind unix@/var/run/haproxy_ssl_ecc.sock accept-proxy ssl crt /usr/local/haproxy/ecc.www.foo.com.pem user nobody
 bind unix@/var/run/haproxy_ssl_rsa.sock accept-proxy ssl crt /usr/local/haproxy/www.foo.com.pem user nobody
 mode http
 server backend_1 192.168.1.1:8000 check

The whole configuration revolves around the newly implemented sample fetch: req.ssl_ec_ext. What this fetch does is that it detects the presence of Supported Elliptic Curves Extension inside the ClientHello message. This extension is defined in RFC4492 and according to it, it SHOULD be sent with every ClientHello message by the client supporting ECC. We have observed that all modern clients send it correctly.

If the extension is detected, the client is sent through a unix socket to the frontend that will serve an ECC certificate. If not, a regular RSA certificate will be served.

Benchmark

We will provide full HAproxy benchmarks in the near future, but for the sake of comparison, let us view the difference present on an E5-2680v3 CPU and OpenSSL 1.0.2.

256bit ECDSA:
sign verify sign/s verify/s
0.0000s 0.0001s 24453.3 9866.9

2048bit RSA:
sign verify sign/s verify/s
0.000682s 0.000028s 1466.4 35225.1

As you can see, looking at the sign/s we are getting over 15 times the performance with ECDSA256 compared to RSA2048.

HAProxy and HTTP Strict Transport Security (HSTS) header in HTTP redirects

SSL/TLS and HSTS

SSL everywhere is on its way.
Unfortunately, many applications were written for HTTP only and switching to HTTPs is not an easy and straight forward path. Read more here about impact of TLS offloading (when a third party tool perform TLS in front of your web application servers).

A mechanism called HTTP Strict Transport Security (HSTS) has been introduced through the RFC 6797.

HSTS main purpose is to let the application server to instruct the client it’s supposed to get connected only a ciphered and secured HTTPs connection when browsing the application.
It means that of course, that both the client and the server must be compatible…
That way, the application cookie is protected on its way from the client’s browser to the remote TLS endpoint (either the load-balancer or the application server). No cookie hijacking is possible on the wire.

HAProxy configuration for Strict-Transport-Security HTTP header

HSTS header insertion in server responses

To insert the header in every server response, you can use the following HAProxy directive, in HAProxy 1.5:

 # 16000000 seconds: a bit more than 6 months
 http-response set-header Strict-Transport-Security max-age=16000000;\ includeSubDomains;\ preload;

With the upcoming HAProxy 1.6, and thanks to William’s work, we can now get rid of these ugly backslashes:

 # 16000000 seconds: a bit more than 6 months
 http-response set-header Strict-Transport-Security "max-age=16000000; includeSubDomains; preload;"

Inserting HSTS header in HTTP redirects


When HAProxy has to perform HTTP redirects, it does in at the moment of the client request, through the http-request rules.
Since we want to insert a header in the response, we can use the http-response rules. Unfortunately, these rules are enabled when HAProxy get traffic from a backend server.
Here is the trick: we do perform the http-request redirect rule in a dedicated frontend where we route traffic to. That way, our application backend or frontend can perform HSTS insertion.

A simple configuration sniplet is usually easier to explain:

frontend fe_myapp
 bind :443 ssl crt /path/to/my/cert.pem
 bind :80
 use_backend be_dummy if !{ ssl_fc }
 default_backend be_myapp

backend be_myapp
 http-response set-header Strict-Transport-Security max-age=16000000;\ includeSubDomains;\ preload;
 server s1 10.0.0.1:80

be_dummy
 server haproxy_fe_dummy_ssl_redirect 127.0.0.1:8000

frontend fe_dummy
 bind 127.0.0.1:8000
 http-request redirect scheme https

Links

HAProxy and sslv3 poodle vulnerability

SSLv3 poodle vulnerability

Yesterday, Google security researchers have disclosed a new vulnerability on SSL protocol.
Fortunately, this vulnerability is only on an old version of the SSL protocol: SSLv3 (15 years old protocol).
An attacker can force a browser to downgrade the protocol version used to cipher traffic to SSLv3 in order to exploit the POODLE vulnerability and access to data in clear.

Some reading about SSLv3 Poodle vulnerability:
* http://googleonlinesecurity.blogspot.fr/2014/10/this-poodle-bites-exploiting-ssl-30.html
* https://www.imperialviolet.org/2014/10/14/poodle.html
* https://www.poodletest.com/

Today’s article is going to explain how to use HAProxy to simply prevent using SSLv3 or to prevent those users to reach your applications and print them a message.

Disable SSLv3 in HAProxy

In SSL offloading mode

In this mode, HAProxy is the SSL endpoint of the connection.
It’s a simple keyword on the frontend bind directive:

  bind 10.0.0.1:443 ssl crt /pat/to/cert.pem no-sslv3

In SSL forward mode


In this mode, HAProxy forwards the SSL traffic to the server without deciphering it.
We must setup an ACL to match the SSL protocol version, then we can refuse the connection. This must be added in a **frontend** section:

  bind 10.0.0.1:443
  tcp-request inspect-delay 2s
  acl sslv3 req.ssl_ver 3
  tcp-request content reject if sslv3

Communicate a message to users

Denying sslv3 is a good way, but a better one would to educate as well users who are using this protocol.
The configuration below shows how to redirect a user to a specific page when they want to use your application over an SSLv3 connection. Of course, HAProxy must allow itself SSLv3:

frontend ft_www
  bind 10.0.0.1:443 ssl crt /pat/to/cert.pem
  acl sslv3 ssl_fc_protocol SSLv3
# first rule after all your 'http-request deny' and
# before all the redirect, rewrite, etc....
  http-request allow if sslv3
[...]
# first content switching rule
  use_backend bk_sslv3 if sslv3

backend bk_sslv3
  mode http
  errorfile 503 /etc/haproxy/pages/sslv3.http

And the content of the file /etc/haproxy/pages/sslv3.http:

HTTP/1.0 200 OK
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<head>
<title>SSLv3 spotted</title>
</head>
<body><h1>SSLv3 spotted</h1></body>
SSLv3 forbidden for your safety:<BR>
http://googleonlinesecurity.blogspot.fr/2014/10/this-poodle-bites-exploiting-ssl-30.html<BR>
<BR>
If you want to browse this website, you should upgrade your browser.
</html>

Links

Configuring HAProxy and Nginx for SPDY

Introduction to SPDY / HTTP-bis

SPDY is a protocol designed by google which aims to fix HTTP/1.1 protocol weaknesses and to adapt this 14 years old protocol to today’s internet devices and requirements.
Back in 1999, when HTTP/1.1 was designed, there was no mobile devices, the web pages were composed by HTML with a few images, almost no javascript and no CSS. The ISP delivered internet over very slow connections.
HTTP/2.0 has to address today’s and tomorrow’s need when much more devices can browse the internet from very different type of connections (very slow with high packet loss or very fast ones) and more and more people want multimedia content and interactivity. Websites became “web applications”.

SPDY is not HTTP 2.0! But HTTP 2.0 will use SPDY as basement

Note that as long as HTTP/2.0 has not been released officially, ALL articles written on SPDY, NPN, ALPN, etc may be outdated at some point.
Sadly, this is true for the present article, BUT I’ll try to keep it up to date 🙂
As an example, NPN is a TLS extension that has been designed to allow a client and a server to negotiate the protocol which will be used at the above network layer (in our case this is application layer).
Well, this NPN TLS extension is going to be outdated soon by an official RFC called “Transport Layer Security (TLS) Application Layer Protocol Negotiation Extension” (shortened to ALPN).

Also, we’re already at the third version of SPDY protocol (3.1 to be accurate). It changes quite often.

HAProxy and SPDY

As Willy explained on HAProxy‘s mailing list, HAProxy won’t implement SPDY.
But, as soon as HTTP/2.0 will be released officially, then the development will focus on it.
Main driver for this is the problem seen in introduction: waste of time because the drafts are updated quite often.

Saying that, HAProxy can be used to:
* load-balance SPDY web servers
* detect SPDY protocol through NPN (or more recent ALPN) TLS extension

Diagram

The picture below shows how HAProxy can be used to detect and split HTTP and SPDY traffic over an HTTPs connection:
spdy

Basically, HAProxy uses the NPN (and later the ALPN) TLS extension to figure out whether the client can browse the website using SPDY. If yes, the connection is forwarded to the SPDY farm (here hosted on nginx), otherwise, the connection is forwarded to the HTTP server farm (here hosted on nginx too).

Configuration

HAProxy configuration example for NPN and SPDY


Below, the HAProxy configuration to detect and split HTTP and SPDY traffic to two different farms:

defaults
 mode tcp
 log global
 option tcplog
 timeout connect           4s
 timeout server          300s
 timeout client          300s

frontend ft_spdy
 bind 64.210.140.163:443 name https ssl crt STAR_haproxylab_net.pem npn spdy/2

# tcp log format + SSL information (TLS version, cipher in use, SNI, NPN)
 log-format %ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq {%sslv/%sslc/%[ssl_fc_sni]/%[ssl_fc_npn]}

# acls: npn
 acl npn_spdy           ssl_fc_npn -i spdy/2

# spdy redirection
 use_backend bk_spdy      if npn_spdy

 default_backend bk_http

backend bk_spdy
 option httpchk HEAD /healthcheck
 server nginx 127.0.0.1:8082 maxconn 100 check port 8081

backend bk_http
 option httpchk HEAD /healthcheck
 http-request set-header Spdy no
 server nginx 127.0.0.1:8081 maxconn 100 check

NGINX configuration example for SPDY


And the corresponding nginx configuration:
Note that I use a LUA script to check if the “Spdy: no” HTTP header is present.
  * if present: then the connection was made over HTTP in HAProxy
  * if not present, then the connection was made over SPDY

   server {
      listen 127.0.0.1:8081;
      listen 127.0.0.1:8082 spdy;
      root /var/www/htdocs/spdy/;

        location = /healthcheck {
          access_log off;
          return 200;
        }

        location / {
             default_type 'text/plain';
             content_by_lua '
               local c = ngx.req.get_headers()[&quot;Spdy&quot;];
               if c then
                 ngx.say(&quot;Currently browsing over HTTP&quot;)
               else
                 ngx.say(&quot;Currently browsing over spdy&quot;)
               end
                return
             ';
        }

   }

Testing


This setup is in production.
Simply browse https://spdy.haproxylab.net/ to give it a try.

Related links

  • SPDY whitepaper: http://www.chromium.org/spdy/spdy-whitepaper
  • SPDY (official ???) page: http://www.chromium.org/spdy
  • IETF Transport Layer Security (TLS) Application Layer Protocol Negotiation Extension: http://tools.ietf.org/html/draft-friedl-tls-applayerprotoneg-02
  • IETF HTTP/1.1 RFC: http://www.ietf.org/rfc/rfc2616.txt

Links

SSL Client certificate information in HTTP headers and logs

HAProxy and SSL

HAProxy has many nice features when speaking about SSL, despite SSL has been introduced in it lately.

One of those features is the client side certificate management, which has already been discussed on the blog.
One thing was missing in the article, since HAProxy did not have the feature when I first write the article. It is the capability of inserting client certificate information in HTTP headers and reporting them as well in the log line.

Fortunately, the devs at HAProxy Technologies keep on improving HAProxy and it is now available (well, for some time now, but I did not have any time to write the article yet).

OpenSSL commands to generate SSL certificates

Well, just take the script from HAProxy Technologies github, follow the instruction and you’ll have an environment setup in a very few seconds.
Here is the script: https://github.com/exceliance/haproxy/tree/master/blog/ssl_client_certificate_management_at_application_level

Configuration

The configuration below shows a frontend and a backend with SSL offloading and with insertion of client certificate information into HTTP headers. As you can see, this is pretty straight forward.

frontend ft_www
 bind 127.0.0.1:8080 name http
 bind 127.0.0.1:8081 name https ssl crt ./server.pem ca-file ./ca.crt verify required
 log-format %ci:%cp [%t] %ft %b/%s %Tq/%Tw/%Tc/%Tr/%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs {%[ssl_c_verify],%{+Q}[ssl_c_s_dn],%{+Q}[ssl_c_i_dn]} %{+Q}r
 http-request set-header X-SSL                  %[ssl_fc]
 http-request set-header X-SSL-Client-Verify    %[ssl_c_verify]
 http-request set-header X-SSL-Client-DN        %{+Q}[ssl_c_s_dn]
 http-request set-header X-SSL-Client-CN        %{+Q}[ssl_c_s_dn(cn)]
 http-request set-header X-SSL-Issuer           %{+Q}[ssl_c_i_dn]
 http-request set-header X-SSL-Client-NotBefore %{+Q}[ssl_c_notbefore]
 http-request set-header X-SSL-Client-NotAfter  %{+Q}[ssl_c_notafter]
 default_backend bk_www

backend bk_www
 cookie SRVID insert nocache
 server server1 127.0.0.1:8088 maxconn 1

To observe the result, I just fake a server using netcat and observe the headers sent by HAProxy:

X-SSL: 1
X-SSL-Client-Verify: 0
X-SSL-Client-DN: "/C=FR/ST=Ile de France/L=Jouy en Josas/O=haproxy.com/CN=client1/emailAddress=ba@haproxy.com"
X-SSL-Client-CN: "client1"
X-SSL-Issuer: "/C=FR/ST=Ile de France/L=Jouy en Josas/O=haproxy.com/CN=haproxy.com/emailAddress=ba@haproxy.com"
X-SSL-Client-NotBefore: "130613144555Z"
X-SSL-Client-NotAfter: "140613144555Z"

And the associated log line which has been generated:

Jun 13 18:09:49 localhost haproxy[32385]: 127.0.0.1:38849 [13/Jun/2013:18:09:45.277] ft_www~ bk_www/server1 
1643/0/1/-1/4645 504 194 - - sHNN 0/0/0/0/0 0/0 
{0,"/C=FR/ST=Ile de France/L=Jouy en Josas/O=haproxy.com/CN=client1/emailAddress=ba@haproxy.com",
"/C=FR/ST=Ile de France/L=Jouy en Josas/O=haproxy.com/CN=haproxy.com/emailAddress=ba@haproxy.com"} "GET /" 

NOTE: I have inserted a few CRLF to make it easily readable.

Now, my HAProxy can deliver the following information to my web server:
  * ssl_fc: did the client used a secured connection (1) or not (0)
  * ssl_c_verify: the status code of the TLS/SSL client connection
  * ssl_c_s_dn: returns the full Distinguished Name of the certificate presented by the client
  * ssl_c_s_dn(cn): same as above, but extracts only the Common Name
  * ssl_c_i_dn: full distinguished name of the issuer of the certificate presented by the client
  * ssl_c_notbefore: start date presented by the client as a formatted string YYMMDDhhmmss
  * ssl_c_notafter: end date presented by the client as a formatted string YYMMDDhhmmss

Related Links

Links

SSL offloading impact on web applications

SSL Offloading

Nowadays, it is common (and convenient) to use the Load-Balancer SSL capabilities to cypher/uncypher traffic from clients to the web application platform.
Performing SSL at the Load-Balancer Layer is called “SSL offloading“, because you offload this process from your application servers.

Note that SSL offloading is also “marketingly” called SSL acceleration. The only acceleration performed is moving SSL processing from application server, which have an other heavy task to perform, to an other devices in the network…
So no acceleration at all, despite some people claming they do accelerate SSL in a dedicated ASIC, they just offload

The diagram below shows what happens on an architecture with SSL offloading. We can clearly see that the traffic is encrypted up to the Load-Balancer, then it’s in clear between the Load-Balancer and the application server:
ssl offloading diagram

Benefits of SSL offloading


Offloading SSL from the application servers to the Load-Balancers have many benefits:
  * offload an heavy task from your application servers, letting them to focus on the application itself
  * save resources on your application servers
  * the Load-Balancers have access to clear HTTP traffic and can perform advanced features such as reverse-proxying, Cookie persistence, traffic regulation, etc…
  * When using an ALOHA Load-Balancer (or HAProxy), there are much more features available on the SSL stack than on any web application server.

It seems there should only be positive impact when offloading SSL.

Counterparts of SSL offloading


As we saw on the diagram above, the load-balancer will hide three important information from the client connection:
  1. protocol scheme: the client was using HTTPS while the connection on the application server is made over HTTP
  2. connection type: was it cyphered or not? Actually, this is linked to the protocol scheme
  3. destination port: on the load-balancer, the port is usually 443 and on the application server should be 80 (this may change)

Most web application will use 301/302 responses with a Location header to redirect the client to a page (most of the time after a login or a POST) as well as requiring an application cookie.

So basically, the worst impact is that your application server may have difficulties to know the client connection information and may not be able to perform right responses: it can break totally the application, preventing it to work.
Fortunately, the ALOHA Load-Balancer or HAProxy can help in such situation!

Basically, the application could return such responses to the client:

Location: http://www.domain.com/login.jsp

And the client will leave the secured connection…

Tracking issues with the load-balancer


In order to know if your application supports well SSL offloading, you can configure your ALOHA Load-Balancer to log some server responses.
Add the configuration below in your application frontend section:

capture response header Location   len 32
capture response header Set-Cookie len 32

Now, in the traffic log, you’ll see clearly if the application is setting up response for HTTP or HTTPS.

NOTE: some other headers may be needed, depending on your application.

Web Applications and SSL offloading


Here we are 🙂

There are 3 answers possible for such situation, detailed below.

Client connection information provided in HTTP header by the Load-Balancer


First of all, the Load-Balancer can provide client side connection information to the application server through HTTP header.
The configuration below shows how to insert a header called X-Forwarded-Proto containing the scheme used by the client.
To be added in your backend section.

http-request set-header X-Forwarded-Proto https if  { ssl_fc }
http-request set-header X-Forwarded-Proto http  if !{ ssl_fc }

Now, the ALOHA Load-Balancer will insert the following header when the connection is made over SSL:

X-Forwarded-Proto: https

and when performed over clear HTTP:

X-Forwarded-Proto: http

It’s up to your application to use this information to answer the right responses: this may require some code update.

HTTP header rewriting on the fly


An other way is to rewrite the response setup by the server on the fly.
The following configuration line would match the Location header and translate it from http to https if needed.
Rewriting rspirep http://www.domain.com:80/url:

rspirep ^Location:\ http://(.*):80(.*)  Location:\ https://\1:443\2   if  { ssl_fc }

NOTE: above example applies on HTTP redirection (Location header), but can be applied on Set-cookie as well (or any other header your application may use).
NOTE2: you can only rewrite the response HTTP headers, not in the body. So this is not compatible with applications setting up hard links in the HTML content.

SSL Bridging


In some cases, the application is not compatible at all with SSL offloading (even with the tricks above) and we must use a ciphered connection to the server but we still may require to perform cookie based persistence, content switching, etc…
This is called SSL bridging, or can also be called a man in the middle.

In the ALOHA, there is nothing to do, just do SSL offloading as usual and add the keyword ssl on your server directive.

Using this method, you can choose a light cipher and a light key between the Load-Balancer and the server and still use the Load-Balancer advanced SSL feature with a stronger key and a stronger cipher.
The application server won’t notice anything and the Load-Balancer can still perform Layer 7 processing.

Links