Tag Archives: reverse-proxy

HAProxy and HTTP errors 408 in Chrome

Lately, there was some discussions on HAProxy’s mailing list about 408 errors printed in Chrome browsers.

Origin of 408 errors


408 is the status code used by web servers or proxies when the client has not sent a whole HTTP request during a certain period of time. It is an absolutely normal behavior!

That said, modern browsers have now some advanced features like “tcp pre-connect”…
As its name designate it, a browser may open tcp connection to web servers before you want to browse them. Purpose is to speed up browsing for the client by saving the time to establish these connections.
The problem, is that in most of the case, these connections won’t be used.
The other problem a very popular website can maintain thousands (or even tens or hundred of thousands) of connections that might be used.
It has an expensive cost on the server side for a so small positive effect on the client side.

When a web server or a proxy accept an incoming connection, in expects a request to come in the next few seconds. If a full request has not arrived after some time, the server or the proxy can close the connection and report a 408 error to the client, to tell him “you were to slow to send me a request on this connection, so I closed it”.
Unfortunately, some browsers, when they want to start using the connection mentioned above, see that data is already available for reading and print it to the user without even checking the connection state (server closing it) neither the response status code (408 in this case).

HAProxy’s timeout related to 408 errors


In HAProxy’s configuration, there are two parameters that may trigger a 408 error:

  timeout http-request 10s
  timeout client 20s

The timeout http-request is the time you let to a client to send it’s request.
If it is not set, then the timeout client will be used.

If both of them are configured, the shortest value is used.

In a default configuration, when answering with a 408, HAProxy sends a message such as this one:

HTTP/1.0 408 Request Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html><body><h1>408 Request Time-out</h1>
Your browser didn't send a complete request in time.
</body></html>

This message is printed by some browsers.

Workaround in HAProxy


One way to workaround this issue with HAProxy is to tell HAProxy to silently close the connection without sending any data.
You can add the line below in your defaults (or frontend/listen) section:

 errorfile 408 /dev/null

Bear in mind this is just a workaround and not a definitive solution

Related Links

Links

Howto write apache ProxyPass* rules in HAProxy

Apache mod_proxy

Apache webserver is a widely deployed modular web server.
One of its module is called mod_proxy. It aims to turn the web server into a proxy / reverse proxy server with load-balancing capabilities.

At HAProxy Technologies, we only use HAProxy :). Heh, what else ???
And during some deployments, customers ask us to migrate Apache mod_proxy configuration into HAProxy.
Present article explains how to translate ProxyPass related rules.

ProxyPass, ProxyPassReverse, etc…

Apache mod_proxy defines a few directives which let it forward traffic to a remote server.
They are listed below with a short description.

Note: I know the directives introduced in this article could be used in much complicated ways.

ProxyPass

ProxyPass maps local server URLs to remote servers + URL.
It applies on traffic from client to server.

For example:

ProxyPass /mirror/foo/ http://backend.example.com/

This makes the external URL http://example.com/mirror/foo/bar to be translated and forwarded to a remote server this way: http://backend.example.com/bar

This directive makes apache to update URL and headers to match both external traffic to internal needs.

ProxyPassReverse

ProxyPassReverse Adjusts the URL in HTTP response headers sent from a reverse proxied server. It only updates Location, Content-Location and URL.
It applies to traffic from server to client.

For example:

ProxyPassReverse /mirror/foo/ http://backend.example.com/

This directive makes apache to adapt responses generated by servers following internal urls to match external urls.

ProxyPassReverseCookieDomain

ProxyPassReverseCookieDomain adjusts the Set-Cookie header sent by the server to match external domain name.

It’s usage is pretty simple. For example:

ProxyPassReverseCookieDomain internal-domain public-domain

ProxyPassReverseCookiePath

ProxyPassReverseCookiePath adjusts the Set-Cookie header sent by the server to match external path.

It’s usage is pretty simple. For example:

ProxyPassReverseCookiePath internal-path public-path

Configure ProxyPass and ProxyPassReverse in HAProxy

Bellow, an example HAProxy configuration to make HAProxy work the same way as apache ProxyPass and ProxyPassReverse configuration. It should be added in the backend section while the frontend ensure that only traffic matching this external URL would be redirected to that backend.

frontend ft_global
 acl host_dom.com    req.hdr(Host) dom.com
 acl path_mirror_foo path -m beg   /mirror/foo/
 use_backend bk_myapp if host_dom.com path_mirror_foo

backend bk_myapp
[...]
# external URL                  =&gt; internal URL
# http://dom.com/mirror/foo/bar =&gt; http://bk.dom.com/bar

 # ProxyPass /mirror/foo/ http://bk.dom.com/bar
 http-request set-header Host bk.dom.com
 reqirep  ^([^ :]*)\ /mirror/foo/(.*)     \1\ /\2

 # ProxyPassReverse /mirror/foo/ http://bk.dom.com/bar
 # Note: we turn the urls into absolute in the mean time
 acl hdr_location res.hdr(Location) -m found
 rspirep ^Location:\ (https?://bk.dom.com(:[0-9]+)?)?(/.*) Location:\ /mirror/foo3 if hdr_location

 # ProxyPassReverseCookieDomain bk.dom.com dom.com
 acl hdr_set_cookie_dom res.hdr(Set-cookie) -m sub Domain= bk.dom.com
 rspirep ^(Set-Cookie:.*)\ Domain=bk.dom.com(.*) \1\ Domain=dom.com\2 if hdr_set_cookie_dom

 # ProxyPassReverseCookieDomain / /mirror/foo/
 acl hdr_set_cookie_path res.hdr(Set-cookie) -m sub Path= 
 rspirep ^(Set-Cookie:.*)\ Path=(.*) \1\ Path=/mirror/foo2 if hdr_set_cookie_path

Notes:
  * http to https redirect rules should be handled by HAProxy itself and not by the application server (to avoid some redirect loops)

Links

Asymmetric routing, multiple default gateways on Linux with HAProxy

Why we may need multiple default gateways?

Nowadays, Application Delivery controllers (aka Load-Balancers) become the entry point for all the applications hosted in a company or administration.
That said, many different type of population could access the applications:
  * internal users from the LAN
  * partners through MPLS or VPNs
  * external users from internet

On the other side, applications could be hosted on different VLANs in the architecture:
  * internal LAN
  * external DMZ

The diagram below shows the “big picture” of this type of architecture:
multiple_default_gateways

Routing in the Linux network stack

I’m not going to deeply explain how it works, sorry… It would deserve a complete blog post ūüôā
That said, any device connected on an IP network needs an IP address to be able to talk to other devices in its LAN. It also needs a default gateway to be able to reach devices which are located outside its LAN.
A Linux kernel can use a single default gateway at a time, but thanks to the metric you can configure many default gateways.
When needed, the Linux Kernel will parse the default gateway table and will use the one with the lowest metric. By default, when no metric is configured, the kernel attributes a metric 0.
Each metric must be unique in your Kernel IP stack.

How HAProxy can help in such situation??


Users access applications through a HAProxy bind. The bind can be hosted on any IP address available or not (play with your sysctl for this purpose) on the server.
By default, the traffic comes in HAProxy through this bind and HAProxy let the kernel choose the most appropriate default gateway to forward the answer to the client. As we’ve seen above, the most appropriate default gateway from the kernel point of view is the one with the lowest metric usually 0.

That said, HAProxy is smart enough to tell the kernel which network interface to use to forward the response to the client. Just add the statement interface ethX (where X is the id of the interface you want to use) on HAProxy bind line.
With this parameter, HAProxy can force the kernel to use the default gateway associated to the network interface ethX if it exists, otherwise, the interface with the lowest metric will be used.

Security concern


From a security point of view, some security manager would say that it is absolutely unsecure to plug a device in multiple DMZ or VLANs. They are right. But usually, this type of company’s business is very important and they can affoard one load-balancer per DMZ or LAN.
That said, there is no security breach with the setup introduced here. HAProxy is a reverse-proxy and so you don’t need to allow ip_forward between all interfaces for this solution to work.
I mean that nobody could use the loadbalancer as a default gateway to reach an other subnet bypassing the firewall…
Then only traffic allowed to pass through is the one load-balanced!

Configuration

The configuration below applies to the ALOHA Loadbalancer. Just update the content to match your Linux distribution configuration syntax.
The configuration is also related to the diagram above.

Network configuration


In your ALOHA, go in the Services tab, then edit the Network configuration.
To keep it simple, I’m not going to add any VRRP configuration.

service network eth0
    ########## eth0.
    auto on
    ip   address 10.0.0.2/24
    ip   route   default 10.0.0.1

service network eth1
    ########## eth1.
    auto on
    ip   address 10.0.1.2/24
    ip   route   default 10.0.1.1 metric 1

service network eth2
    ########## eth2.
    auto on
    ip   address 10.0.2.2/24
    ip   route   default 10.0.2.1 metric 2

service network eth3
    ########## eth3.
    auto on
    ip   address 10.0.3.2/24
    ip   route   default 10.0.3.1 metric 3

service network eth4
    ########## eth4.
    auto on
    ip   address 10.0.4.2/24
    ip   route   default 10.0.4.1 metric 4

The routing table from the ALOHA looks like:

default via 10.0.0.1 dev eth0
default via 10.0.1.1 dev eth1  metric 1
default via 10.0.2.1 dev eth2  metric 2
default via 10.0.3.1 dev eth3  metric 3
default via 10.0.4.1 dev eth4  metric 4

HAProxy configuration for Corporate website or ADFS proxies


These services are used by internet users only.

frontend ft_www
 bind 10.0.0.2:80
[...]

no need to specify any interface here, since the traffic comes from internet, HAProxy can let the kernel to use the default gateway which points in that direction (here eth0).

HAProxy configuration for Exchange 2010 or 2013


This service is used by both internal and internet users.

frontend ft_exchange
 bind 10.0.0.3:443
 bind 10.0.2.3:443 interface eth2
[...]

The responses to internet users will go through eth0 while the one for internal LAN users will use the default gateway configured on eth2 10.0.2.1.

HAProxy configuration for Sharepoint 2010 or 2013


This service is used by MPLS/VPN users and internal users.

frontend ft_exchange
 bind 10.0.1.4:443 interface eth1
 bind 10.0.2.4:443 interface eth2
[...]

The responses to MPLS/VPN users will go through eth1 default gateway 10.0.1.1 while the one for internal LAN users will use the default gateway configured on eth2 10.0.2.1.

Links

Howto transparent proxying and binding with HAProxy and ALOHA Load-Balancer

Transparent Proxy

HAProxy works has a reverse-proxy. Which means it maintains 2 connections when allowing a client to cross it:
  – 1 connection between HAProxy and the client
  – 1 connection between HAProxy and the server

HAProxy then manipulate buffers between these two connections.
One of the drawback of this mode is that HAProxy will let the kernel to establish the connection to the server. The kernel is going to use a local IP address to do this.
Because of this, HAProxy “hides” the client IP by its own one: this can be an issue in some cases.
Here comes the transparent proxy mode: HAProxy can be configured to spoof the client IP address when establishing the TCP connection to the server. That way, the server thinks the connection comes from the client directly (of course, the server must answer back to HAProxy and not to the client, otherwise it can’t work: the client will get an acknowledge from the server IP while it has established the connection on HAProxy‘s IP).

Transparent binding

By default, when one want HAProxy to get traffic, we have to tell it to bind an IP address and a port.
The IP address must exist on the operating system (unless you have setup the sysctl net.ipv4.ip_nonlocal_bind) and the OS must announce the availability to the other devices on the network through ARP protocol.
Well, in some cases we want HAProxy to be able to catch traffic on the fly without configuring any IP address or VRRP or whatever…
This is where transparent binding comes in: HAProxy can be configured to catch traffic on the fly even if the destination IP address is not configured on the server.
These IP addresses will never be pingable, but they’ll deliver the services configured in HAProxy.

HAProxy and the Linux Kernel

Unfortunately, HAProxy can’t do transparent binding or proxying alone. It must stand on a compiled and tuned Linux Kernel and operating system.
Below, I’ll explain how to do this in a standard Linux distribution.
Here is the check list to meet:
  1. appropriate HAProxy compilation option
  2. appropriate Linux Kernel compilation option
  3. sysctl settings
  4. iptables rules
  5. ip route rules
  6. HAProxy configuration

HAProxy compilation requirements


First of all, HAProxy must be compiled with the option TPROXY enabled.
It is enabled by default when you use the target LINUX26 or LINUX2628.

Linux Kernel requirements

You have to ensure your kernel has been compiled with the following options:
  – CONFIG_NETFILTER_TPROXY
  – CONFIG_NETFILTER_XT_TARGET_TPROXY

Of course, iptables must be enabled as well in your kernel ūüôā

sysctl settings

The following sysctls must be enabled:
  – net.ipv4.ip_forward
  – net.ipv4.ip_nonlocal_bind

iptables rules


You must setup the following iptables rules:

iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT

Purpose is to mark packets which matches a socket bound locally (by HAProxy).

IP route rules


Then, tell the Operating System to forward packets marked by iptables to the loopback where HAProxy can catch them:

ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

HAProxy configuration


Finally, you can configure HAProxy.
  * Transparent binding can be configured like this:

[...]
frontend ft_application
  bind 1.1.1.1:80 transparent
[...]

  * Transparent proxying can be configured like this:

[...]
backend bk_application
  source 0.0.0.0 usesrc clientip
[...]

Transparent mode in the ALOHA Load-Balancer


Now, the same steps in the ALOHA Load-balancer, which is an HAProxy based load-balancing appliance:
  1-5. not required, the ALOHA kernel is deeply tuned for this purpose
  6. HAProxy configuration

LB Admin tab (AKA click mode)

  * Transparent binding can be configured like this, when editing a Frontend listener:
frontend_listener_transparent

  * Transparent proxying can be configured like this when editing a farm:
backend_transparent

LB Layer 7 tab (vi in a browser mode)


  * Transparent binding can be configured like this:

[...]
frontend ft_application
  bind 1.1.1.1:80 transparent
[...]

  * Transparent proxying can be configured like this:

[...]
backend bk_application
  source 0.0.0.0 usesrc clientip
[...]

Links

SSL Client certificate information in HTTP headers and logs

HAProxy and SSL

HAProxy has many nice features when speaking about SSL, despite SSL has been introduced in it lately.

One of those features is the client side certificate management, which has already been discussed on the blog.
One thing was missing in the article, since HAProxy did not have the feature when I first write the article. It is the capability of inserting client certificate information in HTTP headers and reporting them as well in the log line.

Fortunately, the devs at HAProxy Technologies keep on improving HAProxy and it is now available (well, for some time now, but I did not have any time to write the article yet).

OpenSSL commands to generate SSL certificates

Well, just take the script from HAProxy Technologies github, follow the instruction and you’ll have an environment setup in a very few seconds.
Here is the script: https://github.com/exceliance/haproxy/tree/master/blog/ssl_client_certificate_management_at_application_level

Configuration

The configuration below shows a frontend and a backend with SSL offloading and with insertion of client certificate information into HTTP headers. As you can see, this is pretty straight forward.

frontend ft_www
 bind 127.0.0.1:8080 name http
 bind 127.0.0.1:8081 name https ssl crt ./server.pem ca-file ./ca.crt verify required
 log-format %ci:%cp [%t] %ft %b/%s %Tq/%Tw/%Tc/%Tr/%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs {%[ssl_c_verify],%{+Q}[ssl_c_s_dn],%{+Q}[ssl_c_i_dn]} %{+Q}r
 http-request set-header X-SSL                  %[ssl_fc]
 http-request set-header X-SSL-Client-Verify    %[ssl_c_verify]
 http-request set-header X-SSL-Client-DN        %{+Q}[ssl_c_s_dn]
 http-request set-header X-SSL-Client-CN        %{+Q}[ssl_c_s_dn(cn)]
 http-request set-header X-SSL-Issuer           %{+Q}[ssl_c_i_dn]
 http-request set-header X-SSL-Client-NotBefore %{+Q}[ssl_c_notbefore]
 http-request set-header X-SSL-Client-NotAfter  %{+Q}[ssl_c_notafter]
 default_backend bk_www

backend bk_www
 cookie SRVID insert nocache
 server server1 127.0.0.1:8088 maxconn 1

To observe the result, I just fake a server using netcat and observe the headers sent by HAProxy:

X-SSL: 1
X-SSL-Client-Verify: 0
X-SSL-Client-DN: "/C=FR/ST=Ile de France/L=Jouy en Josas/O=haproxy.com/CN=client1/emailAddress=ba@haproxy.com"
X-SSL-Client-CN: "client1"
X-SSL-Issuer: "/C=FR/ST=Ile de France/L=Jouy en Josas/O=haproxy.com/CN=haproxy.com/emailAddress=ba@haproxy.com"
X-SSL-Client-NotBefore: "130613144555Z"
X-SSL-Client-NotAfter: "140613144555Z"

And the associated log line which has been generated:

Jun 13 18:09:49 localhost haproxy[32385]: 127.0.0.1:38849 [13/Jun/2013:18:09:45.277] ft_www~ bk_www/server1 
1643/0/1/-1/4645 504 194 - - sHNN 0/0/0/0/0 0/0 
{0,"/C=FR/ST=Ile de France/L=Jouy en Josas/O=haproxy.com/CN=client1/emailAddress=ba@haproxy.com",
"/C=FR/ST=Ile de France/L=Jouy en Josas/O=haproxy.com/CN=haproxy.com/emailAddress=ba@haproxy.com"} "GET /" 

NOTE: I have inserted a few CRLF to make it easily readable.

Now, my HAProxy can deliver the following information to my web server:
  * ssl_fc: did the client used a secured connection (1) or not (0)
  * ssl_c_verify: the status code of the TLS/SSL client connection
  * ssl_c_s_dn: returns the full Distinguished Name of the certificate presented by the client
  * ssl_c_s_dn(cn): same as above, but extracts only the Common Name
  * ssl_c_i_dn: full distinguished name of the issuer of the certificate presented by the client
  * ssl_c_notbefore: start date presented by the client as a formatted string YYMMDDhhmmss
  * ssl_c_notafter: end date presented by the client as a formatted string YYMMDDhhmmss

Related Links

Links

IIS 6.0 appsession cookie and PCI compliance

Synopsis

You’re using HAProxy or the ALOHA Load-Balancer to load-balance IIS 6.0 web applications and you want them to pass successfully PCI compliance test.
One of the pre-requisite is to force the cookie to be “HttpOnly”, in order to tell the browser to use this cookie for HTTP requests only, and “protect” it from local javascript access (to steal session information).
Unfortunately, II 6.0 is not able to setup such cookies. That’s why HAProxy can be used to update the cookie on the fly, when setup by the application server.

Rewriting appsession Cookie with HAProxy

Place the configuration line below in your backend configuration:

rspirep ^Set-Cookie: (appsession.*)    Set-Cookie: 1; HttpOnly

Now, you’re application is “more” secured… Well, at least, you can successfully pass the PCI compliancy tests!

Links

SSL offloading impact on web applications

SSL Offloading

Nowadays, it is common (and convenient) to use the Load-Balancer SSL capabilities to cypher/uncypher traffic from clients to the web application platform.
Performing SSL at the Load-Balancer Layer is called “SSL offloading“, because you offload this process from your application servers.

Note that SSL offloading is also “marketingly” called SSL acceleration. The only acceleration performed is moving SSL processing from application server, which have an other heavy task to perform, to an other devices in the network…
So no acceleration at all, despite some people claming they do accelerate SSL in a dedicated ASIC, they just offload

The diagram below shows what happens on an architecture with SSL offloading. We can clearly see that the traffic is encrypted up to the Load-Balancer, then it’s in clear between the Load-Balancer and the application server:
ssl offloading diagram

Benefits of SSL offloading


Offloading SSL from the application servers to the Load-Balancers have many benefits:
  * offload an heavy task from your application servers, letting them to focus on the application itself
  * save resources on your application servers
  * the Load-Balancers have access to clear HTTP traffic and can perform advanced features such as reverse-proxying, Cookie persistence, traffic regulation, etc…
  * When using an ALOHA Load-Balancer (or HAProxy), there are much more features available on the SSL stack than on any web application server.

It seems there should only be positive impact when offloading SSL.

Counterparts of SSL offloading


As we saw on the diagram above, the load-balancer will hide three important information from the client connection:
  1. protocol scheme: the client was using HTTPS while the connection on the application server is made over HTTP
  2. connection type: was it cyphered or not? Actually, this is linked to the protocol scheme
  3. destination port: on the load-balancer, the port is usually 443 and on the application server should be 80 (this may change)

Most web application will use 301/302 responses with a Location header to redirect the client to a page (most of the time after a login or a POST) as well as requiring an application cookie.

So basically, the worst impact is that your application server may have difficulties to know the client connection information and may not be able to perform right responses: it can break totally the application, preventing it to work.
Fortunately, the ALOHA Load-Balancer or HAProxy can help in such situation!

Basically, the application could return such responses to the client:

Location: http://www.domain.com/login.jsp

And the client will leave the secured connection…

Tracking issues with the load-balancer


In order to know if your application supports well SSL offloading, you can configure your ALOHA Load-Balancer to log some server responses.
Add the configuration below in your application frontend section:

capture response header Location   len 32
capture response header Set-Cookie len 32

Now, in the traffic log, you’ll see clearly if the application is setting up response for HTTP or HTTPS.

NOTE: some other headers may be needed, depending on your application.

Web Applications and SSL offloading


Here we are ūüôā

There are 3 answers possible for such situation, detailed below.

Client connection information provided in HTTP header by the Load-Balancer


First of all, the Load-Balancer can provide client side connection information to the application server through HTTP header.
The configuration below shows how to insert a header called X-Forwarded-Proto containing the scheme used by the client.
To be added in your backend section.

http-request set-header X-Forwarded-Proto https if  { ssl_fc }
http-request set-header X-Forwarded-Proto http  if !{ ssl_fc }

Now, the ALOHA Load-Balancer will insert the following header when the connection is made over SSL:

X-Forwarded-Proto: https

and when performed over clear HTTP:

X-Forwarded-Proto: http

It’s up to your application to use this information to answer the right responses: this may require some code update.

HTTP header rewriting on the fly


An other way is to rewrite the response setup by the server on the fly.
The following configuration line would match the Location header and translate it from http to https if needed.
Rewriting rspirep http://www.domain.com:80/url:

rspirep ^Location:\ http://(.*):80(.*)  Location:\ https://\1:443\2   if  { ssl_fc }

NOTE: above example applies on HTTP redirection (Location header), but can be applied on Set-cookie as well (or any other header your application may use).
NOTE2: you can only rewrite the response HTTP headers, not in the body. So this is not compatible with applications setting up hard links in the HTML content.

SSL Bridging


In some cases, the application is not compatible at all with SSL offloading (even with the tricks above) and we must use a ciphered connection to the server but we still may require to perform cookie based persistence, content switching, etc…
This is called SSL bridging, or can also be called a man in the middle.

In the ALOHA, there is nothing to do, just do SSL offloading as usual and add the keyword ssl on your server directive.

Using this method, you can choose a light cipher and a light key between the Load-Balancer and the server and still use the Load-Balancer advanced SSL feature with a stronger key and a stronger cipher.
The application server won’t notice anything and the Load-Balancer can still perform Layer 7 processing.

Links