Category Archives: Uncategorized

Welcome Tim!

Folks,

I’m realizing that it’s been a while since last post. We’ve been quite busy working on tons of cool stuff that you’ll discover soon and time flies fast.. very fast in fact. In the mean time our team was reinforced with Tim Hinds who will help me animate the blog and communicate on more general subjects than just bits and bytes, so I now have great hopes that many of our regular readers will more naturally share some links without the fear of looking like an alien who reads haproxy directives all the day 🙂

Welcome Tim, it’s really nice to have you on board, and stay tuned!

 

HAProxy Technologies offers free hardware load balancers to students / interns

How to start with load balancers ?

All of us playing with load balancers started the same way. First you prepare a configuration to alternate between two web servers returning different contents, and you verify with your browser that you indeed get a different response on each reload. This is the most basic load balancing setup one can imagine (and the worst one at the same time). Then you start all these “what if” questions. “What if a server dies”. You manage to address this by enabling health checks., but you feel like you’re cheating when you stop the server by killing the process. “What if now the cable is pulled off while transfering data”. This becomes difficult, you need to start a VM and you’re not completely sure you model the proper failure by stopping the VM, maybe the hypervisor will send some resets or ICMP messages.  Then “What if I the load balancer itself  fails”. Cheating with VMs that are started and stopped by hand doesn’t make you feel comfortable in that what you’re doing will work in real production. “What if someone by mistake configures two servers with the same IP address”. “What if I enable VRRP to cover hardware failures and my checks are wrong and I end up with two masters”. “What if I provoke a multicast loop”. “What if I setup multiple VRRP instances in multiple VLANs”.

All of these questions may sound very advanced to newcomers but are in fact quite common. In the world of load balancing, you definitely need to imagine a lot of failure cases and how to deal with them. Playing with pure software helps understand the basic concepts. Playing with VMs helps going a little bit further and already becomes quite painful. Add VLANs to the mix, asymmetric routing, DSR, transparent proxying and you’re quickly screwed.

Most of us started by plugging and pulling off cables , both network and power supply. I personally tortured a lot of Alteon AD3 in 2000-2001, but I was extremely lucky to have access to a lab where there were plenty of them (and they were really great devices to learn load balancing by the way).

But since it remains the best way to learn the concepts and to develop skills, HAProxy Technologies wanted to offer this opportunity to beginners again.

What if you could set up a complete load balancing lab on your desk, at school or at home  ?

load balancing on your desk
Complete Load Balancing platform on a desk

The photo above shows a complete load balancing platform deployed on a desk, with real machines and a real switch. And it’s just an example of what can be achieved. In order to make this possible, we ported our flagship product, the ALOHA load balancer, to a miniature MIPS-based platform which forms the ALOHA Pocket. This platform is designed and built by maker GL-Inet and is originally a WiFi router. But it has quite decent specs (2 FastEthernet ports, a 400 MHz 32-bit CPU, 64 MB of RAM, 16 MB of flash, and powered over USB), which are quite sufficient to run an ALOHA, is very convenient to manipulate, is rock solid and it is affordable. So it’s perfect to mass-produce an ALOHA that we can send around the world to interested participants. For now we have enough in stock to cover several tens of projects, we may order new ones if the stock goes away too quickly.

A box of 50 ALOHAs!
A box of 50 ALOHA Pocket Load balancers!

With this ALOHA Pocket, our goal is very simple : we want to introduce more people to load balancing., because we believe the future is there (load balancing, application delivery, content switching, function chaining, call it as you want, all these concepts are tightly coupled).  Thus we are willing to offer a couple of load balancers for free to any student or intern who can describe a project they are working on that involves load balancing, in exchange for their promise to regularly publish their progress, on a blog for example. We’re not going to verify who writes and when (though we welcome links), we bet that most participants will be honnest and will respect their engagement. We estimate that the more people who show what can be achieved using a load balancer, the more people will be attracted and will fall into that addiction in turn. It is even possible that some participants will face some bugs or will suggest improvements, we’re definitely willing to hear about this as well. Don’t be ashamed, suggest and criticize on your articles, we’re not asking people to send us flowers, just to be open! And maybe your comments will make some companies notice your skills and propose you a job after you finish your classes 🙂

This ALOHA Pocket is full-featured. It runs at quite a decent speed (around 450 connections per second with all features turned on, this is more than the vast majority of web sites). We could not put our anti-DDoS protection, PacketShield, in it because it requires more memory than this device contains. But we don’t consider that this is important for beginners. The devices will be shipped with the last version of our ALOHA platform, 8.0. We will intentionally not provide frequent software updates because we want to be sure that they will be used for educational purposes only and not to run production! But we’ll issue updates if users are facing bugs because we want them to learn in comfortable conditions.

Now, if you are interested, please send an email to contact at haproxy dot com with the subject “ALOHA Pocket”. Introduce yourself, what you intend to do (eg: load balance Apache/Nginx web servers, load balance Postfix mail servers, set up a cloud platform involving haproxy,  design a CDN involving HAProxy and Varnish, etc). Please provide enough details so that we may advise you if we detect some well-known traps in what you’re describing. Please indicate in what context you’re going to do this (at school, at university, for a company), and where you’re going to publish your progress, We don’t ask you to publicly disclose on your blog the name of the organization you’re working for, we know that some large ones still have problems with this. Indicate an estimated time frame for your project, and of course your shipping address. Try to be descriptive, as we consider that people not willing to write a few lines to get two free load balancers do not even deserve a response. By the way, if you believe you already have a compatible hardware and you’d only need the software image and procedure to flash it, contact us as well, we’ll welcome your demand as it means more people will be able to get one.

For logistics reasons, we’re going to send them in batches, so they will take a bit of time to arrive. Please bear with us, we’re handling all this by hand and installing all of them ourselves, just because we believe that this project is cool.

Let’s hope you’ll have as much fun using it as we had creating it 🙂

First HAProxy workshop at Zenika Paris was a success

HAProxy gets more and more contributors. That’s a good thing. There’s a side effect to this, which is that the maintainer (myself) spends quite some time reviewing submissions. I wanted to have the opportunity to exchange with various contributors to give them more autonomy, to present how haproxy works internally, how it’s maintained, what things are acceptable and which ones are not, and more generally to get their feedback as contributors.

Since I had never done this before, I didn’t want to force people to come from far away in case it would be a failure, so I wanted to contact only local contributors for this first round and that we talked french so that everyone would be totally at ease. A few of them couldn’t attend but no less than 8 people responded present! Given that our meeting room in Jouy-en-josas is too small for such a team, we started to consult a few partners. Zenika was kind enough to respond immediately (phone call in the evening, 3 proposals the next morning, who can beat that ?).

So Baptiste, Emeric, William, Thierry, Cyril, Christopher, Emmanuel and I met there in one of Zenika’s training rooms in Paris last Friday. The place was obviously much better than our meeting room, large, fully equipped, silent, and we could spend the whole day there chatting and presenting stuff.

I talked a lot. I’m always said to talk a lot anyway, so I guess nobody was surprized. I presented the overall internal architecture. It was not in great details, but I know the attendees are skilled enough to find their way through the code with these few entry points. What matters to me is that they know where to start from. Emeric talked a bit about the peers protocol. Cyril proposed that the HTML version of the doc be integrated into the official web site instead of as an external link. Then Christopher presented the filters, how they work, the choices he had to make. William explained some limitations he faced with the current design and there was a discussion on the best ways to overcome them. In short, some hooks need to be added to the filters, and proabably an analyzer mask as well. Then Thierry talked about various stuff such as lunch, Lua, lunch, maps, lunch, stats and how he intends to try to exploit the possibilities offered by the new filters. He also talked about lunch. He explained how he managed to implement some inter-process stats aggregation in Lua, which may deserve a rewrite in C.

It was also interesting to discuss the opportunity to use filters to develop the small stupid RAM-based cache that has been present in the roadmap for a few years (the “favicon cache” as I often call it). Thierry explained his first attempt at doing such a thing in Lua and the shortcomings he faced in part due to the Lua implementation and in part due to the uselessness of such a cache which ignored the Vary header. Also he complained about the limits he reached with such a permissive language when it comes to refactoring some existing code.

Emmanuel explained that for his use case (haproxy serves as an SSL offloader in front of Varnish), even a small object cache would bring very limited benefit and that he would probably not use it this way as he prefers to use it in plain TCP mode and deal with HTTP at a single place. He was suggested to run a test with HTTP multiplexing enabled between haproxy and Varnish (possible since 1.6) to estimate any possible performance gains compared to raw TCP. Emmanuel also discussed the possibility of exporting some histogram information for some metrics (eg: response sizes and times).

The question about how haproxy should make better use of the information it receives from the PROXY protocol header surfaced again, especially regarding SSL this time. It turns out that we almost froze the protocol some time ago and that everyone implemented it as it is specified, while haproxy skips the SSL parts. Something probably needs to be done, how is a different story.

The issue of external library dependencies was brought, such as Lua 5.3 and SLZ, which are not packaged in mainstream distros. There wasn’t a broad adoption of the principle of including them in the source tree, but rather to see them packaged and shipped by distros even if that’s in unofficial repos.

I explained how I intend to chain two layers of streams belonging to the same session with a protocol converter in the middle to implement HTTP/2 to HTTP/1 gatewaying, and some of the issues that will come from doing this.

We also discussed about what is still missing to go multithread. In short, still a lot but good practices are already mandatory if we want to make our life easier in the future.

Interestingly, for most users there, there are almost no more local patches except the usual few things that need to bake a bit before being submitted upstream. This is another proof that we need to make the code even easier to deal with for newcomers, to encourage users to develop their own code and submit it once they feel at ease with it.

Well, at the end of the day everyone seemed very satisfied and expressed interest for doing this again if possible at the same place (the place is nice, easily accessible and people were really nice with us).

We learned quite a bit for next rounds. First, everyone must participate and it seems that 10 persons is the maximum for a workshop. We need to make pauses as well. Next time we do it, we’ll have to be better organized (though everyone was good at improvising). We should prepare some rough presentations and ensure everyone has enough time to present their stuff. It’s also possible that we’d need a first part with everyone and a second part cut into small groups by centers of interests.

So thanks again to Zenika for helping us set this up, thanks to all participants for coming, now looking forward to doing this again with more people.

Microsoft Remote Desktop Services (RDS) Load-Balancing

Microsoft Remote Desktop services (RDS)

Remote Desktop Services, formerly Terminal Services, is a technology from Microsoft that allows users to access remotely to a session-based desktop, virtual machine-based desktop or applications hosted in a datacenter from their corporate network or from the internet.

Multiple RDS servers can be used in a farm. Hence we need to balance the load against them.
To achieve this purpose, we have different ways:
* using a connection broker
* using a load-balancer with the connection broker
* using a load-balancer without the connection broker

Of course, our load-balancer of choice is HAProxy!
In this blog article, we’re going to focus only on the case where a load-balancer is used.

The main issue when load-balancing multiple Remote Desktop Services servers is to ensure a user the continuity of his session in case of a network outage.

Current article will focus on session high availability for an optimal end user experience.

HAProxy with a connection broker

The connection broker, formerly Session broker, main purpose is to reconnect a user to his existing session. Since Windows 2008, the connection broker also have some load-balancing mechanism.

So, why using a load-balancer if the connection broker can do load-balance?


Answer is simple: security. Since HAProxy is a Reverse-Proxy, it breaks the TCP connection between the client and the server. HAProxy can be deployed in DMZ to give access to users coming from internet to a RDS farm deployed in the VLAN dedicated to servers.

HAProxy configuration


Note: this configuration works for the ALOHA 6.0 and above and HAPEE (HAProxy Enterprise Edition) 1.5 and above.

frontend ft_rdp
  mode tcp
  bind 192.168.13.128:3389 name rdp
  timeout client 1h
  log global
  option tcplog
  tcp-request inspect-delay 2s
  tcp-request content accept if RDP_COOKIE
  default_backend bk_rdp

backend bk_rdp
  mode tcp
  balance leastconn
  persist rdp-cookie
  timeout server 1h
  timeout connect 4s
  log global
  option tcplog
  option tcp-check
  tcp-check connect port 3389 ssl
  default-server inter 3s rise 2 fall 3
  server srv01 192.168.13.13:3389 weight 10 check
  server srv02 192.168.13.14:3389 weight 10 check

HAProxy without a connection broker

HAProxy can be used on its own to perform session load-balancing and resumption. For this purpose, it needs a stick-table where the user-server association is stored.
A peers section is added to the configuration. So we can share session persistence information between a cluster of ALOHAs or HAPEE servers.

peers aloha
 peer aloha1 192.168.13.1:1023
 peer aloha2 192.168.13.2:1023

frontend ft_rdp
  mode tcp
  bind 192.168.13.128:3389 name rdp
  timeout client 1h
  log global
  option tcplog
  tcp-request inspect-delay 2s
  tcp-request content accept if RDP_COOKIE
  default_backend bk_rdp

backend bk_rdp
  mode tcp
  balance leastconn
  timeout server 1h
  timeout connect 4s
  log global
  option tcplog
  stick-table type string len 32 size 10k expire 8h peers aloha
  stick on rdp_cookie(mstshash)
  option tcp-check
  tcp-check connect port 3389 ssl
  default-server inter 3s rise 2 fall 3
  server srv01 192.168.13.13:3389 weight 10 check
  server srv02 192.168.13.14:3389 weight 10 check

To know the user-server association, we can simply read the content of the stick-table:

echo show table bk_rdp | socat /var/run/haproxy.stat -
# table: bk_rdp, type: string, size:10240, used:5
0x21c7eac: key=Administrator use=0 exp=83332288 server_id=1
0x21c7eac: key=test-001 use=0 exp=83332288 server_id=2

We can easily read the login used by the user, the expiation date (in milliseconds) and the server ID used for the session.

Links

failover and worst case management with HAProxy

Synopsis

One of HAProxy strength is that it is flexible and allows to redirect traffic based on events and internal status.
In the current article, I’ll show how HAProxy can be useful to handle traffic when worst cases happen.

By worst case I mean the moment when something went wrong in your architecture and your application because partially or totally unavailable.

Cases

Backup server

When all servers in a farm are down, we want to redirect traffic to a backup server which delivers either sorry pages or a degraded mode of the application.
This can be done easily in HAProxy by adding the keyword backup on the server line. If multiple backup servers are configured, only the first active one is used.

Below, the HAProxy configuration corresponding to this case:

frontent ft_app
 bind 10.0.0.1:80
 default_backend bk_app_main

backend bk_app_main
 server s1 10.0.0.101:80 check
 server s2 10.0.0.102:80 check
 server s3 10.0.0.103:80 check backup
 server s4 10.0.0.104:80 check backup

In this case, s3 will be used first, until it fails, then s4 will be used.

Multiple backup servers

In some cases, when the farm takes a huge traffic, we may want to use many backup servers at a time. This can be achieved by enabling the option allbackups in HAProxy configuration.

Below, the HAProxy configuration corresponding to this case:

frontent ft_app
 bind 10.0.0.1:80
 default_backend bk_app_main

backend bk_app_main
 option allbackups
 server s1 10.0.0.101:80 check
 server s2 10.0.0.102:80 check
 server s3 10.0.0.103:80 check backup
 server s4 10.0.0.104:80 check backup

In this case, both s3 and s4 will be used if they are available.

Farm failover

Despite the case above improves a bit our failover scenario, it has some weaknesses. For example we must wait until all the production servers are DOWN before using the backup servers.
HAProxy can failover traffic to a backup farm when the main one has not enough capacity or, worst case, no capacity anymore.

Below, the HAProxy configuration corresponding to this case:

frontent ft_app
 bind 10.0.0.1:80

# detect capacity issues in production farm
 acl MAIN_not_enough_capacity nb_srv(bk_app_main) le 2
# failover traffic to backup farm
 use_backend bk_app_backup if MAIN_not_enough_capacity

 default_backend bk_app_main

backend bk_app_main
 server s11 10.0.0.101:80 check
 server s12 10.0.0.102:80 check
 server s13 10.0.0.103:80 check
 server s14 10.0.0.104:80 check

backend bk_app_backup
 server s21 20.0.0.101:80 check
 server s22 20.0.0.102:80 check

Farm failover with backup servers

Of course, we could combine all the options above.
First we want to failover to a backup farm if the production one has not enough capacity, second, we want to use 2 backup servers when all the production servers from the backup farm are DOWN.

Below, the HAProxy configuration corresponding to this case:

frontent ft_app
 bind 10.0.0.1:80

# detect capacity issues in production farm
 acl MAIN_not_enough_capacity nb_srv(bk_app_main) le 2
# failover traffic to backup farm
 use_backend bk_app_backup if MAIN_not_enough_capacity

 default_backend bk_app_main

backend bk_app_main
 server s11 10.0.0.101:80 check
 server s12 10.0.0.102:80 check
 server s13 10.0.0.103:80 check
 server s14 10.0.0.104:80 check

backend bk_app_backup
 option allbackups
 server s21 20.0.0.101:80 check
 server s22 20.0.0.102:80 check
 server s23 20.0.0.103:80 check backup
 server s24 20.0.0.104:80 check backup

Worst case: no servers available anymore

Well, imagine you plugged all your servers on a single switch, HAProxy box has 2 interfaces, one on the public switch, on on the server switch. Of course, this is not how you plugged your servers, don’t you?
Imagine the server switch fails, then no servers are available anymore. Then HAProxy can be used to deliver sorry pages for you.

Below, the HAProxy configuration corresponding to this case:

frontent ft_app
 bind 10.0.0.1:80

# sorry page to return when worst case happens
 errorfile 503 /etc/haproxy/errorfiles/sorry.http

# detect capacity issues in production farm
 acl MAIN_not_enough_capacity nb_srv(bk_app_main) le 2
# failover traffic to backup farm
 use_backend bk_app_backup if MAIN_not_enough_capacity

 default_backend bk_app_main

backend bk_app_main
 server s11 10.0.0.101:80 check
 server s12 10.0.0.102:80 check
 server s13 10.0.0.103:80 check
 server s14 10.0.0.104:80 check

backend bk_app_backup
 option allbackups
 server s21 20.0.0.101:80 check
 server s22 20.0.0.102:80 check
 server s23 20.0.0.103:80 check backup
 server s24 20.0.0.104:80 check backup

And below, the content of the sorry.http page:

HTTP/1.0 200 OK
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html>
<body>
<h1>Sorry page</h1>
Sorry, we're under maintenance
</body>
</html>

Important notes


Health checking


Health checking must be enabled on the servers. Without health checking, HAProxy can’t know the server status and then can’t decide to failover traffic.

Persistence


If a persistence information points to one backup server, then HAProxy will keep on using it, even if production servers are available.

Links