Microsoft Remote Desktop Services (RDS) Load-Balancing and protection

RDS, RDP, TSE, remoteapp

Whatever you call it, it’s the remote desktop protocol from Microsoft, which has been renamed during the product life.
Basically, it allows users to get connected on remote server and run an application or a full desktop remotely.

It’s more and more fashion nowadays, because it’s easier for IT teams to patch and upgrade a single server hosting many remote desktops than maintaining many laptop individually.

VDI or Virtual Desktop Infrastructure


A Virtual Desktop Infrastructure is an architecture purposely built to export desktops to end users. It can use different type of product, such as Citrix or VMWare View.
Of course, you could build yours using Microsoft RDS as well as the ALOHA Load-Balancer to improve scalability and security of such platform.

Diagram

In a previous article, we already demonstrate how good the ALOHA Load-Balancer is to smartly load-balance Microsoft RDS.
Let’s start again from the same platform (I know, I’m a lazy engineer!):
rdp infrastructure

Windows Broker


When using the ALOHA Load-Balancer, you don’t need a windows broker anymore: the ALOHA is able to balance a new user to the less loaded server, as well as resume a broken session.
When an ALOHA outage occurs and a failover from the master to the slave happens, then all the sessions are resumed: the ALOHAs can share session affinity within a cluster, ensuring nobody will notice the issue.

Making an RDS platforms more secured


It’s easy to load-balance RDS protocol using the mstshash cookie, since microsoft described his protocol: it allows advanced persistence. Basically, each user owns a dedicated mstshash cookie value.

That said, being able to detect weird behavior or service abusers would be much better.

Actually, the ALOHA Load-Balancer allows you to monitor each source IP address, and, much better, each mstshash cookie value.
For any object, we can monitor the connection rate over a period of time, the number of current connection, the bytes in and out (over a period of time), etc…

Now, it’s up to you to protect your VDI infrastructure using the ALOHA Load-Balancer. In example:
  * counting the number of active connections per user (mstshash cookie)
  * counting the number of connection tries over the last 5 minutes per user

The above will allow you to detect the following problems / misbehavior on a VDI platform:
  * a user being abnormally connected too many times
  * a user trying to get connected to quickly, or maybe somebody is trying to discover someone else’s password ;)
  * etc…

Configuration

The configuration below explains how to create a highly-available VDI infrastructure with advanced persistence and improved security:

# for High-Availability purpose
peers aloha
  peer aloha1 10.0.0.16:1024
  peer aloha2 10.0.0.17:1024

# RDP / TSE configuration
frontend ft_rdp
  bind 10.0.0.18:3389 name rdp
  mode tcp
  timeout client 1h
  option tcplog
  option log global
  option tcpka
  # Protection 1
  # wait up to 5s for the mstshash cookie, and reject the client if none
  tcp-request inspect-delay 5s
  tcp-request content accept if RDP_COOKIE

  default_backend bk_rdp

backend bk_rdp
  mode tcp
  balance roundrobin
  # Options
  timeout server 1h
  timeout connect 4s
  option redispatch
  option tcpka
  option tcplog
  option log global
  # sticky persistence + monitoring based on mstshash Cookie
  #    established connection
  #    connection tries over the last minute
  stick-table type string len 32 size 10k expire 1d peers aloha store conn_cur,conn_rate(1m)
  stick on rdp_cookie(mstshash)

  # Protection 2
  # Each user is supposed to get a single active connection at a time, block the second one
  tcp-request content reject if { sc1_conn_cur ge 2 }

  # Protection 3
  # if a user tried to get connected at least 10 times over the last minute, 
  # it could be a brute force
  tcp-request content reject if { sc1_conn_rate ge 10 }

  # Server farm
  server tse1 10.0.0.23:3389 weight 10 check inter 2s rise 2 fall 3
  server tse2 10.0.0.24:3389 weight 10 check inter 2s rise 2 fall 3
  server tse3 10.0.0.25:3389 weight 10 check inter 2s rise 2 fall 3
  server tse4 10.0.0.26:3389 weight 10 check inter 2s rise 2 fall 3

Troubleshouting RDS connections issues with the ALOHA


The command below, when run from your ALOHA Load-Balancer CLI, will print out (and refresh every 1 second) the content of the stick-table with the following information:
  * table name, table size and number of current entries in the table
  * mstshash cookie value as sent by RDS client: field key
  * affected server id: field server_id
  * monitored counter and values, fields conn_rate and conn_cur

while true ; do clear ; echo show table bk_rdp | socat /tmp/haproxy.sock - ; sleep 1 ; done

stick-table output in a normal case


Below, an example of the output when everything goes well:

# table: bk_rdp, type: string, size:20480, used:2
0x912f84: key=Administrator use=1 exp=0 server_id=0 conn_rate(60000)=1 conn_cur=1
0x91ba14: key=XLC\\Admin use=1 exp=0 server_id=1 conn_rate(60000)=1 conn_cur=1

stick-table output tracking a RDS abuser

Below, an example of the output when somebody tries many connections to discover someone’s password:

# table: bk_rdp, type: string, size:20480, used:2
0x912f84: key=Administrator use=1 exp=0 server_id=0 conn_rate(60000)=1 conn_cur=1
0x91ba14: key=XLC\\Admin use=1 exp=0 server_id=1 conn_rate(60000)=1 conn_cur=1
0x91ca12: key=XLC\\user1 use=1 exp=0 server_id=2 conn_rate(60000)=15 conn_cur=0

NOTE: in this case, the client has been blocked and the user user1 from the domain XLC can’t get connected for 1 minute (conn_rate monitoring period). I agree, this type of protection can lead to DOS your users…
We can easily configure the load-balancer to track source IP connection rate and is a single source IP tries to connect too often on our RDS farm, then we can block it before he can access it. (this type of attack would succeed only if the user is not only connected on the VDI platform)
Combining both source IP and mstshash cookie allows us to improve security on the VDI platform.

Securing a RDS platform combining both mstshash cookie and source IP address


The configuration below improve the protection by combining source IP address and RDS cookie.
The purpose is to block an abuser at a lower layer (IP) to avoid him making the Load-Balancer to blacklist regular user, even if this is temporary.

# for High-Availability purpose
peers aloha
  peer aloha1 10.0.0.16:1024
  peer aloha2 10.0.0.17:1024

# RDP / TSE configuration
frontend ft_rdp
  mode tcp
  bind 10.0.0.18:3389 name rdp
  timeout client 1h
  option tcpka
  log global
  option tcplog
  default_backend bk_rdp

backend bk_rdp
  mode tcp
  balance roundrobin

  # Options
  timeout server 1h
  timeout connect 4s
  option redispatch
  option tcpka
  log global
  option tcplog

  # sticky persistence + monitoring based on mstshash Cookie
  #    established connection
  #    connection tries over the last minute
  stick-table type string len 32 size 10k expire 1d peers aloha store conn_cur,conn_rate(1m)
  stick on rdp_cookie(mstshash)

  # Protection 1
  # wait up to 5s for the mstshash cookie, and reject the client if none
  tcp-request inspect-delay 5s
  tcp-request content accept if RDP_COOKIE
  tcp-request content track-sc1 rdp_cookie(mstshash) table bk_rdp
  tcp-request content track-sc2 src                  table sourceip

# IP layer protection
  #  Protection 2
  #   each single IP can have up to 2 connections on the VDI infrastructure
  tcp-request content reject if { sc2_conn_cur ge 2 }

  #  Protection 3
  #   each single IP can try up to 5 connections in a single minute
  tcp-request content reject if { sc2_conn_rate ge 10 }

# RDS / TSE application layer protection
  #  Protection 4
  #   Each user is supposed to get a single active connection at a time, block the second one
  tcp-request content reject if { sc1_conn_cur ge 2 }

  #  Protection 5
  #   if a user tried to get connected at least 10 times over the last minute, 
  #   it could be a brute force
  tcp-request content reject if { sc1_conn_rate ge 10 }

  # Server farm
  server tse1 10.0.0.23:3389 weight 10 check inter 2s rise 2 fall 3
  server tse2 10.0.0.24:3389 weight 10 check inter 2s rise 2 fall 3
  server tse3 10.0.0.25:3389 weight 10 check inter 2s rise 2 fall 3
  server tse4 10.0.0.26:3389 weight 10 check inter 2s rise 2 fall 3

backend sourceip
 stick-table type ip size 20k store conn_cur,conn_rate(1m)

Note that if you have remote sites using your VDI infrastructure, it is also possible to manage a source IP white list. And the same for the mstshash cookies: you can allow some users (like Administrator) to get connected multiple time on the VDI infrastructure.

Source IP and mstshash cookie combined protection


To observe the content of both tables (and still refreshed every 1s), just run the command below on your ALOHA CLI:

while true ; do clear ; echo show table bk_rdp | socat /tmp/haproxy.sock - ; echo "=====" ; echo show table sourceip | socat /tmp/haproxy.sock - ; sleep 1 ; done

And the result:

# table: bk_rdp, type: string, size:20480, used:2
0x23f1424: key=Administrator use=1 exp=0 server_id=0 conn_rate(60000)=0 conn_cur=1
0x2402904: key=XLC\\Admin use=1 exp=0 server_id=1 conn_rate(60000)=1 conn_cur=1

=====
# table: sourceip, type: ip, size:20480, used:2
0x23f9f70: key=10.0.4.215 use=2 exp=0 conn_rate(60000)=2 conn_cur=2
0x23f1500: key=10.0.3.20 use=1 exp=0 conn_rate(60000)=0 conn_cur=1

Important Note about Microsoft RDS clients


As Mathew Levett stated on LoadBalancer.org’s blog (here is the article: http://blog.loadbalancer.org/microsoft-drops-support-for-mstshash-cookies/), latest version of Microsoft RDS clients don’t send the full user’s information into the mstshash cookie. Worst, sometime, the client don’t even send the cookie!!!!

In the example we saw above, there was 2 users connected on the infrastructure: the Administrator user was connected from a Linux client using xfreerdp tool, and we saw the cookie properly in the stick table. Second user, Administrator, from the Windows domain XLC, as his credential truncated in the stick table (Value: XLC\\Admin), it was getting connected using Windows 2012 Server Enterprise RDS client.

As lb.org said, hopefully Microsoft will fix this issue. Or maybe there are some alternative RDS client for Windows…

To bypass this issue, we can take advantage of the ALOHA: we can do a persistence based on the source IP address when there is no mstshash cookies in the client connection information.
Just add the line below, right after the stick on rdp_cookie(mstshash):

  stick on src

With such configuration, the ALOHA Load-Balancer will stick on mstshash cookie, if present, and will failover on source IP address otherwise.

Conclusion


Virtualisation Desktop Infrastructure is a very fashion way of providing access to end users to a company’s applications.
If you don’t use the right product to build your RDS / TSE / RemoteApp infrastructure, you may be in trouble at some point.

The ALOHA Load-Balancer, thanks to its flexible configuration has many advantages

You can freely give a try to the ALOHA Load-Balancer, it is available from our website for download, allowing up to 10 connections with no time limitation: Download the ALOHA Load-Balancer

Links

Posted in Aloha, architecture, security, Virtual Desktop Infrastructure | Tagged , , , | 4 Comments

Configure syslog-ng to log readable HTTP URL from HAProxy

This tips is provided by Exosec.
Exosec provides a very good monitoring product called POM, based on Nagios with very strong value added such as very simple administration, application monitoring, etc…
For some of their project, they use either HAProxy or the ALOHA Load-Balancer (heh, what else???) and they export log entries into syslog-ng for storage and later analysis.

HAProxy’s log


HAProxy’s logs are very powerfull since they provide many information about the request and the status of the platform at the moment of the request.
For a readable HAProxy’s log description, please consult the ALOHA memo dedicated to HAProxy’s HTTP log line description.

One of the weakness of the log line is that it logs only the path and the query string of each URL. No server name neither protocol information.
Well, HAProxy allows us to log the Host header, which is fine and there is a tild ‘~’ after the frontend name when the connection is made over SSL.

Using syslog-ng flexible configuration, we can re-order things to make haproxy log the URL exactly like it was sent by the client, like:
http://www.domain.tld/url/path”

Configuration

HAProxy configuration


Note: this is a very minimalistic configuration, not recommended in production :)

global
 log 127.0.0.1:514 local2

frontend ft_http
 bind 127.0.0.1:8080
 bind 127.0.0.1:8081 ssl crt /etc/haproxy/haproxy.pem
 option http-server-close
 mode http
 log global
 option httplog
 # Mandatory to build the URL:
 capture request header Host       len 32
 # Optional, just for statistics:
 capture request header User-Agent len 200
 default_backend bk_http

backend bk_http
 option http-server-close
 mode http
 log global
 option httplog

 server srv1 10.0.0.1:80

Syslog-ng configuration


The configuration below will reproduce an HAProxy log line, but will replace the URL part by something more readable.

Note: if you capture a different number of HTTP headers in HAProxy (current example contains 2 captured headers), you may have to update the parser p_haproxy_headers_req and the destination d_haproxy_full.

source s_loopback { syslog(ip(127.0.0.1) port(514) transport("udp")); };

destination d_haproxy_full {
     file("/var/log/haproxy.$YEAR-$MONTH-$DAY.log"
          template("$DATE $FULLHOST $PROGRAM: ${HAPROXY.CLIENT_IPPORT} \[${HAPROXY.DATE}\] ${HAPROXY.FRONTEND} ${HAPROXY.BACKEND} ${HAPROXY.TIME} ${HAPROXY.STATUS_CODE} ${HAPROXY.BYTES_READ} ${HAPROXY.COOKIE_REQ} ${HAPROXY.COOKIE_RESP} ${HAPROXY.TERM_STATE} ${HAPROXY.RUN_STATE} ${HAPROXY.QUEUE_STATE} \{${HAPROXY.HEADERS_REQ}\} \"${HAPROXY.METHOD} ${HAPROXY.FRONTEND_PROTOCOL}://${HAPROXY.HOST}${HAPROXY.URL} ${HAPROXY.HTTP_VERSION}\"\n")
          group(adm) perm(0640) dir_perm(0750) template_escape(no)
         );
};

filter f_haproxy { program("haproxy"); };
filter f_frontend_ssl { match("~ "); };

rewrite r_set_frontend_protocol {
  set("http", value("HAPROXY.FRONTEND_PROTOCOL") condition(filter(f_haproxy)));
  set("https", value("HAPROXY.FRONTEND_PROTOCOL") condition(filter(f_frontend_ssl)));
};

parser p_haproxy {
  csv-parser(columns("HAPROXY.CLIENT_IPPORT", "HAPROXY.DATE",
                     "HAPROXY.FRONTEND", "HAPROXY.BACKEND",
                     "HAPROXY.TIME", "HAPROXY.STATUS_CODE",
                     "HAPROXY.BYTES_READ", "HAPROXY.COOKIE_REQ",
                     "HAPROXY.COOKIE_RESP", "HAPROXY.TERM_STATE",
                     "HAPROXY.RUN_STATE", "HAPROXY.QUEUE_STATE",
                     "HAPROXY.HEADERS_REQ", "HAPROXY.REQUEST")
             flags(escape-double-char,strip-whitespace)
             delimiters(" ")
             quote-pairs('""[]{}'));
};

parser p_haproxy_request {
  csv-parser(columns("HAPROXY.METHOD", "HAPROXY.URL",
                     "HAPROXY.HTTP_VERSION")
             delimiters(" ")
             flags(escape-none)
             template("${HAPROXY.REQUEST}"));
};

parser p_haproxy_headers_req {
  csv-parser(columns("HAPROXY.HOST", "HAPROXY.USER_AGENT")
             delimiters("|")
             flags(escape-none)
             template("${HAPROXY.HEADERS_REQ}"));
};

log {
  source(s_loopback);
  filter(f_haproxy);
  parser(p_haproxy);
  parser(p_haproxy_request);
  parser(p_haproxy_headers_req);
  rewrite(r_set_frontend_protocol);
  destination(d_haproxy_full);
};

This configuration is downloadable from HAProxy Technologies github: https://raw.github.com/exceliance/haproxy/master/logs/syslog-ng_full_http_url.conf

And the result would look like below at the end of the logged line:

[...] "GET http://test.domain.tld/blah HTTP/1.1"
[...] "GET https://test.domain.tld/blah HTTP/1.1"

Links

Posted in HAProxy, optimization | Tagged , | Leave a comment

IIS 6.0 appsession cookie and PCI compliance

Synopsis

You’re using HAProxy or the ALOHA Load-Balancer to load-balance IIS 6.0 web applications and you want them to pass successfully PCI compliance test.
One of the pre-requisite is to force the cookie to be “HttpOnly”, in order to tell the browser to use this cookie for HTTP requests only, and “protect” it from local javascript access (to steal session information).
Unfortunately, II 6.0 is not able to setup such cookies. That’s why HAProxy can be used to update the cookie on the fly, when setup by the application server.

Rewriting appsession Cookie with HAProxy

Place the configuration line below in your backend configuration:

rspirep ^Set-Cookie:\ (appsession.*)    Set-Cookie:\ \1;\ HttpOnly

Now, you’re application is “more” secured… Well, at least, you can successfully pass the PCI compliancy tests!

Links

Posted in Aloha, HAProxy, layer7, security | Tagged , , , | Leave a comment

SSL offloading impact on web applications

SSL Offloading

Nowadays, it is common (and convenient) to use the Load-Balancer SSL capabilities to cypher/uncypher traffic from clients to the web application platform.
Performing SSL at the Load-Balancer Layer is called “SSL offloading“, because you offload this process from your application servers.

Note that SSL offloading is also “marketingly” called SSL acceleration. The only acceleration performed is moving SSL processing from application server, which have an other heavy task to perform, to an other devices in the network…
So no acceleration at all, despite some people claming they do accelerate SSL in a dedicated ASIC, they just offload

The diagram below shows what happens on an architecture with SSL offloading. We can clearly see that the traffic is encrypted up to the Load-Balancer, then it’s in clear between the Load-Balancer and the application server:
ssl offloading diagram

Benefits of SSL offloading


Offloading SSL from the application servers to the Load-Balancers have many benefits:
  * offload an heavy task from your application servers, letting them to focus on the application itself
  * save resources on your application servers
  * the Load-Balancers have access to clear HTTP traffic and can perform advanced features such as reverse-proxying, Cookie persistence, traffic regulation, etc…
  * When using an ALOHA Load-Balancer (or HAProxy), there are much more features available on the SSL stack than on any web application server.

It seems there should only be positive impact when offloading SSL.

Counterparts of SSL offloading


As we saw on the diagram above, the load-balancer will hide three important information from the client connection:
  1. protocol scheme: the client was using HTTPS while the connection on the application server is made over HTTP
  2. connection type: was it cyphered or not? Actually, this is linked to the protocol scheme
  3. destination port: on the load-balancer, the port is usually 443 and on the application server should be 80 (this may change)

Most web application will use 301/302 responses with a Location header to redirect the client to a page (most of the time after a login or a POST) as well as requiring an application cookie.

So basically, the worst impact is that your application server may have difficulties to know the client connection information and may not be able to perform right responses: it can break totally the application, preventing it to work.
Fortunately, the ALOHA Load-Balancer or HAProxy can help in such situation!

Basically, the application could return such responses to the client:

Location: http://www.domain.com/login.jsp

And the client will leave the secured connection…

Tracking issues with the load-balancer


In order to know if your application supports well SSL offloading, you can configure your ALOHA Load-Balancer to log some server responses.
Add the configuration below in your application frontend section:

capture response header Location   len 32
capture response header Set-Cookie len 32

Now, in the traffic log, you’ll see clearly if the application is setting up response for HTTP or HTTPS.

NOTE: some other headers may be needed, depending on your application.

Web Applications and SSL offloading


Here we are :)

There are 3 answers possible for such situation, detailed below.

Client connection information provided in HTTP header by the Load-Balancer


First of all, the Load-Balancer can provide client side connection information to the application server through HTTP header.
The configuration below shows how to insert a header called X-Forwared-Proto containing the scheme used by the client.
To be added in your backend section.

http-request set-header X-Forwarded-Proto https if  { ssl_fc }
http-request set-header X-Forwarded-Proto http  if !{ ssl_fc }

Now, the ALOHA Load-Balancer will insert the following header when the connection is made over SSL:

X-Forwarded-Proto: https

and when performed over clear HTTP:

X-Forwarded-Proto: http

It’s up to your application to use this information to answer the right responses: this may require some code update.

HTTP header rewriting on the fly


An other way is to rewrite the response setup by the server on the fly.
The following configuration line would match the Location header and translate it from http to https if needed.
Rewriting rspirep http://www.domain.com:80/url:

rspirep ^Location:\ http://(.*):80(.*)  Location:\ https://\1:443\2   if  { ssl_fc }

NOTE: above example applies on HTTP redirection (Location header), but can be applied on Set-cookie as well (or any other header your application may use).
NOTE2: you can only rewrite the response HTTP headers, not in the body. So this is not compatible with applications setting up hard links in the HTML content.

SSL Bridging


In some cases, the application is not compatible at all with SSL offloading (even with the tricks above) and we must use a ciphered connection to the server but we still may require to perform cookie based persistence, content switching, etc…
This is called SSL bridging, or can also be called a man in the middle.

In the ALOHA, there is nothing to do, just do SSL offloading as usual and add the keyword ssl on your server directive.

Using this method, you can choose a light cipher and a light key between the Load-Balancer and the server and still use the Load-Balancer advanced SSL feature with a stronger key and a stronger cipher.
The application server won’t notice anything and the Load-Balancer can still perform Layer 7 processing.

Links

Posted in Aloha, architecture, HAProxy, ssl | Tagged , , , | 8 Comments

Mitigating the SSL Beast attack using the ALOHA Load-Balancer / HAProxy

The beast attack on SSL isn’t new, but we have not yet published an article to explain how to mitigate it with the ALOHA or HAProxy.
First of all, to mitigate this attack, you must use the Load-Balancer as the SSL endpoint, then just append the following parameter on your HAProxy SSL frontend:
  * For the ALOHA Load-Balancer:

bind 10.0.0.9:443 name https ssl crt domain ciphers RC4:HIGH:!aNULL:!MD5

  * For HAProxy OpenSource:

bind 10.0.0.9:443 name https ssl crt /path/to/domain.pem ciphers RC4:HIGH:!aNULL:!MD5

As you may have understood, the most important part is the ciphers RC4:HIGH:!aNULL:!MD5 directive which can be used to force the cipher used during the connection and to force it to be strong enough to resist to the attack.

Related Links

Links

Posted in Aloha, exchange 2010, Exchange 2013, HAProxy, security, ssl | Tagged , , , | 2 Comments

Microsoft Exchange 2013 architectures

Introduction to Microsoft Exchange 2013

There are 2 types of server in Exchange 2013:
  * Mailbox server
  * Client Access server

Definitions from Microsoft Technet website:
  * The Client Access server provides authentication, limited redirection, and proxy services, and offers all the usual client access protocols: HTTP, POP and IMAP, and SMTP. The Client Access server, a thin and stateless server, doesn’t do any data rendering. There’s never anything queued or stored on the Client Access server.
  * The Mailbox server includes all the traditional server components found in Exchange 2010: the Client Access protocols, the Transport service, the Mailbox databases, and Unified Messaging (the Client Access server redirects SIP traffic generated from incoming calls to the Mailbox server). The Mailbox server handles all activity for the active mailboxes on that server.

High availability in Exchange 2013


Mailbox server high availability is achieved by the creation of a DAG: Database Availability Group. All the redundancy is achieved within the DAG and the Client Access server will retrieve the user session through the DAG automatically when a fail over occurs.

Client Access servers must be Load-Balanced to achieve high-availability. There are many ways to load-balance them:
  * DNS round-robin (Com’on, we’re in 2013, stop mentioning this option !!!!)
  * NLB (Microsoft don’t recommend it)
  * Load-Balancers (also known as Hardware Load-Balancer or HLB): this solution is recommended since it can bring smart load-balancing and advanced health checking.

Performance in Exchange 2013


The key performance in Exchange 2013 is obviously the Mailbox server. If this one slows down, all the users are affected. So be careful when designing your Exchange platform and try to create multiple DAGs with one single master per DAG. If one Mailbox server fails or is in maintenance, then you’ll be in degraded mode (which does not mean degraded performance) where one Mailbox server would be master on 2 DAGs. The disks IO may also be important here ;)

Concerning the Client Access server, it’s easy to disable temporarily one of them if it’s slowing down and route connections to the other Client Access server(s).

Exchange 2013 Architecture


Your architecture should meet your requirements:
  * if you don’t need high-availability and performance, then a single server with both Client Access and Mailbox role is sufficient
  * if you need high-availability and have a moderate number of users, a couple of servers, each one hosting both Client Access and Mailbox role is enough. A DAG to ensure high-availability of the Mailbox servers and a Load-Balancer to ensure high-availability of the Client Access servers.
  * if you need high-availability and performance for a huge amount of connections, then multiple Mailbox server, multiple DAGs, multiple Client Access servers and a couple of Load-Balancers.

Note that the path from the first architecture to the third one is doable with almost no effort, thanks to Exchange 2013.

Client Access server Load-Balancing in Exchange 2013


Using a Load-Balancer to balance user connections to the Client Access servers allows multiple type of architectures. The current article mainly focus on these architectures: we do provide load-balancers ;)
If you need help from Exchange architects, we have partners who can help you with the installation, setup and tuning of the whole architecture.

Load-Balancing Client Access servers in Exchange 2013

First of all, bear in mind that Client Access servers in Exchange 2013 are stateless. It is very important from a load-balancing point of view because it makes the configuration much more simple and scalable.
Basically, I can see 3 main types of architectures for Client Access servers load-balancing in Exchange 2013:
  1. Reverse-proxy mode (also known as source NAT)
  2. Layer 4 NAT mode (desitnation NAT)
  3. Layer 4 DSR mode (Direct Server Return, also known as Gateway)

Each type of architecture has pros and cons and may have impact on your global infrastructure. In the next chapter, I’ll provide details for each of them.

To illustrate the article, I’ll use a simple Exchange 2013 configuration:
  * 2 Exchange 2013 servers with hosting both Client Access and Mailbox role
  * 1 ALOHA Load-Balancer
  * 1 client (laptop)
  * all the Exchange services hosted on a single hostname (IE mail.domain.com) (could work as well with an hostname per service)

Reverse-proxy mode (also known as source NAT)


Diagram
The diagram below shows how things work with a Load-balancer in Reverse-proxy:
  1. the client establishes a connection onto the Load-Balancer
  2. the Load-Balancer chooses a Client Access server and establishes the connection on it
  3. client and CAS server discuss through the Load-Balancer
Basically, the Load-Balancer breaks the TCP connection between the client and the server: there are 2 TCP connections established.

exchange_2013_reverse_proxy

Advantages of Reverse-proxy mode
  * not intrusive: no infrastructure changes neither server changes
  * ability to perform SSL offloading and layer 7 persistence
  * client, Client Access servers and load-balancers can be located anywhere in the infrastructure (same subnet or different subnet)
  * can be used in a DMZ
  * A single interface is required on the Load-Balancer

Limitations of Reverse-proxy mode
  * the servers don’t see the client IPs (only the LB one)
  * maximum 65K connections on the Client Access farm (without tricking the configuration)

Configuration
To perform this type of configuration, you can use the LB Admin tab and do it through the GUI or just copy/paste the following lines into the LB Layer 7 tab:

######## Default values for all entries till next defaults section
defaults
  option  dontlognull             # Do not log connections with no requests
  option  redispatch              # Try another server in case of connection failure
  option  contstats               # Enable continuous traffic statistics updates
  retries 3                       # Try to connect up to 3 times in case of failure 
  backlog 10000                   # Size of SYN backlog queue
  mode tcp                        #alctl: protocol analyser
  log global                      #alctl: log activation
  option tcplog                   #alctl: log format
  timeout client  300s            #alctl: client inactivity timeout
  timeout server  300s            #alctl: server inactivity timeout
  timeout connect   5s            # 5 seconds max to connect or to stay in queue
  default-server inter 3s rise 2 fall 3   #alctl: default check parameters

frontend ft_exchange
  bind 10.0.0.9:443 name https    #alctl: listener https configuration.
  maxconn 10000                   #alctl: max connections (dep. on ALOHA capacity)
  default_backend bk_exchange     #alctl: default farm to use

backend bk_exchange
  balance roundrobin              #alctl: load balancing algorithm
  server exchange1 10.0.0.13:443 check
  server exchange2 10.0.0.14:443 check

(update IPs for your infrastructure and update the DNS name with the IP configured on the frontend bind line.)

Layer 4 NAT mode (desitnation NAT)


Diagram
The diagram below shows how things work with a Load-balancer in Layer 4 NAT mode:
  1. the client establishes a connection to the Client Access server through the Load-Balancer
  2. client and CAS server discuss through the Load-Balancer
Basically, the Load-Balancer acts as a router between the client and the server: a single TCP connection is established directly between the client and the Client Access server, and the LB just forward packets between both of them.
exchange_2013_layer4_nat

Advantages of Layer 4 NAT mode
  * no connection limit
  * Client Access servers can see the client IP address

Limitations of Layer 4 NAT mode
  * infrastructure intrusive: the server default gateway must be the load-balancer
  * clients and Client Access servers can’t be in the same subnet
  * no SSL acceleration, no advanced persistence
  * Must use 2 interfaces on the Load-Balancer (or VLAN interface)

Configuration
To perform this type of configuration, you can use the LB Admin tab and do it through the GUI or just copy/paste the following lines into the LB Layer 4 tab:

director exchange 10.0.1.9:443 TCP
  balance roundrobin                               #alctl: load balancing algorythm
  mode nat                                         #alctl: forwarding mode
  check interval 10 timeout 2                      #alctl: check parameters
  option tcpcheck                                  #alctl: adv check parameters
  server exchange1 10.0.0.13:443 weight 10 check   #alctl: server exchange1
  server exchange2 10.0.0.14:443 weight 10 check   #alctl: server exchange2

(update IPs for your infrastructure and update the DNS name with the IP configured on the frontend bind line.)

Layer 4 DSR mode (Direct Server Return, also known as Gateway)


Diagram
The diagram below shows how things work with a Load-balancer in Layer 4 DSR mode:
  1. the client establishes a connection to the Client Access server through the Load-Balancer
  2. the Client Access server acknowledges the connection directly to the client, bypassing the Load-Balancer on the way back. The Client Access server must have the Load-Balancer Virtual IP configured on a loopback interface.
  3. client and CAS server keep on discussing the same way: request through the load-balancer and responses directly from server to client.
Basically, the Load-Balancer acts as a router between the client and the server: a single TCP connection is established directly between the client and the Client Access server.
The Client and the server can be in the same subnet, like in the diagram, or can be in two different subnet. In the second case, the server would use its default gateway (no need to forward the traffic back through the load-balancer).
exchange_2013_layer4_dsr

Advantages of Layer 4 DSR mode
  * no connection limit
  * Client Access servers can see the client IP address
  * clients and Client Access servers can be in the same subnet
  * A single interface is required on the Load-Balancer

Limitations of Layer 4 DSR mode
  * infrastructure intrusive: the Load-Balancer Virtual IP must be configured on the Client Access server (Loopback).
  * no SSL acceleration, no advanced persistence

Configuration
To perform this type of configuration, you can use the LB Admin tab and do it though the web form or just copy/paste the following lines into the LB Layer 4 tab:

director exchange 10.0.0.9:443 TCP
  balance roundrobin                               #alctl: load balancing algorythm
  mode gateway                                     #alctl: forwarding mode
  check interval 10 timeout 2                      #alctl: check parameters
  option tcpcheck                                  #alctl: adv check parameters
  server exchange1 10.0.0.13:443 weight 10 check   #alctl: server exchange1
  server exchange2 10.0.0.14:443 weight 10 check   #alctl: server exchange2

Related links

Links

Posted in Aloha, architecture, exchange, Exchange 2013, layer4, layer7 | Tagged , , , , , , | 5 Comments

Microsoft Exchange 2013 load-balancing with HAProxy

Introduction to Microsoft Exchange server 2013

Note: I’ll introduce exchange from a Load-Balancing point of view. For a detailed information about exchange history and new features, please read the pages linked in the Related links at the bottom of this article.

Exchange is the name of the Microsoft software which provides a business-class mail / calendar / contact platform. It’s an old software, starting with version 4.0 back in 1996…
Each new version of Exchange Server brings in new features, both expanding Exchange perimeter and making it easier to deploy and administrate.

Exchange 2007


Introduction of Outlook Anywhere, AKA RPC over HTTP: allows remote users to get connected on Exchange 2007 platform using HTTPs protocol.

Exchange 2010


In example, Exchange 2010 introduced CAS arrays, making client side services high-available and scalable. DAG also brings mail database high-availability. All the client access services required persistence: a user must be sticked to a single CAS server.
Exchange 2010 introduced as well a “layer” between the MAPI RPC clients and the mailbox servers (through the CAS servers), making the failover of a database transparent.

Exchange 2013


Exchange 2013 improved again the changes brought by Exchange 2010: the CAS servers are now state-less and independent from each other (no arrays anymore): no persistence required anymore.
In exchange 2013, raw TCP MAPI RPC services have disappeared and have definitively been replaced by Outlook Anywhere (RPC over HTTP).
Last but not least, SSL offloading does not seem to be allowed for now.

Load-Balancing Microsoft Exchange 2013

First of all, I’m pleased to announce that HAProxy and the ALOHA Load-Balancer are both able to load-balance Exchange 2013 (as well as 2010).

Exchange 2013 Services


As explained in introduction, the table below summarizes the TCP ports and services involved in an Exchange 2013 platform:




TCP PortProtocolCAS Service name (abbreviation)
443HTTPS- Autodiscover (AS)
- Exchange ActiveSync (EAS)
- Exchange Control Panel (ECP)
- Offline Address Book (OAB)
- Outlook Anywhere (OA)
- Outlook Web App (OWA)
110 and 995POP3 / POP3sPOP3
143 and 993IMAP4 / IMAP4sIMAP4

Diagram

There are two main types of architecture doable:
1. All the services are hosted on a single host name
2. Each service owns its own host name

Exhange 2013 and the Single host name diagram


exchange_2013_single_hostname

Exhange 2013 and the Multiple host name diagram


exchange_2013_multiple_hostnames

Configuration

There are two types of configuration with the ALOHA:
Layer 4 mode: the LB act as a router, infrastrcuture intrusive, ability to manage millions of connections
layer 7 mode: the LB act as a reverse-proxy, non-intrusive implementation (source NAT), ability to manage thousands of connections, perform SSL offloading, DDOS protection, advanced persistence, etc…

The present article describe the layer 7 configuration, even if we’re going to use it at layer 4 (mode tcp).

Note that it’s up to you to update your DNS configuration to make the hostname point to your Load-Balancer service Virtual IP.

Template:
Use the configuration below as templates and just change the IP addresses:
bind line to your client facing service IPs
server line IPs to match your CAS servers (and add as many line as you need)
Once updated, just copy/paste the whole configuration, including the default section to the bottom of your ALOHA Layer 7 configuration.

Load-Balancing Exhange 2013 services hosted on a Single host name

######## Default values for all entries till next defaults section
defaults
  option  dontlognull             # Do not log connections with no requests
  option  redispatch              # Try another server in case of connection failure
  option  contstats               # Enable continuous traffic statistics updates
  retries 3                       # Try to connect up to 3 times in case of failure 
  timeout connect 5s              # 5 seconds max to connect or to stay in queue
  timeout http-keep-alive 1s      # 1 second max for the client to post next request
  timeout http-request 15s        # 15 seconds max for the client to send a request
  timeout queue 30s               # 30 seconds max queued on load balancer
  timeout tarpit 1m               # tarpit hold tim
  backlog 10000                   # Size of SYN backlog queue

  balance roundrobin                      #alctl: load balancing algorithm
  mode tcp                                #alctl: protocol analyser
  option tcplog                           #alctl: log format
  log global                              #alctl: log activation
  timeout client 300s                     #alctl: client inactivity timeout
  timeout server 300s                     #alctl: server inactivity timeout
  default-server inter 3s rise 2 fall 3   #alctl: default check parameters

frontend ft_exchange_tcp
  bind 10.0.0.9:443 name https          #alctl: listener https configuration.
  maxconn 10000                         #alctl: connection max (depends on capacity)
  default_backend bk_exchange_tcp       #alctl: default farm to use

backend bk_exchange_tcp
  server cas1 10.0.0.15:443 maxconn 10000 check    #alctl: server cas1 configuration.
  server cas2 10.0.0.16:443 maxconn 10000 check    #alctl: server cas2 configuration.

And the result (LB Admin tab):
– Virtual Service:
aloha_exchange2013_single_domain_virtual_services
– Server Farm:
aloha_exchange2013_single_domain_server_farm

Load-Balancing Exhange 2013 services hosted on Multiple host name

######## Default values for all entries till next defaults section
defaults
  option  dontlognull             # Do not log connections with no requests
  option  redispatch              # Try another server in case of connection failure
  option  contstats               # Enable continuous traffic statistics updates
  retries 3                       # Try to connect up to 3 times in case of failure 
  timeout connect 5s              # 5 seconds max to connect or to stay in queue
  timeout http-keep-alive 1s      # 1 second max for the client to post next request
  timeout http-request 15s        # 15 seconds max for the client to send a request
  timeout queue 30s               # 30 seconds max queued on load balancer
  timeout tarpit 1m               # tarpit hold tim
  backlog 10000                   # Size of SYN backlog queue

  balance roundrobin                      #alctl: load balancing algorithm
  mode tcp                                #alctl: protocol analyser
  option tcplog                           #alctl: log format
  log global                              #alctl: log activation
  timeout client 300s                     #alctl: client inactivity timeout
  timeout server 300s                     #alctl: server inactivity timeout
  default-server inter 3s rise 2 fall 3   #alctl: default check parameters

frontend ft_exchange_tcp
  bind 10.0.0.5:443  name as        #alctl: listener: autodiscover service
  bind 10.0.0.6:443  name eas       #alctl: listener: Exchange ActiveSync service
  bind 10.0.0.7:443  name ecp       #alctl: listener: Exchange Control Panel service
  bind 10.0.0.8:443  name ews       #alctl: listener: Exchange Web Service service
  bind 10.0.0.8:443  name oa        #alctl: listener: Outlook Anywhere service
  maxconn 10000                     #alctl: connection max (depends on capacity)
  default_backend bk_exchange_tcp   #alctl: default farm to use

backend bk_exchange_tcp
  server cas1 10.0.0.15:443 maxconn 10000 check   #alctl: server cas1 configuration.
  server cas2 10.0.0.16:443 maxconn 10000 check   #alctl: server cas2 configuration.

And the result (LB Admin tab):
– Virtual Service:
aloha_exchange2013_multiple_domain_virtual_services
– Server Farm:
aloha_exchange2013_multiple_domain_server_farm

Conclusion


This is a very basic and straight forward configuration. We could make it much more complete and improve timeouts per services, better health checking, DDOS protection, etc…
I may write later articles about Exchange 2013 Load-Balancing with our products.

Related links

Exchange 2013 installation steps
Exchange 2013 first configuration
Microsoft Exchange Server (Wikipedia)
Microsft Exchange official webpage

Links

Posted in exchange, Exchange 2013 | Tagged , , , , , | 7 Comments