Category Archives: Exchange 2013

Mitigating the SSL Beast attack using the ALOHA Load-Balancer / HAProxy

The beast attack on SSL isn’t new, but we have not yet published an article to explain how to mitigate it with the ALOHA or HAProxy.
First of all, to mitigate this attack, you must use the Load-Balancer as the SSL endpoint, then just append the following parameter on your HAProxy SSL frontend:
  * For the ALOHA Load-Balancer:

bind 10.0.0.9:443 name https ssl crt domain ciphers RC4:HIGH:!aNULL:!MD5

  * For HAProxy OpenSource:

bind 10.0.0.9:443 name https ssl crt /path/to/domain.pem ciphers RC4:HIGH:!aNULL:!MD5

As you may have understood, the most important part is the ciphers RC4:HIGH:!aNULL:!MD5 directive which can be used to force the cipher used during the connection and to force it to be strong enough to resist to the attack.

Related Links

Links

Microsoft Exchange 2013 architectures

Introduction to Microsoft Exchange 2013

There are 2 types of server in Exchange 2013:
  * Mailbox server
  * Client Access server

Definitions from Microsoft Technet website:
  * The Client Access server provides authentication, limited redirection, and proxy services, and offers all the usual client access protocols: HTTP, POP and IMAP, and SMTP. The Client Access server, a thin and stateless server, doesn’t do any data rendering. There’s never anything queued or stored on the Client Access server.
  * The Mailbox server includes all the traditional server components found in Exchange 2010: the Client Access protocols, the Transport service, the Mailbox databases, and Unified Messaging (the Client Access server redirects SIP traffic generated from incoming calls to the Mailbox server). The Mailbox server handles all activity for the active mailboxes on that server.

High availability in Exchange 2013


Mailbox server high availability is achieved by the creation of a DAG: Database Availability Group. All the redundancy is achieved within the DAG and the Client Access server will retrieve the user session through the DAG automatically when a fail over occurs.

Client Access servers must be Load-Balanced to achieve high-availability. There are many ways to load-balance them:
  * DNS round-robin (Com’on, we’re in 2013, stop mentioning this option !!!!)
  * NLB (Microsoft don’t recommend it)
  * Load-Balancers (also known as Hardware Load-Balancer or HLB): this solution is recommended since it can bring smart load-balancing and advanced health checking.

Performance in Exchange 2013


The key performance in Exchange 2013 is obviously the Mailbox server. If this one slows down, all the users are affected. So be careful when designing your Exchange platform and try to create multiple DAGs with one single master per DAG. If one Mailbox server fails or is in maintenance, then you’ll be in degraded mode (which does not mean degraded performance) where one Mailbox server would be master on 2 DAGs. The disks IO may also be important here 😉

Concerning the Client Access server, it’s easy to disable temporarily one of them if it’s slowing down and route connections to the other Client Access server(s).

Exchange 2013 Architecture


Your architecture should meet your requirements:
  * if you don’t need high-availability and performance, then a single server with both Client Access and Mailbox role is sufficient
  * if you need high-availability and have a moderate number of users, a couple of servers, each one hosting both Client Access and Mailbox role is enough. A DAG to ensure high-availability of the Mailbox servers and a Load-Balancer to ensure high-availability of the Client Access servers.
  * if you need high-availability and performance for a huge amount of connections, then multiple Mailbox server, multiple DAGs, multiple Client Access servers and a couple of Load-Balancers.

Note that the path from the first architecture to the third one is doable with almost no effort, thanks to Exchange 2013.

Client Access server Load-Balancing in Exchange 2013


Using a Load-Balancer to balance user connections to the Client Access servers allows multiple type of architectures. The current article mainly focus on these architectures: we do provide load-balancers 😉
If you need help from Exchange architects, we have partners who can help you with the installation, setup and tuning of the whole architecture.

Load-Balancing Client Access servers in Exchange 2013

First of all, bear in mind that Client Access servers in Exchange 2013 are stateless. It is very important from a load-balancing point of view because it makes the configuration much more simple and scalable.
Basically, I can see 3 main types of architectures for Client Access servers load-balancing in Exchange 2013:
  1. Reverse-proxy mode (also known as source NAT)
  2. Layer 4 NAT mode (desitnation NAT)
  3. Layer 4 DSR mode (Direct Server Return, also known as Gateway)

Each type of architecture has pros and cons and may have impact on your global infrastructure. In the next chapter, I’ll provide details for each of them.

To illustrate the article, I’ll use a simple Exchange 2013 configuration:
  * 2 Exchange 2013 servers with hosting both Client Access and Mailbox role
  * 1 ALOHA Load-Balancer
  * 1 client (laptop)
  * all the Exchange services hosted on a single hostname (IE mail.domain.com) (could work as well with an hostname per service)

Reverse-proxy mode (also known as source NAT)


Diagram
The diagram below shows how things work with a Load-balancer in Reverse-proxy:
  1. the client establishes a connection onto the Load-Balancer
  2. the Load-Balancer chooses a Client Access server and establishes the connection on it
  3. client and CAS server discuss through the Load-Balancer
Basically, the Load-Balancer breaks the TCP connection between the client and the server: there are 2 TCP connections established.

exchange_2013_reverse_proxy

Advantages of Reverse-proxy mode
  * not intrusive: no infrastructure changes neither server changes
  * ability to perform SSL offloading and layer 7 persistence
  * client, Client Access servers and load-balancers can be located anywhere in the infrastructure (same subnet or different subnet)
  * can be used in a DMZ
  * A single interface is required on the Load-Balancer

Limitations of Reverse-proxy mode
  * the servers don’t see the client IPs (only the LB one)
  * maximum 65K connections on the Client Access farm (without tricking the configuration)

Configuration
To perform this type of configuration, you can use the LB Admin tab and do it through the GUI or just copy/paste the following lines into the LB Layer 7 tab:

######## Default values for all entries till next defaults section
defaults
  option  dontlognull             # Do not log connections with no requests
  option  redispatch              # Try another server in case of connection failure
  option  contstats               # Enable continuous traffic statistics updates
  retries 3                       # Try to connect up to 3 times in case of failure 
  backlog 10000                   # Size of SYN backlog queue
  mode tcp                        #alctl: protocol analyser
  log global                      #alctl: log activation
  option tcplog                   #alctl: log format
  timeout client  300s            #alctl: client inactivity timeout
  timeout server  300s            #alctl: server inactivity timeout
  timeout connect   5s            # 5 seconds max to connect or to stay in queue
  default-server inter 3s rise 2 fall 3   #alctl: default check parameters

frontend ft_exchange
  bind 10.0.0.9:443 name https    #alctl: listener https configuration.
  maxconn 10000                   #alctl: max connections (dep. on ALOHA capacity)
  default_backend bk_exchange     #alctl: default farm to use

backend bk_exchange
  balance roundrobin              #alctl: load balancing algorithm
  server exchange1 10.0.0.13:443 check
  server exchange2 10.0.0.14:443 check

(update IPs for your infrastructure and update the DNS name with the IP configured on the frontend bind line.)

Layer 4 NAT mode (desitnation NAT)


Diagram
The diagram below shows how things work with a Load-balancer in Layer 4 NAT mode:
  1. the client establishes a connection to the Client Access server through the Load-Balancer
  2. client and CAS server discuss through the Load-Balancer
Basically, the Load-Balancer acts as a router between the client and the server: a single TCP connection is established directly between the client and the Client Access server, and the LB just forward packets between both of them.
exchange_2013_layer4_nat

Advantages of Layer 4 NAT mode
  * no connection limit
  * Client Access servers can see the client IP address

Limitations of Layer 4 NAT mode
  * infrastructure intrusive: the server default gateway must be the load-balancer
  * clients and Client Access servers can’t be in the same subnet
  * no SSL acceleration, no advanced persistence
  * Must use 2 interfaces on the Load-Balancer (or VLAN interface)

Configuration
To perform this type of configuration, you can use the LB Admin tab and do it through the GUI or just copy/paste the following lines into the LB Layer 4 tab:

director exchange 10.0.1.9:443 TCP
  balance roundrobin                               #alctl: load balancing algorythm
  mode nat                                         #alctl: forwarding mode
  check interval 10 timeout 2                      #alctl: check parameters
  option tcpcheck                                  #alctl: adv check parameters
  server exchange1 10.0.0.13:443 weight 10 check   #alctl: server exchange1
  server exchange2 10.0.0.14:443 weight 10 check   #alctl: server exchange2

(update IPs for your infrastructure and update the DNS name with the IP configured on the frontend bind line.)

Layer 4 DSR mode (Direct Server Return, also known as Gateway)


Diagram
The diagram below shows how things work with a Load-balancer in Layer 4 DSR mode:
  1. the client establishes a connection to the Client Access server through the Load-Balancer
  2. the Client Access server acknowledges the connection directly to the client, bypassing the Load-Balancer on the way back. The Client Access server must have the Load-Balancer Virtual IP configured on a loopback interface.
  3. client and CAS server keep on discussing the same way: request through the load-balancer and responses directly from server to client.
Basically, the Load-Balancer acts as a router between the client and the server: a single TCP connection is established directly between the client and the Client Access server.
The Client and the server can be in the same subnet, like in the diagram, or can be in two different subnet. In the second case, the server would use its default gateway (no need to forward the traffic back through the load-balancer).
exchange_2013_layer4_dsr

Advantages of Layer 4 DSR mode
  * no connection limit
  * Client Access servers can see the client IP address
  * clients and Client Access servers can be in the same subnet
  * A single interface is required on the Load-Balancer

Limitations of Layer 4 DSR mode
  * infrastructure intrusive: the Load-Balancer Virtual IP must be configured on the Client Access server (Loopback).
  * no SSL acceleration, no advanced persistence

Configuration
To perform this type of configuration, you can use the LB Admin tab and do it though the web form or just copy/paste the following lines into the LB Layer 4 tab:

director exchange 10.0.0.9:443 TCP
  balance roundrobin                               #alctl: load balancing algorythm
  mode gateway                                     #alctl: forwarding mode
  check interval 10 timeout 2                      #alctl: check parameters
  option tcpcheck                                  #alctl: adv check parameters
  server exchange1 10.0.0.13:443 weight 10 check   #alctl: server exchange1
  server exchange2 10.0.0.14:443 weight 10 check   #alctl: server exchange2

Related links

Links

Microsoft Exchange 2013 load-balancing with HAProxy

Introduction to Microsoft Exchange server 2013

Note: I’ll introduce exchange from a Load-Balancing point of view. For a detailed information about exchange history and new features, please read the pages linked in the Related links at the bottom of this article.

Exchange is the name of the Microsoft software which provides a business-class mail / calendar / contact platform. It’s an old software, starting with version 4.0 back in 1996…
Each new version of Exchange Server brings in new features, both expanding Exchange perimeter and making it easier to deploy and administrate.

Exchange 2007


Introduction of Outlook Anywhere, AKA RPC over HTTP: allows remote users to get connected on Exchange 2007 platform using HTTPs protocol.

Exchange 2010


In example, Exchange 2010 introduced CAS arrays, making client side services high-available and scalable. DAG also brings mail database high-availability. All the client access services required persistence: a user must be sticked to a single CAS server.
Exchange 2010 introduced as well a “layer” between the MAPI RPC clients and the mailbox servers (through the CAS servers), making the failover of a database transparent.

Exchange 2013


Exchange 2013 improved again the changes brought by Exchange 2010: the CAS servers are now state-less and independent from each other (no arrays anymore): no persistence required anymore.
In exchange 2013, raw TCP MAPI RPC services have disappeared and have definitively been replaced by Outlook Anywhere (RPC over HTTP).
Last but not least, SSL offloading does not seem to be allowed for now.

Load-Balancing Microsoft Exchange 2013

First of all, I’m pleased to announce that HAProxy and the ALOHA Load-Balancer are both able to load-balance Exchange 2013 (as well as 2010).

Exchange 2013 Services


As explained in introduction, the table below summarizes the TCP ports and services involved in an Exchange 2013 platform:




TCP PortProtocolCAS Service name (abbreviation)
443HTTPS– Autodiscover (AS)
– Exchange ActiveSync (EAS)
– Exchange Control Panel (ECP)
– Offline Address Book (OAB)
– Outlook Anywhere (OA)
– Outlook Web App (OWA)
110 and 995POP3 / POP3sPOP3
143 and 993IMAP4 / IMAP4sIMAP4

Diagram

There are two main types of architecture doable:
1. All the services are hosted on a single host name
2. Each service owns its own host name

Exhange 2013 and the Single host name diagram


exchange_2013_single_hostname

Exhange 2013 and the Multiple host name diagram


exchange_2013_multiple_hostnames

Configuration

There are two types of configuration with the ALOHA:
Layer 4 mode: the LB act as a router, infrastrcuture intrusive, ability to manage millions of connections
layer 7 mode: the LB act as a reverse-proxy, non-intrusive implementation (source NAT), ability to manage thousands of connections, perform SSL offloading, DDOS protection, advanced persistence, etc…

The present article describe the layer 7 configuration, even if we’re going to use it at layer 4 (mode tcp).

Note that it’s up to you to update your DNS configuration to make the hostname point to your Load-Balancer service Virtual IP.

Template:
Use the configuration below as templates and just change the IP addresses:
bind line to your client facing service IPs
server line IPs to match your CAS servers (and add as many line as you need)
Once updated, just copy/paste the whole configuration, including the default section to the bottom of your ALOHA Layer 7 configuration.

Load-Balancing Exhange 2013 services hosted on a Single host name

######## Default values for all entries till next defaults section
defaults
  option  dontlognull             # Do not log connections with no requests
  option  redispatch              # Try another server in case of connection failure
  option  contstats               # Enable continuous traffic statistics updates
  retries 3                       # Try to connect up to 3 times in case of failure 
  timeout connect 5s              # 5 seconds max to connect or to stay in queue
  timeout http-keep-alive 1s      # 1 second max for the client to post next request
  timeout http-request 15s        # 15 seconds max for the client to send a request
  timeout queue 30s               # 30 seconds max queued on load balancer
  timeout tarpit 1m               # tarpit hold tim
  backlog 10000                   # Size of SYN backlog queue

  balance roundrobin                      #alctl: load balancing algorithm
  mode tcp                                #alctl: protocol analyser
  option tcplog                           #alctl: log format
  log global                              #alctl: log activation
  timeout client 300s                     #alctl: client inactivity timeout
  timeout server 300s                     #alctl: server inactivity timeout
  default-server inter 3s rise 2 fall 3   #alctl: default check parameters

frontend ft_exchange_tcp
  bind 10.0.0.9:443 name https          #alctl: listener https configuration.
  maxconn 10000                         #alctl: connection max (depends on capacity)
  default_backend bk_exchange_tcp       #alctl: default farm to use

backend bk_exchange_tcp
  server cas1 10.0.0.15:443 maxconn 10000 check    #alctl: server cas1 configuration.
  server cas2 10.0.0.16:443 maxconn 10000 check    #alctl: server cas2 configuration.

And the result (LB Admin tab):
– Virtual Service:
aloha_exchange2013_single_domain_virtual_services
– Server Farm:
aloha_exchange2013_single_domain_server_farm

Load-Balancing Exhange 2013 services hosted on Multiple host name

######## Default values for all entries till next defaults section
defaults
  option  dontlognull             # Do not log connections with no requests
  option  redispatch              # Try another server in case of connection failure
  option  contstats               # Enable continuous traffic statistics updates
  retries 3                       # Try to connect up to 3 times in case of failure 
  timeout connect 5s              # 5 seconds max to connect or to stay in queue
  timeout http-keep-alive 1s      # 1 second max for the client to post next request
  timeout http-request 15s        # 15 seconds max for the client to send a request
  timeout queue 30s               # 30 seconds max queued on load balancer
  timeout tarpit 1m               # tarpit hold tim
  backlog 10000                   # Size of SYN backlog queue

  balance roundrobin                      #alctl: load balancing algorithm
  mode tcp                                #alctl: protocol analyser
  option tcplog                           #alctl: log format
  log global                              #alctl: log activation
  timeout client 300s                     #alctl: client inactivity timeout
  timeout server 300s                     #alctl: server inactivity timeout
  default-server inter 3s rise 2 fall 3   #alctl: default check parameters

frontend ft_exchange_tcp
  bind 10.0.0.5:443  name as        #alctl: listener: autodiscover service
  bind 10.0.0.6:443  name eas       #alctl: listener: Exchange ActiveSync service
  bind 10.0.0.7:443  name ecp       #alctl: listener: Exchange Control Panel service
  bind 10.0.0.8:443  name ews       #alctl: listener: Exchange Web Service service
  bind 10.0.0.8:443  name oa        #alctl: listener: Outlook Anywhere service
  maxconn 10000                     #alctl: connection max (depends on capacity)
  default_backend bk_exchange_tcp   #alctl: default farm to use

backend bk_exchange_tcp
  server cas1 10.0.0.15:443 maxconn 10000 check   #alctl: server cas1 configuration.
  server cas2 10.0.0.16:443 maxconn 10000 check   #alctl: server cas2 configuration.

And the result (LB Admin tab):
– Virtual Service:
aloha_exchange2013_multiple_domain_virtual_services
– Server Farm:
aloha_exchange2013_multiple_domain_server_farm

Conclusion


This is a very basic and straight forward configuration. We could make it much more complete and improve timeouts per services, better health checking, DDOS protection, etc…
I may write later articles about Exchange 2013 Load-Balancing with our products.

Related links

Exchange 2013 installation steps
Exchange 2013 first configuration
Microsoft Exchange Server (Wikipedia)
Microsft Exchange official webpage

Links