Category Archives: Virtual Desktop Infrastructure

Microsoft Remote Desktop Services (RDS) Load-Balancing and protection

RDS, RDP, TSE, remoteapp

Whatever you call it, it’s the remote desktop protocol from Microsoft, which has been renamed during the product life.
Basically, it allows users to get connected on remote server and run an application or a full desktop remotely.

It’s more and more fashion nowadays, because it’s easier for IT teams to patch and upgrade a single server hosting many remote desktops than maintaining many laptop individually.

VDI or Virtual Desktop Infrastructure


A Virtual Desktop Infrastructure is an architecture purposely built to export desktops to end users. It can use different type of product, such as Citrix or VMWare View.
Of course, you could build yours using Microsoft RDS as well as the ALOHA Load-Balancer to improve scalability and security of such platform.

Diagram

In a previous article, we already demonstrate how good the ALOHA Load-Balancer is to smartly load-balance Microsoft RDS.
Let’s start again from the same platform (I know, I’m a lazy engineer!):
rdp infrastructure

Windows Broker


When using the ALOHA Load-Balancer, you don’t need a windows broker anymore: the ALOHA is able to balance a new user to the less loaded server, as well as resume a broken session.
When an ALOHA outage occurs and a failover from the master to the slave happens, then all the sessions are resumed: the ALOHAs can share session affinity within a cluster, ensuring nobody will notice the issue.

Making an RDS platforms more secured


It’s easy to load-balance RDS protocol using the mstshash cookie, since microsoft described his protocol: it allows advanced persistence. Basically, each user owns a dedicated mstshash cookie value.

That said, being able to detect weird behavior or service abusers would be much better.

Actually, the ALOHA Load-Balancer allows you to monitor each source IP address, and, much better, each mstshash cookie value.
For any object, we can monitor the connection rate over a period of time, the number of current connection, the bytes in and out (over a period of time), etc…

Now, it’s up to you to protect your VDI infrastructure using the ALOHA Load-Balancer. In example:
  * counting the number of active connections per user (mstshash cookie)
  * counting the number of connection tries over the last 5 minutes per user

The above will allow you to detect the following problems / misbehavior on a VDI platform:
  * a user being abnormally connected too many times
  * a user trying to get connected to quickly, or maybe somebody is trying to discover someone else’s password 😉
  * etc…

Configuration

The configuration below explains how to create a highly-available VDI infrastructure with advanced persistence and improved security:

# for High-Availability purpose
peers aloha
  peer aloha1 10.0.0.16:1024
  peer aloha2 10.0.0.17:1024

# RDP / TSE configuration
frontend ft_rdp
  bind 10.0.0.18:3389 name rdp
  mode tcp
  timeout client 1h
  option tcplog
  option log global
  option tcpka
  # Protection 1
  # wait up to 5s for the mstshash cookie, and reject the client if none
  tcp-request inspect-delay 5s
  tcp-request content accept if RDP_COOKIE

  default_backend bk_rdp

backend bk_rdp
  mode tcp
  balance roundrobin
  # Options
  timeout server 1h
  timeout connect 4s
  option redispatch
  option tcpka
  option tcplog
  option log global
  # sticky persistence + monitoring based on mstshash Cookie
  #    established connection
  #    connection tries over the last minute
  stick-table type string len 32 size 10k expire 1d peers aloha store conn_cur,conn_rate(1m)
  stick on rdp_cookie(mstshash)

  # Protection 2
  # Each user is supposed to get a single active connection at a time, block the second one
  tcp-request content reject if { sc1_conn_cur ge 2 }

  # Protection 3
  # if a user tried to get connected at least 10 times over the last minute, 
  # it could be a brute force
  tcp-request content reject if { sc1_conn_rate ge 10 }

  # Server farm
  server tse1 10.0.0.23:3389 weight 10 check inter 2s rise 2 fall 3
  server tse2 10.0.0.24:3389 weight 10 check inter 2s rise 2 fall 3
  server tse3 10.0.0.25:3389 weight 10 check inter 2s rise 2 fall 3
  server tse4 10.0.0.26:3389 weight 10 check inter 2s rise 2 fall 3

Troubleshouting RDS connections issues with the ALOHA


The command below, when run from your ALOHA Load-Balancer CLI, will print out (and refresh every 1 second) the content of the stick-table with the following information:
  * table name, table size and number of current entries in the table
  * mstshash cookie value as sent by RDS client: field key
  * affected server id: field server_id
  * monitored counter and values, fields conn_rate and conn_cur

while true ; do clear ; echo show table bk_rdp | socat /tmp/haproxy.sock - ; sleep 1 ; done

stick-table output in a normal case


Below, an example of the output when everything goes well:

# table: bk_rdp, type: string, size:20480, used:2
0x912f84: key=Administrator use=1 exp=0 server_id=0 conn_rate(60000)=1 conn_cur=1
0x91ba14: key=XLC\Admin use=1 exp=0 server_id=1 conn_rate(60000)=1 conn_cur=1

stick-table output tracking a RDS abuser

Below, an example of the output when somebody tries many connections to discover someone’s password:

# table: bk_rdp, type: string, size:20480, used:2
0x912f84: key=Administrator use=1 exp=0 server_id=0 conn_rate(60000)=1 conn_cur=1
0x91ba14: key=XLC\Admin use=1 exp=0 server_id=1 conn_rate(60000)=1 conn_cur=1
0x91ca12: key=XLC\user1 use=1 exp=0 server_id=2 conn_rate(60000)=15 conn_cur=0

NOTE: in this case, the client has been blocked and the user user1 from the domain XLC can’t get connected for 1 minute (conn_rate monitoring period). I agree, this type of protection can lead to DOS your users…
We can easily configure the load-balancer to track source IP connection rate and is a single source IP tries to connect too often on our RDS farm, then we can block it before he can access it. (this type of attack would succeed only if the user is not only connected on the VDI platform)
Combining both source IP and mstshash cookie allows us to improve security on the VDI platform.

Securing a RDS platform combining both mstshash cookie and source IP address


The configuration below improve the protection by combining source IP address and RDS cookie.
The purpose is to block an abuser at a lower layer (IP) to avoid him making the Load-Balancer to blacklist regular user, even if this is temporary.

# for High-Availability purpose
peers aloha
  peer aloha1 10.0.0.16:1024
  peer aloha2 10.0.0.17:1024

# RDP / TSE configuration
frontend ft_rdp
  mode tcp
  bind 10.0.0.18:3389 name rdp
  timeout client 1h
  option tcpka
  log global
  option tcplog
  default_backend bk_rdp

backend bk_rdp
  mode tcp
  balance roundrobin

  # Options
  timeout server 1h
  timeout connect 4s
  option redispatch
  option tcpka
  log global
  option tcplog

  # sticky persistence + monitoring based on mstshash Cookie
  #    established connection
  #    connection tries over the last minute
  stick-table type string len 32 size 10k expire 1d peers aloha store conn_cur,conn_rate(1m)
  stick on rdp_cookie(mstshash)

  # Protection 1
  # wait up to 5s for the mstshash cookie, and reject the client if none
  tcp-request inspect-delay 5s
  tcp-request content accept if RDP_COOKIE
  tcp-request content track-sc1 rdp_cookie(mstshash) table bk_rdp
  tcp-request content track-sc2 src                  table sourceip

# IP layer protection
  #  Protection 2
  #   each single IP can have up to 2 connections on the VDI infrastructure
  tcp-request content reject if { sc2_conn_cur ge 2 }

  #  Protection 3
  #   each single IP can try up to 5 connections in a single minute
  tcp-request content reject if { sc2_conn_rate ge 10 }

# RDS / TSE application layer protection
  #  Protection 4
  #   Each user is supposed to get a single active connection at a time, block the second one
  tcp-request content reject if { sc1_conn_cur ge 2 }

  #  Protection 5
  #   if a user tried to get connected at least 10 times over the last minute, 
  #   it could be a brute force
  tcp-request content reject if { sc1_conn_rate ge 10 }

  # Server farm
  server tse1 10.0.0.23:3389 weight 10 check inter 2s rise 2 fall 3
  server tse2 10.0.0.24:3389 weight 10 check inter 2s rise 2 fall 3
  server tse3 10.0.0.25:3389 weight 10 check inter 2s rise 2 fall 3
  server tse4 10.0.0.26:3389 weight 10 check inter 2s rise 2 fall 3

backend sourceip
 stick-table type ip size 20k store conn_cur,conn_rate(1m)

Note that if you have remote sites using your VDI infrastructure, it is also possible to manage a source IP white list. And the same for the mstshash cookies: you can allow some users (like Administrator) to get connected multiple time on the VDI infrastructure.

Source IP and mstshash cookie combined protection


To observe the content of both tables (and still refreshed every 1s), just run the command below on your ALOHA CLI:

while true ; do clear ; echo show table bk_rdp | socat /tmp/haproxy.sock - ; echo "=====" ; echo show table sourceip | socat /tmp/haproxy.sock - ; sleep 1 ; done

And the result:

# table: bk_rdp, type: string, size:20480, used:2
0x23f1424: key=Administrator use=1 exp=0 server_id=0 conn_rate(60000)=0 conn_cur=1
0x2402904: key=XLC\Admin use=1 exp=0 server_id=1 conn_rate(60000)=1 conn_cur=1

=====
# table: sourceip, type: ip, size:20480, used:2
0x23f9f70: key=10.0.4.215 use=2 exp=0 conn_rate(60000)=2 conn_cur=2
0x23f1500: key=10.0.3.20 use=1 exp=0 conn_rate(60000)=0 conn_cur=1

Important Note about Microsoft RDS clients


As Mathew Levett stated on LoadBalancer.org’s blog (here is the article: http://blog.loadbalancer.org/microsoft-drops-support-for-mstshash-cookies/), latest version of Microsoft RDS clients don’t send the full user’s information into the mstshash cookie. Worst, sometime, the client don’t even send the cookie!!!!

In the example we saw above, there was 2 users connected on the infrastructure: the Administrator user was connected from a Linux client using xfreerdp tool, and we saw the cookie properly in the stick table. Second user, Administrator, from the Windows domain XLC, as his credential truncated in the stick table (Value: XLC\Admin), it was getting connected using Windows 2012 Server Enterprise RDS client.

As lb.org said, hopefully Microsoft will fix this issue. Or maybe there are some alternative RDS client for Windows…

To bypass this issue, we can take advantage of the ALOHA: we can do a persistence based on the source IP address when there is no mstshash cookies in the client connection information.
Just add the line below, right after the stick on rdp_cookie(mstshash):

  stick on src

With such configuration, the ALOHA Load-Balancer will stick on mstshash cookie, if present, and will failover on source IP address otherwise.

Conclusion


Virtualisation Desktop Infrastructure is a very fashion way of providing access to end users to a company’s applications.
If you don’t use the right product to build your RDS / TSE / RemoteApp infrastructure, you may be in trouble at some point.

The ALOHA Load-Balancer, thanks to its flexible configuration has many advantages

You can freely give a try to the ALOHA Load-Balancer, it is available from our website for download, allowing up to 10 connections with no time limitation: Download the ALOHA Load-Balancer

Links

Microsoft Terminal Server / remoteapp load-balancing

Synopsis

You need to load-balance Microsoft Terminal Services or remoteapps.
Both of them use the RDP protocol.

One main issue when using RDP is to be able to stick a user to a server.
The configuration below shows you how you can build outage-free Terminal server infrastructure thanks to the Aloha and HAProxy.

Diagram

The TSE infrasctructure looks like the image below:
rdp infrastructure

Basically, we have a couple of Aloha load-balancer, in active/passive mode.
The Alohas share a VIP, 10.0.0.18, on which clients will get connected. In order to ensure stickiness, the Alohas will use the RDP cookie, store it in a table where it is associated to the server serving the request.
Finally, this table is shared between both Alohas, in order to ensure safe failover.

Configuration

find below the configuration for such infrastructure:

peers aloha
  peer aloha1 10.0.0.16:1024
  peer aloha2 10.0.0.17:1024

# RDP / TSE configuration
frontend ft_rdp
 mode tcp
 bind 10.0.0.18:3389 name rdp
 timeout client 1h
 option tcpka
 option tcplog
 log global
 # wait up to 5s for an RDP cookie in the request
 tcp-request inspect-delay 5s
 tcp-request content accept if RDP_COOKIE
 default_backend bk_rdp

backend bk_rdp
 mode tcp
 balance rdp-cookie
 persist rdp-cookie

 # Options
 timeout server 1h
 timeout connect 4s
 option redispatch
 option tcpka
 option tcplog
 log global

  # sticky persistence
  stick-table type string len 32 size 10k expire 1d peers aloha
  stick on rdp_cookie(mstshash)

  # Server farm
  server tse1 10.0.0.23:3389 weight 10 check inter 2s rise 2 fall 3
  server tse2 10.0.0.24:3389 weight 10 check inter 2s rise 2 fall 3
  server tse3 10.0.0.25:3389 weight 10 check inter 2s rise 2 fall 3
  server tse4 10.0.0.26:3389 weight 10 check inter 2s rise 2 fall 3

Now, the users can point their RDP client to the VIP and get load-balanced.
In case of client side network outage or Aloha failover, users will keep on being redirected to the same server.
The users won’t suffer any application outage.

Links