Tag Archives: smtp

Efficient SMTP relay infrastructure with Postfix and load-balancers

Scalable architecture

In order to make your architecture scalable, you may often want to use a load-balancer or an application delivery controller.
When using one of them (or a reverse-proxy), the client information is almost all the time hidden. Or if you want to get them, it requires huge modifications in the architecture.

Unfortunately, for troubleshooting and security purpose it would be interesting to know the client information (mainly source IP address)…
That’s where the proxy protocol comes in.

The proxy protocol???

As explained in a previous article“preserve source ip address despite reverse proxies”, the proxy protocol was developped to maintain client information when chaining proxies and reverse-proxies.
Two main advantages when using it:

  • you can provide a downstream proxy or server (aka next hop) the client information (for now, mainly IP and port source)
  • you can use servers in multiple datacenter without a complex network architecture (just need to provide routing for a TCP connection)

Why simply not using TPROXY (transparent proxy) mode?


TPROXY allows a load-balancer or reverse-proxy to open the TCP connection to the server using the client IP address.
One of the drawback of TPROXY is that the default gateway of the application servers must be the load-balancer.
Or you must do policy based routing on your network which could be painfull.

Why Postfix and HAProxy?

HAProxy was the first software to implement the proxy protocol.
Note that you’ll have to use HAProxy 1.5 branch or patched HAProxy 1.4.
< advertisement >
An other solution would to use Aloha load-balancer which does everything for you in a box (from the OS to HAPrxoxy) with all the nice features you could expect. 😉
< /advertisement >

Lately, Postfix implemented it. It is available in Postfix 2.10.
It is the first application server first application server to ship with it: THANKS and CONGRATULATION!!!!
Hopefully other MTAs will implement it soon. It is simple and brings so many improvements to an architecture.

SMTP, spam and securtiy


In SMTP, it is really important to know the client IP, since we use it most of the time through RBL to fight spam.
For security purpose as well: we may want to allow only some hosts to use our SMTP relays and block any other clients.
Without the proxy protocol, the load-balancer will hide the client IP with its own IP. You would have to maintain whitelists into the load-balancer (which is doable). Thanks to proxy protocol, Postscreen would be aware of the client IP, it means you could maintain lists directly into the MTA.

HAProxy and Postfix connection flow

The diagram below shows the protocols and the process in place in this kind of architecture:

           smtp              proxy-protocol
                             + smtp
(INTERNET) ---> 25 (HAPROXY)      --->      srv1:10024 (Postscreen
                                                       / smtpd)
                                  --->      srv2:10024 (Postscreen
                                                       / smtpd)

Note that the default gateway of the MTA servers is not anymore the load-balancer.
Both servers migt be in the same LAN or datacenter. Any type of architecture is now doable.

Configuration

HAProxy

frontend ft_smtp
  bind 0.0.0.0:25
  mode tcp
  no option http-server-close
  timeout client 1m
  log global
  option tcplog
  default_backend bk_postfix

backend bk_postfix
  mode tcp
  no option http-server-close
  log global
  option tcplog
  timeout server 1m
  timeout connect 5s
  server postfix 127.0.0.1:10024 send-proxy

Postfix

Note: I installed postfix in /opt/postfix directory

main.cf

queue_directory = /opt/postfix/var/spool/postfix
command_directory = /opt/postfix/usr/sbin
daemon_directory = /opt/postfix/usr/libexec/postfix
data_directory = /opt/postfix/var/lib/postfix
mail_owner = postfix
unknown_local_recipient_reject_code = 550
inet_interfaces = localhost
sendmail_path = /opt/postfix/usr/sbin/sendmail
newaliases_path = /opt/postfix/usr/bin/newaliases
mailq_path = /opt/postfix/usr/bin/mailq
setgid_group = postdrop
html_directory = no
manpage_directory = /opt/postfix/usr/local/man
sample_directory = /opt/postfix/etc/postfix
readme_directory = no
inet_protocols = ipv4
postscreen_upstream_proxy_protocol = haproxy

master.cf

10024     inet  n       -       n       -       1       postscreen
smtpd     pass  -       -       n       -       -       smtpd

See the results in Postfix logs

No proxy protocol

Jun 30 01:18:14 sd-33932 postfix/postscreen[2375]: 
       CONNECT from [127.0.0.1]:52841 to [127.0.0.1]:10024
Jun 30 01:18:22 sd-33932 postfix/smtpd[2376]: 
       disconnect from localhost[127.0.0.1]

With proxy protocol

Jun 29 09:13:41 sd-33932 postfix/postscreen[30505]: 
       CONNECT from [<client public IP>]:59338 to [<server IP>]:25
Jun 29 09:13:52 sd-33932 postfix/postscreen[30505]: 
       DISCONNECT [<client public IP>]:59338

Related Links

Links

Enhanced SSL load-balancing with Server Name Indication (SNI) TLS extension

Synopsis

Some time ago, we wrote an article which explained how to load-balance SSL services, maintaining affinity using the SSLID.
The main limitation of this kind of architecture is that you must dedicate a public IP address and port per service.
If you’re hosting web or mail services, you could run out of public IP address quickly.

TLS protocol has been extended in 2003, RFC 3546, by an extension called SNI: Server Name Indication, which allows a client to announce in clear the server name it is contacting.

NOTE: two RFC have obsoleted the one above, the latest one is RFC 6066

The Aloha Load-balancer can use this information to choose a backend or a server.
This allows people to share a single VIP for several services.

Of course, we can use SNI switching with SSLID affinity to build a smart and reliable SSL load-balanced platform.

NOTE: Server Name information is sent with each SSL Handshake, whether you’re establishing a new session or you’re resuming an old one.

SNI is independent from the protocol used at layer 7. So basically, it will work with IMAP, HTTP, SMTP, POP, etc…

Limitation

Bear in mind, that in 2012, not all clients are compatible with SNI.
Concerning web browsers, a few of used in 2012 them are still not compatible with this TLS protocol extension.

We strongly recommend you to read the Wikipedia Server Name Indication page which lists all the limitation of this extension.

  • Only HAProxy nightly snapshots from 8th of April are compatible (with no bug knows) with it.
  • Concerning Aloha, it will be available by Aloha Load-balancer firmware 5.0.2.

Diagram

The picture below shows a platform with a single VIP which host services for 2 applications:
sni_loadbalancing

We can use SNI information to choose a backend, then, inside a backend, we can use SSLID affinity.

Configuration

Choose a backend using SNI TLS extension


The configuration below matches names provided by the SNI extention and choose a farm based on it.
In the farm, it provides SSLID affinity.
If no SNI extention is sent, then we redirect the user to a server farm which can be used to tell the user to upgrade its software.

# Adjust the timeout to your needs
defaults
  timeout client 30s
  timeout server 30s
  timeout connect 5s

# Single VIP with sni content switching
frontend ft_ssl_vip
  bind 10.0.0.10:443
  mode tcp

  tcp-request inspect-delay 5s
  tcp-request content accept if { req_ssl_hello_type 1 }
  
  acl application_1 req_ssl_sni -i application1.domain.com
  acl application_2 req_ssl_sni -i application2.domain.com

  use_backend bk_ssl_application_1 if application_1
  use_backend bk_ssl_application_2 if application_2

  default_backend bk_ssl_default

# Application 1 farm description
backend bk_ssl_application_1
  mode tcp
  balance roundrobin

  # maximum SSL session ID length is 32 bytes.
  stick-table type binary len 32 size 30k expire 30m

  acl clienthello req_ssl_hello_type 1
  acl serverhello rep_ssl_hello_type 2

  # use tcp content accepts to detects ssl client and server hello.
  tcp-request inspect-delay 5s
  tcp-request content accept if clienthello

  # no timeout on response inspect delay by default.
  tcp-response content accept if serverhello

  stick on payload_lv(43,1) if clienthello

  # Learn on response if server hello.
  stick store-response payload_lv(43,1) if serverhello

  option ssl-hello-chk
  server server1 192.168.1.1:443 check
  server server2 192.168.1.2:443 check

# Application 2 farm description
backend bk_ssl_application_2
  mode tcp
  balance roundrobin

  # maximum SSL session ID length is 32 bytes.
  stick-table type binary len 32 size 30k expire 30m

  acl clienthello req_ssl_hello_type 1
  acl serverhello rep_ssl_hello_type 2

  # use tcp content accepts to detects ssl client and server hello.
  tcp-request inspect-delay 5s
  tcp-request content accept if clienthello

  # no timeout on response inspect delay by default.
  tcp-response content accept if serverhello

  stick on payload_lv(43,1) if clienthello

  # Learn on response if server hello.
  stick store-response payload_lv(43,1) if serverhello

  option ssl-hello-chk
  server server1 192.168.2.1:443 check
  server server2 192.168.2.2:443 check

# Sorry backend which should invite the user to update its client
backend bk_ssl_default
  mode tcp
  balance roundrobin
  
  # maximum SSL session ID length is 32 bytes.
  stick-table type binary len 32 size 30k expire 30m

  acl clienthello req_ssl_hello_type 1
  acl serverhello rep_ssl_hello_type 2

  # use tcp content accepts to detects ssl client and server hello.
  tcp-request inspect-delay 5s
  tcp-request content accept if clienthello

  # no timeout on response inspect delay by default.
  tcp-response content accept if serverhello

  stick on payload_lv(43,1) if clienthello

  # Learn on response if server hello.
  stick store-response payload_lv(43,1) if serverhello

  option ssl-hello-chk
  server server1 10.0.0.11:443 check
  server server2 10.0.0.12:443 check

Choose a server using SNI: aka SSL routing


The configuration below matches names provided by the SNI extention and choose a server based on it.
If no SNI is provided or we can’t find the expected name, then the traffic is forwarded to server3 which can be used to tell the user to upgrade its software.

# Adjust the timeout to your needs
defaults
  timeout client 30s
  timeout server 30s
  timeout connect 5s

# Single VIP 
frontend ft_ssl_vip
  bind 10.0.0.10:443
  mode tcp

  tcp-request inspect-delay 5s
  tcp-request content accept if { req_ssl_hello_type 1 }

  default_backend bk_ssl_default

# Using SNI to take routing decision
backend bk_ssl_default
  mode tcp

  acl application_1 req_ssl_sni -i application1.domain.com
  acl application_2 req_ssl_sni -i application2.domain.com

  use-server server1 if application_1
  use-server server2 if application_2
  use-server server3 if !application_1 !application_2

  option ssl-hello-chk
  server server1 10.0.0.11:443 check
  server server2 10.0.0.12:443 check
  server server3 10.0.0.13:443 check

Related Links

Links

Send users to the same server for imap and smtp

Synopsis

You want to use Aloha load balancer in front of a mail platform, relying on HAProxy to achieve load-balancing.
You mail platform delivers IMAP acts as a SMTP relay for your users.
You want your users to use the same server for both protocols in order to take advantage of different caches level (FS cache, authentication, etc…)

Configuration

The HAProxy configuration below explain how to force the SMTP relay for IMAP connected users.
Users relaying mail through SMTP will be connected to any server in the farm using round robin.

frontend ft_smtp
	bind :25
	mode tcp
	default_backend bk_smtp

frontend ft_imap
	bind :143
	mode tcp
	default_backend bk_imap

backend bk_smtp
	mode tcp
	balance roundrobin
	stick match src table bk_imap
	server s1 192.168.1.1:25 check
	server s2 192.168.1.2:25 check

backend bk_imap
	mode tcp
	# imap connections are usually long
	balance leastconn
	stick store-request src
	stick-table type ip size 200k expire 30m
	server s1 192.168.1.1:143 check
	server s2 192.168.1.2:143 check

Links