Howto transparent proxying and binding with HAProxy and ALOHA Load-Balancer

Transparent Proxy

HAProxy works has a reverse-proxy. Which means it maintains 2 connections when allowing a client to cross it:
  – 1 connection between HAProxy and the client
  – 1 connection between HAProxy and the server

HAProxy then manipulate buffers between these two connections.
One of the drawback of this mode is that HAProxy will let the kernel to establish the connection to the server. The kernel is going to use a local IP address to do this.
Because of this, HAProxy “hides” the client IP by its own one: this can be an issue in some cases.
Here comes the transparent proxy mode: HAProxy can be configured to spoof the client IP address when establishing the TCP connection to the server. That way, the server thinks the connection comes from the client directly (of course, the server must answer back to HAProxy and not to the client, otherwise it can’t work: the client will get an acknowledge from the server IP while it has established the connection on HAProxy‘s IP).

Transparent binding

By default, when one want HAProxy to get traffic, we have to tell it to bind an IP address and a port.
The IP address must exist on the operating system (unless you have setup the sysctl net.ipv4.ip_nonlocal_bind) and the OS must announce the availability to the other devices on the network through ARP protocol.
Well, in some cases we want HAProxy to be able to catch traffic on the fly without configuring any IP address or VRRP or whatever…
This is where transparent binding comes in: HAProxy can be configured to catch traffic on the fly even if the destination IP address is not configured on the server.
These IP addresses will never be pingable, but they’ll deliver the services configured in HAProxy.

HAProxy and the Linux Kernel

Unfortunately, HAProxy can’t do transparent binding or proxying alone. It must stand on a compiled and tuned Linux Kernel and operating system.
Below, I’ll explain how to do this in a standard Linux distribution.
Here is the check list to meet:
  1. appropriate HAProxy compilation option
  2. appropriate Linux Kernel compilation option
  3. sysctl settings
  4. iptables rules
  5. ip route rules
  6. HAProxy configuration

HAProxy compilation requirements


First of all, HAProxy must be compiled with the option TPROXY enabled.
It is enabled by default when you use the target LINUX26 or LINUX2628.

Linux Kernel requirements

You have to ensure your kernel has been compiled with the following options:
  – CONFIG_NETFILTER_TPROXY
  – CONFIG_NETFILTER_XT_TARGET_TPROXY

Of course, iptables must be enabled as well in your kernel 🙂

sysctl settings

The following sysctls must be enabled:
  – net.ipv4.ip_forward
  – net.ipv4.ip_nonlocal_bind

iptables rules


You must setup the following iptables rules:

iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT

Purpose is to mark packets which matches a socket bound locally (by HAProxy).

IP route rules


Then, tell the Operating System to forward packets marked by iptables to the loopback where HAProxy can catch them:

ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

HAProxy configuration


Finally, you can configure HAProxy.
  * Transparent binding can be configured like this:

[...]
frontend ft_application
  bind 1.1.1.1:80 transparent
[...]

  * Transparent proxying can be configured like this:

[...]
backend bk_application
  source 0.0.0.0 usesrc clientip
[...]

Transparent mode in the ALOHA Load-Balancer


Now, the same steps in the ALOHA Load-balancer, which is an HAProxy based load-balancing appliance:
  1-5. not required, the ALOHA kernel is deeply tuned for this purpose
  6. HAProxy configuration

LB Admin tab (AKA click mode)

  * Transparent binding can be configured like this, when editing a Frontend listener:
frontend_listener_transparent

  * Transparent proxying can be configured like this when editing a farm:
backend_transparent

LB Layer 7 tab (vi in a browser mode)


  * Transparent binding can be configured like this:

[...]
frontend ft_application
  bind 1.1.1.1:80 transparent
[...]

  * Transparent proxying can be configured like this:

[...]
backend bk_application
  source 0.0.0.0 usesrc clientip
[...]

Links

9 thoughts on “Howto transparent proxying and binding with HAProxy and ALOHA Load-Balancer”

  1. am having an issue with this. I have setup a galera cluster, 5 machines total 2 with haproxy and 3 with mariadb 10. These are all rhel 7 machines.

    The kernel is version 3.10, with 1.5.14 of haproxy. Looking at the documentation this tells me this should work, as it is enabled by default in the kernel and the target system for haproxy should enable it.

    when I add the line:

    source 0.0.0.0 usersrc clientip

    As well as the iptables and route rules I start to get the error :

    ERROR 2013 (HY000): Lost connection to MySQL server at ‘reading initial communication packet’, system error: 0 “Internal error/check (Not system error)”

    This is my haproxy configuration file


    # This file managed by Puppet
    global
    chroot /var/lib/haproxy
    daemon
    group haproxy
    log 10.76.10.99 local0
    maxconn 4000
    pidfile /var/run/haproxy.pid
    stats socket /var/lib/haproxy/stats
    # user haproxy

    defaults
    log global
    maxconn 8000
    option httplog
    option dontlognull
    option http-server-close
    option forwardfor except 127.0.0.0/8
    option redispatch
    retries 3
    stats enable
    timeout queue 1m
    timeout connect 10s
    timeout client 10m
    timeout server 10m
    timeout check 10s

    frontend mysqlcl
    bind 10.76.10.105:3306 transparent
    mode tcp
    default_backend galera-cluster

    backend galera-cluster
    balance roundrobin
    option httpchk
    source 0.0.0.0 usesrc clientip
    server mysqlcl00.lib.ncsu.edu 10.76.10.101:3306 check fall 2 inter 5000 maxconn 5000 port 9200 rise 2 weight 1
    server mysqlcl01.lib.ncsu.edu 10.76.10.102:3306 check fall 2 inter 5000 maxconn 5000 port 9200 rise 2 weight 1
    server mysqlcl02.lib.ncsu.edu 10.76.10.103:3306 check fall 2 inter 5000 maxconn 5000 port 9200 rise 2 weight 1

    listen mysqlcl-web
    bind 10.76.10.105:80
    balance roundrobin
    option tcplog
    option ssl-hello-chk

          1. Do you happen to have a link to a document for the best way to do that? When I try and set that up I get ERROR 2013 (HY000): Lost connection to MySQL server at ‘reading initial communication packet’, system error: 0 “Internal error/check (Not system error)”, just in testing before I even set the gateway to the haproxy servers.

Leave a Reply

Your email address will not be published. Required fields are marked *