Cisco CSS Load Balancer – Traffic Drop

I just wanted to comment here on a problem I had which took me a while to understand and i couldn’t find any solution anywhere.

The setup is a cisco CSS 11500 Series with 2 web servers and 2 application server.

it was a Cisco ASA Firewall (192.168.1.1) -> Cisco CSS (192.168.1.2)

|-> Servers (192.168.1.10,11,12,13)

So the cisco CSS and the servers were sitting on the same network as the internal interface of the ASA Firewall. servers ofcourse has the CSS as the default gateway.

The application content group was configured with sticky sessions at the time.

The problem was that now and then our connection to the website was dropped and we couldn’t connect back for few minutes and then it was back up, during the time the website was accessible from anywhere else.

This problem drove us nuts, and took us a while to resolve this.

CSS was noticing quite a lot SYN ATTACKS. (CSS never drop traffic because of a DOS attack its just marks them )

so we were thinking maybe there’s a DDoS mitigation device along the way suspecting an attack and blocking the traffic, that was not it.

The problem was an ICMP redirect packets sent from the CSS telling the servers about a better route for the external addresses, the firewall.

for example coming from IP 2.2.2.34 the CSS would send an ICMP redirect message to the servers (windows server 2003) the windows machine would then inject this route in the routing table.

traffic would come from the CSS to the server , packets back were sent to the firewall and got lost there. this was causing the blockage.

By default, ICMP redirect routes were stayed in the routing table for 10 minutes , and then removed.

so to avoid this from happening, you either disable ICMP redirects on the CSS, (I don’t know why its turned on by default anyway)

or change a registry value in windows to disable icmp redirects route injection,(possible to disable in unix as well)

HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Services\\Tcpip\\Parametersby

change the setting of EnableICMPRedirect entry to 0 (in most cases it will set to active – 1 by default)

Hope that would help others as well,

Lior