Load Balancer best practice

working closely with load balancers throughout the years made me aware of quite a few difficulties and tricks, and hard learned lessons about the problems and the best ways of implementing a load balancer into your web farm.

I hope you enjoy the things i have to say here, and i will listen to what you have to say , correct and update this article.

1. Web application, clock synchronization.

In load sharing setup, one must not count on the servers clocks to be used for writing data to the database or presenting data to the user.

The clock of every server skew, some servers skew more than others, so even a clock sync of once a day, doesn’t make it sure that the servers are synchronized, lets look at the example:

User login to your website, your application took the server’s time and inserted the login time as 14:00.

Now the user submits a form, and the second request is being sent through a different server, now another entry is being sent to the database at 13:59, because  there was a few seconds difference between the clocks.

A later action was registered in the database as if it happened before the earlier one.

The best way to avoid it, is to use the database clock for all read and write actions, that ensures a steady time across various actions, through no matter how many servers you have in the pool.

2. application sessions type

If your application uses sessions, its better to use database based sessions, this will keep the user session alive no matter which server in your pool the request is being made from.

be sure to keep your servers equally loaded. you should be able to set your content group up as round robin with no stickiness.

If you must use server based session, then you must setup some kind of stickiness on your Load Balancer content group, so if you do, make sure the stickiness is based on source IP AND source port.

Pay attention, with some Load Balancers, when you chose sticky source IP they disregard the source port, this result in all requests coming from a certain IP to go to one server regardless of the amount of different requests, or different people initiating the requests, make sure the stickiness is based on source port as well, This will avoid search engines bots to load one of the servers while indexing your site, this could cause lack of performance on the loaded server.

4.server keep alive

if your Load Balancer support keep alive URI, which being used by the Load Balancer to make sure the server is alive, be sure to use a keep alive file which also uses the servers scripting engine(.php,.aspx,etc…) sometimes there is a problem with the scripting engine while the server itself is fine and still handing out responses, and you want that server to be removed from the pool and not resulting in application errors for your users.

Using a URI keep alive rather than other methods such as icmp, gives you the possibility to remove a server from the pool easily by renaming the keep alive file that the Load Balancer is checking for.

This can be used for maintenance, version updates, without doing any action on the Load Balancer itself.

5. Alerts

If possible, configure email/snmp traps alerts for major incidents, such as server unavailable, service unavailable,etc.

This helps identifying problems in the system, and to give you information when your environment status is being degraded.

6. Monitoring

Graph your interfaces, the bandwidth of each service, the hits of each service, and anything else you can possibly graph.

When investigating a fault, graphs of past behavior will help you to understand what could possibly went wrong.

7. Access lists

Setup access lists to restrict traffic going through the Load Balancer, maybe you are serving different internal networks, some Load Balancers by default behave like a router between different networks, which results in an unfiltered traffic between supposedly filtered networks.

Put explicitly allow access lists on all interfaces, and use logging, this will help identify access problems and/or unwanted traffic generated on the network.

Feel free to comment,

Lior.