Hello There!

One of the exciting new technologies to come out in the last few years is a tremendously efficient and dynamic caching system called Varnish (see : http://www.varnish-cache.org).

We have been employing the use of Varnish for high traffic websites for the purposes of user experience improvements as well as for redundancy and load balancing purposes.

Varnish can do it all – complex load balancing and polling based on many different weighting methodologies for fail over, as well as holding on to a “stale” cache in the event of a back end web server outage, or perhaps for geographic redundancy (holding a stale cache in a secondary data center).

One of the challenges we have faced in the many different implementations of varnish into web stacks, is dealing with dynamic and user session (i.e. “logged in”) content.

If the Internet was full of only static (see 1995) html files, varnish would work beautifully out of the box. Unfortunately the web is a complicated mess of session based authentication, POSTS, GETS and query strings among a few things.

One of our recent accomplishments was getting the Joomla 1.5 content management system to work with Varnish 2.1.

The biggest challenge for Joomla was that it creates a session cookie for all users. This means the session is created and established for any guest visiting the site, and if they decide to log in , that same session is used to establish a logged in session through authentication. This is an apparent effort to deter or avoid session hijacking.

The problem with this is that Varnish ends up caching all the logged in session content, as well as the anonymous front page content.

I spent a significant amount of time fine tuning my VCL (varnish configuration language) to play nice with Joomla. Unfortunately it became apparent that some minor modifications to the Joomla code was necessary in order for it to communicate properly with Varnish.

Step 1 : Move the login form off the front page

I realize this might be a hard decision. I cant offer an alternative. If you have an integrated login form on the front page of your site, and you wish to cache that page with varnish, you will likely have to chose one or the other. It would probably be ideal to replace that login form with a button to bring the user to a secondary page off the main page.

For the sake of argument, lets call our site “example.com” and the login page url within Joomla should look like the following :

http://www.example.com/index.php?option=com_user&view=login

Take note of login URI in this string.

The reason we need the login form on a secondary page is because we need an almost “sandboxed” section of the site where the anonymous session cookie can be established, and passed through the authentication process to a logged in session. We will tell varnish to essentially ignore this page.

Step 2 : Modify Joomla to send HTTP headers for user/guest sessions

This isn’t that hard. In the Joomla code, there is a section where it defines the HTTP headers it sends to the browser for cache variables such as expire times and whatnot. I’m going to assume you have turned off the built-in Joomla caching system.

What you need to do is tell Joomla to send a special HTTP header that will give either a True or False value if the user is logged in or not. This is useful information. It will allow varnish to not cache any logged in content such as “Welcome back, USERNAME” after the user is passed back to the front page from logging in.

In my joomla installation, I modified the following file :

The parent folder being the public_html / root folder for your Joomla installation. In this file, please find the line that determines if the Joomla caching system is disabled :

After this line, you will see about 5 HTTP header declarations (expires, last-modified, cache-control, cache-control again and pragma). Above those declarations , add the following 6 lines of code :

If you read the above code, its fairly straight forward. I do a check to see if the user is a guest (aka anonymous) or not. If they are logged in I send an HTTP header called “X-Logged-In”, and assign a “True” value to it. If the user is not logged in, it sets it to “False”.

Pretty easy, right?

This will allow varnish to avoid caching a logged in user’s page.

Step 3 : Configure Varnish

This is the part that took the most time during this entire process. Mind you patching the Joomla code and whatnot took some time as well, this process took a lot of experimentation and long hours examining session cookies and host headers.

What I will do is break down the generalized configuration directives into two groups : VCL_RECV and VCL_FETCH.

VCL_RECV

In here, I set a bunch of IF statement directives to tell varnish what it should look up in the cache and what it should pipe to the backend and what it should pass. This could probably be optimized and improved upon, but it works for me :

VCL_FETCH

The fetch section is a little bit easier. I only have about 5 directives. The first one is the most important one you want to look at. It “unsets” the cookie from any page on the site, EXCEPT the login page. This allows varnish to properly establish the logged in session. The subsequent rules determine what to deliver and what to pass based on URI or HTTP header checks :

Thats it! I just saved you many sleepless nights (I hope!). Hopefully your headers will look something like this after you implement varnish in front of Joomla :

UPDATE : 12/08/2011

I realize I made a mistake and have corrected this post. In vcl_fetch, i had the following :

Well I realize I should be unsetting the response cookie, not the set cookie. For some reason, the above (erroneous) directive works only right after you login. If you start clicking around the site, your logged in session disappears. I suspect this is because either joomla or varnish is mistakenly unsetting a logged in session.

This is the correct entry (I have fixed it in my original post as well) :

After making the above change, I can login and browse the site and my session stays intact. Mind you, the Joomla site I am testing with is definitely not a vanilla Joomla installation.

I’d love to hear from anyone who has accomplished the above scenario either way!

This post is intended to be a general guide for configuring “stickied” load balanced HTTP servers. Whether it’s F5 load balancers, foundry load balancers or open source based load balancers (keepalived/lvs), the concepts are the same and can be migrated across said platforms.

If you have a paid of foundry’s and are looking to configure stickied load balanced HTTP servers, hopefully this guide will provide some assistance.

Logging into the load balancer

Telnet to the box and ‘enable’ to allow admin access. The first thing you want to do is show the current configuration to view the existing setup for other working boxes :

Real servers : defining the multiple load balanced boxes

Show the existing configuration on the foundary :

Take a look at the configuration of two “real” servers, which are the two servers that are behind the load balancer that will have balanced sticky connections :

The above example is balancing TCP 8001 traffic, which is for TOMCAT. Here are entries for two servers doing simple HTTP traffic :

This example is similar to the tomcat example, except you have several options. “port default disable” disables all other ports. “port http keepalive” and “port http url “HEAD /”" define the http checks that take place to ensure apache is running on that box. If not , it will fail over to the second box and stop sending traffic to it.

SSL Connections

SSL incoming connections are handled by the load balancer initially, then passed off to the actual server as regular http / port 80 traffic. The internal box configuration would be similar to the above configuration examples :

Configuring the external IP to NAT to the internal virtual

Typically, you will have a firewall in front of the load balancer that actaully holds the external ip addresses. The traffic is filtered initially by the firewall, then NAT’d to the virtual ip (VIP) of the load balancer, which then handles balancing the traffic.

You will need to either establish a new external ip , or use an existing one (for instance, if you are moving from 1 web server to 2 web servers , and want to balance the traffic using the load balancer). You need to setup the external IP address, and NAT it to the internal VIP.

Verifying the configuration works

Once everything is setup properly, and the external IP is being NAT’d to the load balancer, it is time to ensure the load balancer is seeing the connections. You could do this before doing the switchover on the firewall as well, just to ensure everything looks right before actually doing the switchover.

To see the active connections being load balanced, issue the following command (replacing the servername for whichever one you want to check) :

That should display information similar to this :

The above is displaying the specific connection details for a single real server. To check the VIP / Virtual server :

Which will display the following :

You can see that “ServerConn” is displaying 46 connections. Thats it!