Hello There!
One of the exciting new technologies to come out in the last few years is a tremendously efficient and dynamic caching system called Varnish (see : http://www.varnish-cache.org).
We have been employing the use of Varnish for high traffic websites for the purposes of user experience improvements as well as for redundancy and load balancing purposes.
Varnish can do it all – complex load balancing and polling based on many different weighting methodologies for fail over, as well as holding on to a “stale” cache in the event of a back end web server outage, or perhaps for geographic redundancy (holding a stale cache in a secondary data center).
One of the challenges we have faced in the many different implementations of varnish into web stacks, is dealing with dynamic and user session (i.e. “logged in”) content.
If the Internet was full of only static (see 1995) html files, varnish would work beautifully out of the box. Unfortunately the web is a complicated mess of session based authentication, POSTS, GETS and query strings among a few things.
One of our recent accomplishments was getting the Joomla 1.5 content management system to work with Varnish 2.1.
The biggest challenge for Joomla was that it creates a session cookie for all users. This means the session is created and established for any guest visiting the site, and if they decide to log in , that same session is used to establish a logged in session through authentication. This is an apparent effort to deter or avoid session hijacking.
The problem with this is that Varnish ends up caching all the logged in session content, as well as the anonymous front page content.
I spent a significant amount of time fine tuning my VCL (varnish configuration language) to play nice with Joomla. Unfortunately it became apparent that some minor modifications to the Joomla code was necessary in order for it to communicate properly with Varnish.
Step 1 : Move the login form off the front page
I realize this might be a hard decision. I cant offer an alternative. If you have an integrated login form on the front page of your site, and you wish to cache that page with varnish, you will likely have to chose one or the other. It would probably be ideal to replace that login form with a button to bring the user to a secondary page off the main page.
For the sake of argument, lets call our site “example.com” and the login page url within Joomla should look like the following :
http://www.example.com/index.php?option=com_user&view=login
Take note of login URI in this string.
The reason we need the login form on a secondary page is because we need an almost “sandboxed” section of the site where the anonymous session cookie can be established, and passed through the authentication process to a logged in session. We will tell varnish to essentially ignore this page.
Step 2 : Modify Joomla to send HTTP headers for user/guest sessions
This isn’t that hard. In the Joomla code, there is a section where it defines the HTTP headers it sends to the browser for cache variables such as expire times and whatnot. I’m going to assume you have turned off the built-in Joomla caching system.
What you need to do is tell Joomla to send a special HTTP header that will give either a True or False value if the user is logged in or not. This is useful information. It will allow varnish to not cache any logged in content such as “Welcome back, USERNAME” after the user is passed back to the front page from logging in.
In my joomla installation, I modified the following file :
The parent folder being the public_html / root folder for your Joomla installation. In this file, please find the line that determines if the Joomla caching system is disabled :
After this line, you will see about 5 HTTP header declarations (expires, last-modified, cache-control, cache-control again and pragma). Above those declarations , add the following 6 lines of code :
if (!$user->guest) {
JResponse::setHeader( 'X-Logged-In', 'True', true);
} else {
JResponse::setHeader( 'X-Logged-In', 'False', true );
}
If you read the above code, its fairly straight forward. I do a check to see if the user is a guest (aka anonymous) or not. If they are logged in I send an HTTP header called “X-Logged-In”, and assign a “True” value to it. If the user is not logged in, it sets it to “False”.
Pretty easy, right?
This will allow varnish to avoid caching a logged in user’s page.
Step 3 : Configure Varnish
This is the part that took the most time during this entire process. Mind you patching the Joomla code and whatnot took some time as well, this process took a lot of experimentation and long hours examining session cookies and host headers.
What I will do is break down the generalized configuration directives into two groups : VCL_RECV and VCL_FETCH.
VCL_RECV
In here, I set a bunch of IF statement directives to tell varnish what it should look up in the cache and what it should pipe to the backend and what it should pass. This could probably be optimized and improved upon, but it works for me :
if (req.request == "POST") {
set req.backend = iamloggedin;
return(pipe);
}
# http authenticated sessions are piped
if (req.http.Authenticate || req.http.Authorization) {
set req.backend = iamloggedin;
return(pipe);
}
# if the user is coming FROM the login page, pipe to backend
if (req.http.referer ~ "(?i)(com_user|login)") {
set req.backend = iamloggedin;
return(pipe);
}
VCL_FETCH
The fetch section is a little bit easier. I only have about 5 directives. The first one is the most important one you want to look at. It “unsets” the cookie from any page on the site, EXCEPT the login page. This allows varnish to properly establish the logged in session. The subsequent rules determine what to deliver and what to pass based on URI or HTTP header checks :
if (!req.url ~ "(?i)(login|com_user|user|logout)") {
unset beresp.http.Set-Cookie;
}
if (req.http.referer ~ "(?i)(com_user|login|logout)") {
set req.backend = iamloggedin;
return(pass);
}
if (beresp.http.x-logged-in ~ "False"){
set req.backend = webfarm;
return(deliver);
}
if (beresp.http.x-logged-in ~ "True"){
set req.backend = iamloggedin;
return(pass);
}
if (req.http.Authenticate || req.http.Authorization) {
set req.backend = iamloggedin;
return(pass);
}
Thats it! I just saved you many sleepless nights (I hope!). Hopefully your headers will look something like this after you implement varnish in front of Joomla :
P3P CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM"
X-Logged-In False
Expires Mon, 1 Jan 2001 00:00:00 GMT
Last-Modified Mon, 08 Aug 2011 20:49:37 GMT
Cache-Control post-check=0, pre-check=0
Pragma no-cache
Content-Type text/html; charset=utf-8
Content-Length 85898
Date Mon, 08 Aug 2011 21:01:52 GMT
X-Varnish 761778669 761751685
Age 735
Via 1.1 varnish
Connection keep-alive
X-Cache-Svr cache.example.com
X-Cache HIT
X-Cache-Hits 121
UPDATE : 12/08/2011
I realize I made a mistake and have corrected this post. In vcl_fetch, i had the following :
if (!req.url ~ "(?i)(login|com_user|user|logout)") {
unset req.http.Set-Cookie;
}
Well I realize I should be unsetting the response cookie, not the set cookie. For some reason, the above (erroneous) directive works only right after you login. If you start clicking around the site, your logged in session disappears. I suspect this is because either joomla or varnish is mistakenly unsetting a logged in session.
This is the correct entry (I have fixed it in my original post as well) :
if (!req.url ~ "(?i)(login|com_user|user|logout)") {
unset beresp.http.Set-Cookie;
}
After making the above change, I can login and browse the site and my session stays intact. Mind you, the Joomla site I am testing with is definitely not a vanilla Joomla installation.
I’d love to hear from anyone who has accomplished the above scenario either way!
How does this work? You send the x-logged-in header from joomla (the backend) but then check it against headers received from client (req.http). Did you check your hitrate?
Your absolutely correct. That was a mistake. I have taken it out of the vcl_recv code. The most important thing for vcl_recv is to pipe anyone with the login page as their referrer. During my testing, I had a lot of problems with the main page still being cached after a user returns from the login page. Either the “Welcome, USERNAME” message never appears , or the “Welcome, USERNAME” message stays on the front page even after logging out. Thanks for catching that!
Hi, I managed to configure Varnish + Joomla without many problems. I was only in doubt backend settings. The backend webfarm is the production web server. What is the configuration of the backend iamloggedin??
The backend setting for “iamloggedin” is really not complicated : director iamloggedin client { { .backend = web1_1; .weight = 5; } } Is that what you needed?
Hi, thanks for this great article, but after I implemented varnish 3 with joomla 1.5 + virtuemart 1.5, there is a problem with the logout, which just can’t function. so I just wonder if you can send me the full vcl file at your convenience! thanks in advance! I don’t know how to configure multi back-end for varnish as I have 3 sites on the same server.
hi there, thanks for this, i got a question about that backend “webfarm” on here: if (beresp.http.x-logged-in ~ “False”){ set req.backend = webfarm; return(deliver); That webfarm should be varnish? i dont get it
I do have imlogeddin as apache backend but i dont know what webfarm is for
Julián: If you don’t have a backend named ‘webfarm’, then just use the default varnish backend named: ‘default’
Hi Kevin !! Thanks for this tutorial. I just wants to know whether this would work also in Joomla! 2.5 ? Btw, thanks a-lot for all those ‘Related Links’ – was worth it
In theory it should work, although I haven’t tried it. The modifications to the Joomla code to add those logged in / not logged in headers should be applicable to that version although you’d have to test it. As long as you are able to do that for that version of joomla, I dont see why you cant. Ideally Joomla should build this functionality into its system so you dont have to edit the code like this. Its that kind of development / planning that would allow Joomla to more naturally scale for the larger sites. I think its things like this that cause people considering using Joomla for larger sites to have second thoughts
Good luck and let me know how it goes!
Good reply Kevin !! Here is something cool. I found a large site using what we are talking about. Just few moments back I was checking this joomla 1.5 site mydailymoment(dot)com which is using varnish and LiteSpeed. This site has an Alexa rank of #15,509 and Quantcast Rank #1,614 – this must be really something!!
I tried some areas on their site and found it to be loading good. I tried their voting system and found to be bugged. I also saw that they were using a CDN. I read their person say that they had to do some modifications to achieve this. As you said Kevin, me too believe is the same – it is better to expect Joomla team to bring in-built support for Varnish like they have brought in-built support for APC and also Memcached. Verdict: Varnish is of no use if the user is logged-in and Varnish will slow down as it stands between the Nginx(This is what i know.) But it really really works great for delivering cached contents. But at the present stage because of no proper syncing with Joomla – this would take away the quality of the web pages delivered and would also take away the full-functionality of a web page as not everything on a page should be cached. I have a feeling that almost the same speed and performance can be achieved without Varnish and at the same time deliver full-rich webpages. Present Server Configuration i believe which is good for a Joomla site is: Nginx – httpserver, can also be used to gzip and compress APC – PHP Opcode caching Memcached – For database-calls caching Jotcache – A free joomla component and a plugin (content caching + browser caching +more) This has the features like exclude the pages not to be cached and the modules not to be cached, a-must for delivering quality cached pages. I think its good to have this as it is a native extension which was tailor-made for Joomla; plus it has good reviews on JED. God bless this man. 
Until Varnish can’t do something like this for Joomla, i think it can be a troublesome experience for running a Joomla site with Varnish. PHP-FPM – FastCGI Process Manager. I read on howtoforge(dot)com that its good to make PHP-FPM use a Unix socket which avoids the TCP overhead. cloudflare – For CDN, Security & plenty of the other useful features. This is what is in my mind as of now. It would be great to know what you think about this Kevin? *Varnish? Yes, will be looking towards it. 