Running Drupal application on a webfarm (scalability ) , HOW?!!!

Question

Hey everyone,

We have created a website using Drupal , but the problem ( good problem ) is that we are getting WAAAAYY too many hits on the server to the point where the traffic is brining the server down to its knees.

now we want to run the application off of 3 servers behind a load balancer, one to just serve mysql and the other 2 to serve the web application , i have accomplished this using Symofony before for a different project and it was relatively easy.

However i don't seem to be able to get far with Drupal, the major problem at this point is to be able to save all session variables into mysql, so indepdent of which server the load balancer sends the request to , the application has a single point of reference in regards to sessions.

Any help would be greatly appreciated, i am looking for a good tutorial or documentation since i have inherited this project off of a guy who no longer works here :/

Thanks

Answer 1

Nice problem to have. Not one easy answer.

Have you looked at memecached for the caching that can help.

As can using varnish in front of Drupal for certain caching.

Of course I assume that you already have tweaked your performance settings. For normal or even aggressive caching if you can.

For sessions, you can get your load balancer set to sticky sessions so that users will keep gonig to the same server. You can even use varnish as your load balancer for this.

Answer 2

You may get some ideas from Drupal.org's Server tuning considerations and Simple Decision Tree for Drupal Enterprise Scalability.

Answer 3

Note: in my answer answer I'm assuming you are using Apache. My answer will be correct even if you are using some other server like nginx etc.

You write:

However i don't seem to be able to get far with Drupal, the major problem at this point is to be able to save all session variables into mysql, so indepdent of which server the load balancer sends the request to , the application has a single point of reference in regards to sessions.reference in regards to sessions.

Sessions are something that are handled at the core PHP level and at the Drupal+MySQL level. Essentially when a browser accesses your server, PHP core session handling logic assigns a unique PHPSESSID cookie. This cookie is sent back by that browser on every subsequent request.

[On a side note, using PHPSESSID, PHP core session logic might associate some other data like comment preferences, drupal messages that need to be shown on next page view etc. All this is done using the $_SESSION PHP variable. PHP does this quite seamlessly. Note that MySQL still does not enter in the picture until this point. MySQL only enters the picture when additional data needs to be associated with a PHPSESSID such as User ID etc. by Drupal]

Long story short, PHP does some session handling by assigning the PHPSESSID cookie. Now lets say the load balancer sends the request to Apache Webserver 1 and mod_php (PHP apache module) assigns a unique PHPSESSID, say, "563" (its a longer string in real life). Now next time this client accesses your website, the cookie PHPSESSID is sent across with value 563. Now two things different cases might occur:

1. The load balancer (coincidently) sends the request to Apache 1 which originally assigned the PHPSESSID cookie. It recognizes 567 and things work fine
2. The load balancer sends the request to Apache 2. PHPSESSID is 567 which Apache 2 mod_php never assigned. So it gets confused and assigns a fresh PHPSESSID. This is where your problems occur.

How to solve your problem: The problem you're facing is a common issue. You simply need to tell the load balancer that once a client is sent to a particular web server, the same webs server should continue dealing with that request. This is usually done by telling the load balancer itself to send a cookie saying which server deal with the initial request. In the future, the client presents this cookie to the load balancer and the load balancer directs the request to the original server that deal with the request. This as I explained above is important because only that server knows about the PHPSESSID it assigned.

All decent load balancers have the ability to assign cookies. Look up the configuration details for your load balancer.reference in regards to sessions.

More mind bending stuff After you've solved your sessions issue by configuring your load balancer to assign cookies you will need to consider one more important issue. The files folder of both your server MUST be shared some how. This makes sense. If an image is uploaded by a user on one server, other people accessing the site through the other server should have access to the same image. This is accomplished by an NFS (Networked File System) mount or a SAN.

Only then you will have a fully functioning multi server installation of Drupal. As other people have pointed out, you might want to refer to some reference articles on the net. Further optimizations are recommended such as storing the sessions table in memcache and not in MySQL but again this has nothing to do with what I wrote above. Load balancer issuing cookies is really required.

Why go through so much grief I ask? I've done multi server stuff in the past and its not really worth it unless your site is getting some serious traffic. Is your traffic huge enough? Putting up a caching layer like Varnish in front of Drupal or even better, using the boost module should solve your problems if most of your users are anonymous.

Check out this video http://sf2010.drupal.org/conference/sessions/24-million-page-views-day-60-m-month-one-server . The guy is serving out a crazy number of page views using only 1 server. Everything is so much simpler with one server. Try it out! Only the hugest of websites may require multiple servers.