Scalable web application with lot of image servings

Question

I started working on a web application. This application needs lot of image handling. I started off with PHP as it was the easiest and cheapest to host. I have used the .NET framework for some of my previous applications and I'm very comfortable with Python.

But I'm not at all comfortable using PHP now, so I have decided to use something easier for me.

Can anyone help me understand if the .NET framework or Python (currently web.py looks good to me) has some edge over others, considering a lot of image manipulation and let's say about 200 requests per second?

Also I would appreciate if someone can suggest a proper host for either of them.

EDIT:

Sorry for the confusion. By image handling I mean the users of the application are allowed to upload pictures that would be stored in the flat file system while their entries are in the database.

By image manipulation, I mean I would need to create thumbnails for these images too which would be used in the application.

Answer 1

If you want performance when serving images, you have to take the FaceBook approach of 'never go to disk unless absolutely necessary' - meaning use as much caching as possible between your image servers and the end user. There are many products that can help you out both commercial and free, including just configuring your webservers correctly - google and see what works for your cost and platform.

Answer 2

From what You have written, either .NET or Python would be a good choice for You. Personally, I would go for Python. Why?

• It is free.
• It is scalable.
• With Python Imaging Library You can do almost anything with images.
• Hey, it is python - less code, same result.

To be honest Your choice is not important - just choose the one You feel comfortable with and stick with it.You mentioned web.py - this site is made with web.py: colr.org - and it is made with 1304 lines of code, not counting external libraries.

Answer 3

Please buy Schlossnagle's book, Scalable Internet Architectures.

You should not be serving the images from Python (or PHP or .Net) but from Apache and Squid. Same is true for Javascript and CSS files -- they're static media, and Python should never touch them.

You should only be processing the HTML portion of the transaction.

This, BTW, is the architecture you get with things like Django. Static media is handled outside Python. Python handles validation and the HTML part of the processing.

Turns out that you'll spend much of your time fussing around with Squid and Apache trying to get things to go quickly. Python (and the Django framework) are fast enough if you limit their responsibilities.

Answer 4

I do not think switching languages will help much with your problem, It's just the architecture you chose initially only works for small amounts of data. I would recommend you to visit http://highscalability.com , It's time you started looking how the big guys scale their applications.

Answer 5

As mentioned previously, any number of development platforms will work, it really depends on your approach to caching the content.

If you are comfortable with Python I would recommend Django. There is a large development community and a number of large applications and sites running on the framework.

Django internally supports caching through use of memcached. You are able to customize quite greatly how and what you want to cache, while being able to keep many of the settings for the caching in your actual Django application (I find this nice when using third party hosting services where I do not have complete control of the system).

Here are a few links that may help:

• Django framework - General information on the Django framework.
• Memcached with Django - Covers how to configure caching specifically for a Django project.
• Memcached website
• The Django Book - A free online book to learn Django (it also covers caching and scaling quetsions).
  • Scaling Chapter
  • Caching Chapter

There are a number of hosting companies that offer both shared and dedicated hosting plans. I would visit http://djangohosting.org/ to determine which host may work best for your need. I have used WebFaction quite a bit and have been extremely pleased with their service.

Answer 6

I would also recommend looking at MogileFS. It is a distributed file system which runs on Unix based OS's. I know that digg use it to store their avatar images.

It's from the same guys who created memcached and live journal.