How to efficiently store and serve lots of small images on the web site?

Question

On my web site I want to serve user a page with a lot of small pictures on it, kind of like thumbnails in Windows.
All pictures are files ranging in size from 1KB to over 300KB, but usually no bigger than 5KB.

Currently all those images are located in a folder on the web server (linux+apache), but I don't think this is the best solution, especially if I'll have to expand to more than one web server and when I'll have thousands of them.

So, should I maybe place these files in the database (can I)? Or have a separate file server? Other suggestions?

Answer 1

You want to put as little CPU time / picture as possible. I would suggest to use a proxy to serve those pictures (as static content). I have good experience with nginx, it's fast an reliable.

Your config could look something like this

worker_processes  1;
events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;

    keepalive_timeout  300;

    server {
        listen       8080;
        server_name  localhost;

        location /static/ {root static/pictures/;}

        location / { proxy_pass http://127.0.0.1:8081; }
    }
}

Where you have your server on port 8081 and you directly serve your pictures from the folder /static/pictures. (Note that all URLs starting with '/static' will be served from this folder.)

Answer 2

A few things to consider:

• Don't put the images in a database unless you can keep it in memory. Ordinary files are often the perfect solution for binary data, and you'll still be issuing separate HTTP requests for each of them.
• If you can mirror the images on several servers and balance between them, great.
• If you don't have extra servers, try faking it with virtual subdomains to circumvent concurrent connection limits. See this article. Also relevant to this: spread the images on each page across your subdomains, 2 or more on each.
• Don't try to compress the traffic. You'll only waste CPU time.
• Make sure you're using proper caching (e.g. far future Expires) for static images. You can always change the address if an image is updated.
• Make sure you serve all static content from cookie-less (sub)domains.