Lightweight HTTP application/server for static content

Question

Hi,

I am in need of a scalable and performant HTTP application/server that will be used for static file serving/uploading. So I only need support for GET and PUT operations.

However, there are a few extra features that I need:

• Custom authentication: I need to check credentials against a database for each request. Thus I must be able to integrate propietary database interaction.
• Support for signed access keys: The access to resources via PUT should be signed using a key like http://uri/?key=foo The key then contains information about the request like md5(user + path + secret) which allows me to block unwanted requests. The application/server should allow me to check for this.
• Performance: I'd like to avoid piping content as much as possible. Otherwise the whole application could be implemented in Perl/etc. in a few lines as CGI.

Perlbal (in webserver mode) looks nice, however the single-threaded model does not fit with my database lookup and it does also not support query strings.

Lighttp/Nginx/… have some modules for these tasks, however it is not feasible putting everything together without ending up writing own extensions/modules.

So how would you solve this? Are there other leightweight webservers available for this? Should I implement an application inside of a webserver (i.e. CGI). How can I avoid/speed up piping content between the webserver and my application.

Thanks in advance!

Answer 1

Have a look at nodejs http://nodejs.org/

There are a few modules for static web servers and database interfaces: http://wiki.github.com/ry/node/modules

You might have to write your own file upload handler, or use one from this example http://www.componentix.com/blog/13

Answer 2

nginx + spawn-fcgi + fcgi application written in C + memcached + sqlite serves for similar task well, latency is about 20-30 ms for small data and fast connections from the same local network. As far as I know production server handles about 100-150 requests per second with no problem. On test server I peaked up to 20k requests per second, again with no problem, average latency were about 60 ms. Aggressive caching and UNIX domain sockets is the key.

Do not know how that configuration will behave on frequent PUT requests, in our task they are very rare and typically batched.