Web Service with lots of connections

Question

Hi guys,

I'm planning the development of a web service that has to be very scalable, in order to process many concurrent connections, probably thousands. The service wil act as an API. It has to be highly responsive, a delay of 3 seconds between request and reply is considered too much.

Maybe the service could be distributed among many internal servers, with a load-balancing gateway acting as flow management, so the actual processing the service does will not hog up the gateway server. I am considering using Amazon EC2, but I can also go with an array of in-house servers, as long as I can find a good use for them. The database will probably be MySQL (unless you have a better idea).

I realise that the best solution would be to develop my own web server, but I it would take too much time and I doubt I can do a better job than some other web servers that I'm not aware of.

Also, I'm currently trying to decide between persistent connections or not, but I will probably stick with the latter.

So, any recommendations for a good, scalable solution?

Answer 1

I don't agree that developing your own web server is "the best solution." Use one that exists. You shouldn't need to write a server just to deploy a web service.

Both .NET and Java EE offer ways to create and deploy web services. You don't say which language is your choice, but certainly you can write a web service in Java EE and deploy it on Tomcat without having to write a custom web server.

As far as load balancing goes, F5 is a good hardware solution if you can afford it.

Answer 2

You don't mention the protocol. If you use a RESTful approach, this is a very easy piece of code to write. If you use a SOAP approach, this is more complex.

Apache + FastCGI will do this, but your service will be a CGI program. You'll have to write in C or C++ which can be unpleasant. You can use Apache + mod_wsgi plus one of the Python frameworks; it isn't as fast, but it will be very fast and you don't have to write much code.

Glassfish will do this, and you get to write in Java.

A commercial product (like Sun's JCAPS) will do this.

There are numerous web services frameworks -- inventing your own is not a good idea.

Edit The "MaxClients" issue.

A web service request should be fast -- it's a resource -- you fetch it from cache or the database and respond. The limiting factor isn't MaxClients -- it's threads that can successfully coexist and it's socket negotiation.

If your GET requests are idempotent, they can be cached in squid (or some other reverse proxy server). You can have a large number of these. Note that using a reverse proxy server has nothing to do with your web service per se; it has everything to do with overall throughput, however.

Answer 3

Amazon EC2 works well, but don't base your pricing on the $0.10 instances -- they are very underpowered. I recommend starting with at least c1.mediums. I like to use nginx instances as load balances on m1.small's infront of c1.medium web server instances (all the apps I've built recently are CPU bound and not memory bound).

3 seconds is a long time. I typically use 200ms-400ms for a performance objective. Of course this changes depending on how time sensitive your app is and how much work has to be done.

If there's really 3 seconds of work to be done, you can probably have a backend grid pre-processing data. You then probably need fewer web servers since they are mostly returning premade data.

If I were in your shoes, I'd build a prototype in whatever language/platform I'm most comfortable in. Then you can get an idea of where you'll need to take it.

Don't forget that it takes time to build a high performance web app. You need to code, benchmark, repeat to scale and get your response time down.

The number of instances in which you need to write your own webserver are very slim. This is not one of them. There are many to choose from. Assuming a nix based platform, you can use Apache, nginx or lighttpd. There's a bunch of other ones, but they are typically used as app servers (tomcat, zope, mongrel, etc) and have apache/nginx/lighttpd/squid proxy in front of them.

I haven't really used an off the shelf platform (solution) for building web apps (I'm assuming you mean some sort of java stack or .Net stack). I can't really help you there. Most of the tools I work with (LAMP type stuff) are componentized and allow drop in replacements at every piece of the stack. It's not uncommon to outgrow one component and have to replace it with another.

For example, no load balancer -> software load balancer -> software load balancer capable of buffering -> hardware load balancer. All of these are cost effective for different levels of scale.

Answer 4

Commercial load balancing solutions can be out of many's budget. You can look at Ultra Monkey to create a load-balanced pool for $0. Of course what you give up is time. And you will spend a fair amount of it researching and troubleshooting, but it's totally doable. At least for an early proof of concept.

Think about this - even if you spend a solid month setting up load-balancing, you can save tens of thousands of dollars. For most people, that's a good deal.