How do you stress test a web application?

Question

Stress testing is something that gets very little love in most death mar.., er I mean web projects. It's usually done at the last minute (or not at all), next to no time gets allocated to it, etc.

In the past I've picked a tool, installed it on my machine first, hit the home page, upping the concurrency settings. Then I'd write a simple login script, a simple site walkthrough (in an ecommerce site adding a few items to a cart and checkout). Then I'd rope in as many developers as I could, install the stress test tool on their machines and do a massive attack. If nothing broke I'd give up and wait for the actual traffic to kill the site for real.

Just hitting the homepage hard almost always would locate a major problem. More scalability problems would surface at the second stage, and even more - after the launch.

The tools I used were Microsoft Homer (aka Microsoft Web Application Stress Tool) and Pylot.

The reports that these tools generated never made much sense to me, and I spent many hours trying to figure out what kind of concurrent load the site would be able to support. It was always worth it because the stupidest bugs and bottlenecks would always come up (like web server misconfigurations, for instance).

What have you done, what tools have you used, and what success have you had with your approach? The part that is most interesting to me is coming up with some kind of a meaningful formula for calculating the number of concurrent users an app can support from the numbers reported by the stress test application.

Answer 1

I've used The Grinder. It's open source, pretty easy to use, and very configurable. It is Java based and uses Jython for the scripts. We ran it against a .NET web application, so don't think it's a Java only tool (by their nature, any web stress tool should not be tied to the platform it uses).

We did some neat stuff with it... we were a web based telecom application, so one cool use I set up was to mimick dialing a number through our web application, then used an auto answer tool we had (which was basically a tutorial app from Microsoft to connect to their RTC LCS server... which is what Microsoft Office Communicator connects to on a local network... then modified to just pick up calls automatically). This then allowed us to use this instead of an expensive telephony tool called The Hammer (or something like that).

Anyways, we also used the tool to see how our application held up under high load, and it was very effective in finding bottlenecks. The tool has built in reporting to show how long requests are taking, but we never used it. The logs can also store all the responses and whatnot, or custom logging.

I highly recommend this tool, very useful for the price... but expect to do some custom setup with it (it has a built in proxy to record a script, but it may need customization for capturing something like sessions... I know I had to customize it to utilize a unique session per thread).

Answer 2

One more note, for our web application, I found that we had huge performance issues due to contention between threads over locks... so the moral was to think over the locking scheme very carefully. We ended up having worker threads to throttle too many requests using an asynchronous http handler, otherwise the application would just get overwhelmed and crash and burn. It meant a huge backlog could pile up, but at least the site would stay up.

Answer 3

I've used JMeter. Besides testing the web server you can also test your database backend, messaging services and email servers.

Answer 4

The policy at my current place of work...

Make it live and hope for the best.

Answer 5

Take a look at TestComplete.

Answer 6

I found IBM Page Detailer also an interesting tool to work with.

Answer 7

I've used openSTA.

This allows a session with a web site to be recorded and then played back via a relatively simple script language.

You can easily test web services and write your own scripts.

It allows you to put scripts together in a test in any way you want and configure the number of iterations, the number of users in each iteration, the ramp up time to introduce each new user and the delay between each iteration. Tests can also be scheduled in the future.

It's open source and free.

It produces a number of reports which can be saved to a spreadsheet. We then use a pivot table to easily analyse and graph the results.

Answer 8

Here's another vote for JMeter.

JMeter is an open-source load testing tool, written in Java. It's capable of testing a number of different server types (for example, web, web services, database, just about anything that uses requests basically).

It does however have a steep learning curve once you start getting to complicated tests, but it's well worth it. You can get up and running very quickly, and depending on what sort of stress-testing you want to do, that might be fine.

Pros:

• Open-Source/Free tool from the Apache project (helps with buy-in)
• Easy to get started with, and easy to use once you grasp the core concepts. (Ie, how to create a request, how to create an assertion, how to work with variables etc).
• Very scalable. I've run tests with 11 machines generating load on the server to the tune of almost a million hits/hour. It was much easier to setup than I was expecting.
• Has an active community and good resources to help you get up and running. Read the tutorials first and play with it for a while.

Cons:

• The UI is written in Swing. (ugh!)
• JMeter works by parsing the response text returned by the server. So if you're looking to validate any sort of javascript behaviours, you're out of luck.
• Learning curve is steep for non-programmers. If you're familiar with regular expressions, you're already ahead of the game.
• There are large numbers of (insert expletive) idiots in the support forum asking stupid questions that could be easily solve if they'd give the documentation even a cursory glance. ('How do I use JMeter to stress-test my Windows GUI' shows up quite frequently).
• Reporting 'out of the box' leaves much to be desired, particularly for larger tests. In the test I mentioned above, I ended up having to write a quick console app to do some of the 'xml-logfile' to 'html' conversions. That was a few years ago though, so it's probable that this would no longer be required.

Answer 9

I second the opensta suggestion. I would just add that it allows you to do things to monitor the server you're testing using SMTP. We keep track of processor load, memory used, byes sent, etc. The only downside is that if you find something boken and want to do a fix it relies on several open-source libraries that are no longer kept up, so getting a compiling version of the source is more tricky than with most OSS.

Answer 10

We use the Microsoft tool mentioned - Microsoft Web Application Stress Tool. It is the easiest tool I have used. It is limited in many ways, including only being able to hit port 80 on manually created tests. But, its ease of use means it actually gets used.

We supplement the load from this tool with other tools including OpenSTA and link check spiders.

JMeter looks good from my initial evaluation, I hope to include it in our continuous integration going forward. But, JMeter is complex and non trivial to roll out.

I'd suggest opening another question regarding interpreting the MS stress tool results.

Answer 11

I tried WebLoad it's a pretty neat tool. It comes with and test script IDE which allows you to record user action on a website. It also draws a graph as it perform stress test on your web server. Try it out, I highly recommend it.

Answer 12

have your mom play around with it for a bit; It will help you find things you never thought to look for. and as far as stress.... well it fits some definitions :)

Answer 13

I played with JMeter. One think it could not not test was ASP.NET Webforms. The viewstate broke my tests. I am not shure why, but there are a couple of tools out there that dont handle viewstate right. My current project is ASP.NET MVC and JMeter works well with it.

Answer 14

A little late to this party. I agree that Pylot is the best up-and-coming open source tool out there. It's simple to use and is actively worked on by a great guy (Corey Goldberg). As the founder of OpenQA, I'm also happy that Pylot now is listed on our home page and uses some of our infrastructure (namely the forums).

However, I also recently decided that the entire concept of load testing was flawed: emulating HTTP traffic, with applications as complex as they have become, is a pain in the butt. That's why I created BrowserMob. It's an external load testing service that uses Selenium to control real web browsers when playing back load.

The approach obviously requires a ton more hardware than normal load testing techniques, but hardware is actually pretty cheap when you are using cloud computing. And a nice side effect of this is that the scripting is much easier than normal load testing. You don't have to do any advanced regex matching (like JMeter requires) to extract out cookies, .NET session state, Ajax request parameters, etc. Since you're using real browsers, they just do what they are supposed to do.

Sorry to blatantly pitch a commercial product, but hopefully the concept is interesting to some folks and at least gets them thinking about some new ways to deal with load testing when you have access to a bunch of extra hardware!

Answer 15

You asked this question almost a year ago and I don't know if you still are looking for another way of benchmarking your website. However since this question is still not marked as solved I would like to suggest the free webservice LoadImpact (btw. not affiliated). Just got this link via twitter and would like to share this find. They create a reasonable good overview and for a few bucks more you get the "full impact mode". This probably sounds strange, but good luck pushing and braking your service :)

Answer 16

Visual Studio Test Edition 2010 (2008 good too). This is a really easy and powerful tool to create web/load tests with.

There is a separate and expensive licence required to use several machines to load test the application however.

Answer 17

For simple usage, I perfer ab(apache benchmark) and siege, later one is needed as ab don't support cookie and would create endless sessions from dynamic site.

both are simple to start:

ab -c n -t 30 url

siege -b -c n -t 30s url

siege can run with more urls.

last siege version turn verbose on in siegerc, which is annoy. you can only disable it by edit that file(/usr/local/etc/siegerc).

Answer 18

Trying all mentioned here, I found curl-loader as best for my purposes. very easy interface, real-time monitoring, useful statistics, from which I build graphs of performance. All features of libcurl are included.