Hello,
I recently created a big portal site. It's time for putting it to test.
How do you guys test a site rigorously?
What are the ways and tools for that?
Can we sort of mimic hundreds of virtual users visiting the site to see its load handling?
The test should be for both security and speed
Thanks in advance.
If this is just supposed to be a Stress Test, try http://freshmeat.net/projects/siege/
Siege is a regression test and benchmark utility. It can stress test a single URL with a user defined number of simulated users, or it can read many URLs into memory and stress them simultaneously. The program reports the total number of hits recorded, bytes transferred, response time, concurrency, and return status. Siege supports HTTP/1.0 and 1.1 protocols, GET and POST directives, cookies, transaction logging, and basic authentication. Its features are configurable on a per user basis.
or http://httpd.apache.org/docs/1.3/programs/ab.html
ab is a tool for benchmarking the performance of your Apache HyperText Transfer Protocol (HTTP) server. It does this by giving you an indication of how many requests per second your Apache installation can serve.
To test your PHP code, use PHPUnit. To test your frontend, use Selenium.
For security tests, I strongly suggest to buy an audit and leave that to dedicated experts. There is just too many possible attack vectors that go beyond your app code.
You could try Google's Skipfish. It's a security scanner but it will incidentally generate quite a lot of requests to your site.
I suggest you read this post, its very useful I check my dynamic web sites each time I build one.
Edit :
http://jakarta.apache.org/jmeter/
Apache JMeter is open source software, a 100% pure Java desktop application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.
http://seleniumhq.org/ is the technology to look for.
Quotes from their site:
Selenium is a robust set of tools that supports rapid development of test automation for web-based applications. Selenium provides a rich set of testing functions specifically geared to the needs of testing of a web application. These operations are highly flexible, allowing many options for locating UI elements and comparing expected test results against actual application behavior.
One of Selenium’s key features is the support for executing one’s tests on multiple browser platforms.
The most important part is probably: Selenium-RC (Remote Control)
Selenium-RC allows the test automation developer to use a programming language for maximum flexibility and extensibility in developing test logic. For instance, if the application under test returns a result set, and if the automated test program needs to run tests on each element in the result set, the programming language’s iteration support can be used to iterate through the result set, calling Selenium commands to run tests on each item.
Selenium-RC provides an API (Application Programming Interface) and library for each of its supported languages: HTML, Java, C#, Perl, PHP, Python, and Ruby. This ability to use Selenium-RC with a high-level programming language to develop test cases also allows the automated testing to be integrated with a project’s automated build environment.
you can use WEBLOAD . Its a tool for load testing .. goto www.webload.com
Also, I would mention Tsung.
Reding from their introduction:
Tsung is an open-source multi-protocol distributed load testing tool
It can be used to stress HTTP, WebDAV, SOAP, PostgreSQL, MySQL, LDAP and Jabber/XMPP servers. Tsung (formerly known as IDX-Tsunami) is a free software released under the GPLv2 license.
The purpose of Tsung is to simulate users in order to test the scalability and performance of IP based client/server applications. You can use it to do load and stress testing of your servers. Many protocols have been implemented and tested, and it can be easily extended. WebDAV, LDAP and MySQL support have been added recently (experimental).
It can be distributed on several client machines and is able to simulate hundreds of thousands of virtual users concurrently (or even millions if you have enough hardware ...).
Tsung is developed in Erlang, an open-source language made by Ericsson for building robust fault-tolerant distributed applications.
Tsung is supported by Process-One
have you checked out apachebenchmark?
http://httpd.apache.org/docs/1.3/programs/ab.html
it is useful for test how you can handle load and concurrent connections
I'd like to just expand on a couple of suggestions since I do this a lot.
Siege is a great sanity checker and it has a real low barrier to entry. It's a standard package in Ubuntu and others and only requires command line parameters to run. So you can slam your website as hard as you want (or dare) to see where the breaking point. My experience is that few web servers degrade in a linear fashion. As load increases they get a little bit slower, then a little bit slower, then BAM your load time jumps to 20 seconds. Knowing when that happens gives you an idea of how much to worry and if things need to be done. If you don't hit the wall until 500+ concurrent users, then you have some leeway.
But note that siege is really more about testing your web server than your web SITE.
As the same poster previously mentioned Selenium is a great tool for that, however, expect to invest some time. You can actually pop off a few tests just using the Selenium IDE and running them either in your browser under Selenium RC as an HTTP suite without getting into scripting (although you will want to do that). Both the up and downside of this approach is that this is powering a real browser, so running the test on just one machine will not get you real-world world results because your machine can just not keep up. You need to be running and setting up SRC and Selenium Grid can be a daunting task, but totally worth it if this is something you will be doing again and again, especially if you have a little VM knowledge.
The guys who created Selenium now offer a commercial cloud-based RC that runs all the standard browsers and captures video of the session (which you can turn off for a performance test). It's at http://saucelabs.com and for either load testing, or care free quick tests on all browsers it's great. (I still run functional tests locally). Again, it costs, but it's reasonable.
I spell this out more because your first point is testing your site "vigorously", and I am assuming that means testing all functionality, not just testing it with a lot of energy.
I don't know any magic bullet for this except for writing thorough tests (which is where you usually get into scripting since a lot of things can re-used, e.g. filling out a form).
Last thing is, before you launch it, if you don't have a usability program in place, let some non-programmers/web-people use it and watch them and ask them to do various "tasks" (sign up, find something, no "click on"'s) This sounds like usability testing but it's too late for that, but you will be amazed how quickly people who don't think like you find bugs in your code. This should hopefully open up new horizons of areas you haven't tested.
I haven't found a "coverage" tool out there for web apps. If there is one, but I would love to hear about it.