How to do performance and scalability testing without clear requirements?

Question

Any idea how to do performance and scalability testing if no clear performance requirements have been defined?

More information about my application.

The application has 3 components. One component can only run on Linux, the other two components are Java programs so they can run on Linux/Windows/Mac... The 3 components can be deployed to one box or each component can be deployed to one box. Deployment is very flexible. The Linux-only component will capture raw TCP/IP packages over the network, then one Java component will get those raw data from it and assemble them into the data end users will need and output them to hard disk as data files. The last Java component will upload data from data files to my database in batch.

Answer 1

Define your own. Take the initiative and describe the performance goals yourself.

To answer any better, we'd have to know more about your project.

Answer 2

In the absence of 'must be able to perform X iterations within Y seconds...' type requirements, how about these kinds of things:

• Does it take twice as long for twice the size of dataset? (yes = good)
• Does it take 10x as long for twice the size of dataset? (yes = bad)
• Is it CPU bound?
• Is it RAM bound (eg lots of swapping to virtual memory)?
• Is it IO / Disk bound?
• Is there a certain data-set size at which performance suddenly falls off a cliff?

Answer 3

rely on tools (fxcop comes to mind) rely on common sense

Answer 4

If you want to test performance and scalability with no requirements then you should create your own requirements / specs that can be done in the timeline / deadline given to you. After defining the said requirements, you should then tell your supervisor about it if he/she agrees.

To test scalability (assuming you're testing a program/website): Create lots of users and data and check if your system and database can handle it. MyISAM table type in MySQL can get the job done.

To test performance: Optimize codes, check it in a slow internet connection, etc.

Answer 5

Surprisingly this is how most perf and scalability tests start.

You can clearly do the testing without criteria, you just define the tests and measure the results. I think your question is more in the lines 'how can I establish test passing criteria without performance requirements'. Actually this is not at all uncommon. Many new projects have no clear criteria established. Informally it would be something like 'if it cannot do X per second we failed'. But once you passed X per second (and you better do!) is X the 'pass' criteria? Usually not, what happens is that you establish a new baseline and your performance tests guard against regression: you compare your current numbers with the best you got, and decide if the new build is 'acceptable' as build validation pass (usually orgs will settle here at something like 70-80% as acceptable, open perf bugs, and make sure that by ship time you get back to 90-95% or 100%+. So basically the performance test themselves become their own requirement.

Scalability is a bit more complicated, because there there is no limit. The scope of your test should be to find out where does the product break. Throw enough load at anything and eventually it will break. You need to know where that limit is and, very importantly, find out how does your product break. Does it give a nice error message and revert or does it spills its guts on the floor?

Answer 6

If there has been 'no performance requirement defined', then why are you even testing this?

If there is a performance requirement defined, but it is 'vague', can you indicate in what way it is vague, so that we can better help you?

Short of that, start from the 'vague' requirement, and pick a reasonable target that at least in your opinion meets or exceeds the vague requirement, then go back to the customer and get them to confirm that your clarification meets their requirements and ideally get formal sign-off on that.

Answer 7

Some definitions / assumptions:

Performance = how quickly the application responds to user input, e.g. web page load times

Scalability = how many peak concurrent users the applicaiton can handle.

Firstly perfomance. Performance testing can be quite simple, such as measuring and recording page load times in a development environment and using techniques like applicaiton profiling to identify and fix bottlenecks.

Load. To execute a load test there are four key factors, you will need to get all of these in place to be successfull.

1. Good usage models of how users will use your site and/or application. This can be easy of the application is already in use, but it can be extermely difficult if you are launching a something new, e.g. a Facebook application.

If you can't get targets as requirements, do some research and make some educated assumptions, document and circulate them for feedback.

2. Tools. You need to have performance testing scripts and tools that can excute the scenarios defined in step 1, with the number of expected users in step 1. (This can be quite expensive)

3. Environment. You will need a production like environment that is isolated so your tests can produce repoducible results. (This can also be very expensive.)

4. Technical experts. Once the applicaiton and environment starts breaking you will need to be able to identify the faults and re-configure the environment and or re-code the application once faults are found.

Generally most projects have a "performance testing" box that they need to tick because of some past failure, however they never plan or budget to do it properley. I normally recommend to do budget for and do scalability testing properley or save your money and don't do it at all. Trying to half do it on the cheap is a waste of time.

However any good developer should be able to do performance testing on their local machine and get some good benefits.

Answer 8

Short answer: Don't do it!

In order to get a (better) definition write a performance test concept you can discuss with the experts that should define the requirements.

Make assumptions for everything you don't know and document these assumptions explicitly. Assumptions comprise everything that may be relevant to your system's behaviour under load. Correct assumptions will be approved by the experts, incorrect ones will provoke reactions.

For all of those who have read Tom DeMarcos latest book (Adrenaline Junkies ...): This is the strawman pattern. Most people who are not willing to write some specification from scratch will not hesitate to give feedback to your document. Because you need to guess several times when writing your version you need to prepare for being laughed at when being reviewed. But at least you will have better information.

Answer 9

The way I usually approach problems like this is just to get a real or simulated realistic workload and make the program go as fast as possible, within reason. Then if it can't handle the load I need to think about faster hardware, doing parts of the job in parallel, etc.

The performance tuning is in two parts.

Part 1 is the synchronous part, where I tune each "thread", under realistic workload, until it really has little room for improvement.

Part 2 is the asynchronous part, and it is hard work, but needs to be done. For each "thread" I extract a time-stamped log file of when each message sent, each message received, and when each received message is acted upon. I merge these logs into a common timeline of events. Then I go through all of it, or randomly selected parts, and trace the flow of messages between processes. I want to identify, for each message-sequence, what its purpose is (i.e. is it truly necessary), and are there delays between the time of receipt and time of processing, and if so, why.

I've found in this way I can "cut out the fat", and asynchronous processes can run very quickly.

Then if they don't meet requirements, whatever they are, it's not like the software can do any better. It will either take hardware or a fundamental redesign.

Answer 10

Although no clear performance and scalability goals are defined, we can use the high level description of the three components you mention to drive general performance/scalability goals.

1. Component 1: It seems like a network I/O bound component, so you can use any available network load simulators to generate various work load to saturate the link. Scalability can be measure by varying the workload (10MB, 100MB, 1000MB link ), and measuring the response time , or in a more precise way, the delay associated with receiving the raw data. You can also measure the working set of the links box to drive a realistic idea about your sever requirement ( how much extra memory needed to receive X more workload of packets, ..etc )

2. Component 2: This component has 2 parts, an I/O bound part ( receiving data from Component 1 ), and a CPU bound part ( assembling the packets ), you can look at the problem as a whole, make sure to saturate your link when you want to measure the CPU bound part, if is is a multi threaded component, you can look for ways to improve look if you don't get 100% CPU utilization, and you can measure time required to assembly X messages, from this you can calculate average wait time to process a message, this can be used later to drive the general performance characteristic of your system and provide and SLA for your users ( you are going to guarantee a response time within X millisecond for example ).

3. Component 3: Completely I/O bound, and depends on both your hard disk bandwidth, and the back-end database server you use, however you can measure how much do you saturate disk I/O to optimize throughput, how much I/O counts do you require to read X MB of data, and improve around these parameters.

Hope that helps. Thanks