Is this scenario a valid stress test?

Question

I have a server application that handles clients requests in different manner.

I want to know how many users can be served with minimal latency, so I made a small stress test application that simulate the users requests; at the same time another application monitor the memory/CPU utilization.

The stress test tool creates thread every second where every thread represents a user. If the stress test can not create a new thread due to lack of resources it starts a new instance of the stress test tool.

The problem is, every thread writes into the file the latency for each request and the current number of threads running so this causes I/O problem as after couple of minutes you have a lot of threads that need to write into disk also this behavior will not be exist in the real scenario as the client only request the data.

How can I overcome this problem as I want to measure the maximum latency per user?

PS:

Some answers say to run on different machine to take into consideration the network latency ok, this well be my final stress test currently I am doing this test on the same server to find how many users are supported with minimal latency.

Answer 1

If you are interested in the maximum latency per user, why not just collect this in the thread and when stopping the test have all the threads write our there max latency. You could do statistics as well, calculating min/max/variance and number of threads/users running. You shouldn't update screen output either. if you fear data loss, write the data out to disk frequently.

Threads are suboptimal doing this test for a client/server app. Having only a limited number of cores, only very few of the threads really run in parallel, but get their timeslices. It is much better, and gives you some figures on network latency as well, to start your program on several clients. The the server software can - if able to do so - use it's hardware as it will in the final setting, where clients will run in a LAN or WAN.

Obviously you will have a mixed environment, as you cannot have a many client machines as users simulated, but scenarios like simultaneous calls from independent hardware will show up in such a stresstest as calls are not quasi serialized through timeslicing.

Answer 2

It's not really clear whether this is a networked application or not. If it's networked then you can simply scale the stress test by stealing everyone's desktop over the weekend to run the stress test. This may be the easiest way to scale the test if it's just a few ad-hoc tests.

However, it does sound like there could be some simple improvements. If this is meant to be a long running stress test, instead of creating a new thread for every request, you can create a pool of threads to work from (or even easier, use the thread pool, which will scale automatically). So you would define a test to be say 2000 users, and spin up 2000 threads that hammer the server. Each thread would essentially be in a loop that does the test, and repeats.

Another item that isn't clear is whether all you're threads are trying to share a single file. One way to make this less of a bottleneck would be to keep the information in memory until the program is shutting down. Or spin up a writer thread, that is responsible for the file write, and all you're other threads give it information. If IO does get backed up, you're writer thread will simply hold in memory until IO is available, and you're worker threads can continue to hammer the server in the mean-time. Just keep in mind, that due to the thread synchronization involved, this may not scale well, so you may want to buffer some entries in the worker thread and only synchronize to the file writer thread once every 100 requests. I don't think this will be much of an issue since it doesn't sound like you're tracking anything more than response times.

Edit: Based on comment I would suggest trying to use a single thread to manager you're IO operations in this case. All of you're worker threads would instead of writing to file, create an object with whatever the details are, and pass it to a queue to be written to file. To cut down on lock / unlocks, use a queue within the worker thread as well, and only sync every so often. Make sure you do lock when you're exchanging the info in the thread. Also, i'd maybe watch the memory usage since this will allow anything pending to build up in memory. If this is still causing you're io to block, i'd look at either writing less, or maybe tuning or adding a faster hard disk.