Can unit tests have a execution time criteria?

Question

Is it good practice to have a unit test that specifies how long a certain function takes to return a value. I'm unit testing a curl function and want to specify that it needs to time out after 5 seconds. Is this in line with the way unit tests work?

Answer 1

I don't think, that this is a test of the contract of the method. That's why I would say no, it's not an unit-test. But it sound like some sort of system-test.

Another point that disqualifies your example as unit-test is the fact, that it needs long time (5 seconds) to execute. Typically you should be able to execute the complete suite of unit-tests every time you do a change. You cannot wait long time.

That's why I think your test should be in another set of tests, than the unit-tests. It should be executed in your daily builds/continuous integration but not before every checkin from every programmer.

Answer 2

I think this is probably deserving of a subjective tag, as everyone you ask will have their own One True Definition of unit testing.

In my opinion timing falls outside the domain of unit testing in most cases, but for the specific case of verifying correct behaviour of a timeout, that it is appropriate so long as the execution time is reasonable.

Answer 3

It depends. If the function is specified to time out after 5 sec then this is something to unit test. I don't think it's a good idea to measure performance in unit tests like this because it adds too many other reasons for unit tests to fail. If you start running unit tests in paralel on your build server you dont want the extra load to fail your tests for example.

Answer 4

Purist answer: Yes a long running unit test is a code smell and should be changed

Pragmatic answer: If you have a problem with a long running test and you want to ensure that your test suite fails quickly then add a timeout.

Answer 5

I think you should be mocking the method the function uses to detect that it has spent long enough waiting, then you can test the timeout instantly and not wait in real time.

Answer 6

maybe more of a performance test, but yes. nunit has them: http://weblogs.asp.net/egarmon/archive/2005/02/14/372114.aspx. junitperf has them: http://clarkware.com/software/JUnitPerf.html

Answer 7

What if the answer is no? Does that mean you wouldn't write the test?

If it is special case I would write the test and now worry about it.

If I stared having a lot of tests involving timing I would probably separate them out from the normal unit tests such that I could run the sets independently or together.

Answer 8

Testing your function plus curl plus the server is more integration test than unit test. That being said, you don't have to be that dogmatic, if you only have one test like that this certainly is viable. I've written, and still have tests that are not pure unit tests, but they do their job. I'll keep them as far as they don't go in my way.

The only thing I'd be bothered with is the five seconds timeout, which is very long for an unit test. Now, it depends on how often it occurs. Did you consider a basic test (e.g. pinging the server) prior launching curl to avoid starting a needless test.

If you're looking for alternatives, what about splitting your test in two parts: 1/ test how curl is invoked ; 2/ test what your function does with the result ? That way you'll be isolated from the server and you won't need any timeout.