Automatically generating tests by recording and replaying how methods are called

Question

Here is a program that I think should exist: When an application is running, it automatically records all calls to all methods. It then creates a test based on each call. (Such a test might be called a unit test, but I'm not doing so here because there is an issue of capturing state.)

For a typical app, this program would generates thousands and thousands of tests automatically after just a few uses of the program.

Does this program exist? If not, why not? Is there a program that does something close to this?

A more sophisticated version of this program would do the following:

1. It would recombine parameters from multiple calls to create a new test. For example: A method is called with (1,A) and (2,B). This utility would generate unit tests that call the method with (1,B) and (2,A).
2. It would perturb the calls to create new tests. For example: A method is called with two integers (1,2). This utility would create tests that call the method with (0,2), (1,3), (0,3), etc.

I appreciate the fact that some of the tests generated by such a utility would be incorrect. Nevertheless, I think such a utility would be extremely useful, particularly when testing legacy apps.

Answer 1

In the 3D graphics world they exist in a similiar manner - they record all API calls and you can "play" them back later and get the exact same sequence of frames, etc.

I don't see why the utility you mention would need to create additional tests (that (1,A) business). Is (2, B) even a valid combination? The method-recording is a good enough idea in it's own right without having it recombining things.

Answer 2

If you're doing functional testing of web applications, there are frameworks like Selenium and such available that record the user input in the browser and can replay it for testing purposes.

Answer 3

There are similar tools, but they're based on capturing user input and replaying it.

What you described would IMO be not very useful, since the tests produced that way would not test desired behavior but simply existing behavior. Even if it's wrong. Even if it's an insignificant implementation detail. The tests would be extremely brittle and extremely redundant - i.e. they would lack two of most important properties of automated tests: stability and independence.

Any change in the application, any refactoring at all, would result in thousands of broken tests. The end result would either be an abandoned and ignored test suite, or an app frozen in time that nobody can work up the motivation to change in any way because it entails 10 times as much work adjusting test cases.

It would perturb the calls to create new tests. For example: A method is called with two integers (1,2). This utility would create tests that call the method with (0,2), (1,3), (0,3), etc.

How would the tool know what effect these generated calls should have? Fuzz testing is a similar concept, but does not test application correctness, only the absence of buffer overflows and similar catastrophic errors.