Testing whole programs. best practices

Question

I am currently developing a library and a set of programs using this library, in python. Unit testing dictates that I import each module from the library, and test the classes and routines within. No problem with that. I have a separate test directory containing all my tests and importing the library modules, which I run while developing.

However, when it comes to testing the programs, things change. To be tested, the programs must run as a whole. The programs assume to find the library installed (which could actually be the case, albeit wrong, if I installed a previous version of it on my machine, adding further trouble). At the moment, my programs are run by a testsuite with a PYTHONPATH definition that I perform by hand, before the deployment (IOW, I don't perform the install), but I don't think I am doing it right. I feel like in general, a program should be tested for functionality when fully deployed, but this would mean that I have to install it every time I want to perform functional testing.

What is your experience and suggestions concerning functional testing of whole programs ? do you do it before or after deployment, and how?

Thanks

Note that I don't include the python tag on purpose. Although my problem is python specific, and I would prefer python-related answers, I think that contribute can be brought also from experts in other languages.

---

Edit: as reported in a comment, the fact is that my program, when installed, has to import modules whose path can be found only when deployed (I download and install the dependencies on the fly, they are not installed on my machine). I cannot manipulate sys.path from the test, because that would imply that I modify the sys.path of a program (my executable) from another program (the testsuite, which runs and spawn a system() call).

In other words, the only way I have to test the program without deploying is to execute the program with PYTHONPATH set to the dir containing the deps and the library that program uses installed by the make script (which, as I said, downloads, compiles and "installs" everything in a temporary directory).

At deployment, the deps and the executables are packaged together in a "OSX bundle"-like structure, which is fully executable and relocatable.

Edit:

added a 150 bounty to see if I can get a little more feedback.

Edit:

I appreciated all the answers and voted up all of them. The choice has been a hard call for me, but I've been recalled by LudoMC of the V-model approach to testing I studied long ago. Thanks to all for the very good answers.

Answer 1

Well, (automated) testing is always a tradeoff, so there is no single right way to do it.

But yes, ideally you should automatically do a full install/deployment of your program before running the tests. That way you also test your installer.

Maybe you can write an installer wrapper that automates this for you. If it is too much work, you can probably also run the functional test inside a deployment that was created manually.

As a compromise, you could have your installation run only once at the start of a test run, then run all the functional tests w/o reinstallation, to make the tests run faster.

Answer 2

Test as much as you can within reason. If something would be great to test but it's a lot of effort, then don't test it ... yet. Only when you find that you run into issues at that place again and again, then spend the effort. Never assume in advance where the problems will lie (except when you know ahead of time ... but then, knowing isn't assuming!).

So if installing the program rarely causes any problems, don't try to test it. If your deployment is brittle, maybe write a test which checks that the install archive is complete rather than trying to install: You're not testing the system's installer, you're testing your package.

Answer 3

In our company, we are using a pretty commonly used V-Model as development process, where the unit tests are done within the implementation phase, the integration tests are done against the architecture/design phase, and the system tests against the requirements phase.

So in your case, from what I understand, you want to test your application as a whole, on a functional level. So it has to be done against the requirements.

Thus, you need a requirement document, whether a full-text scenario or (better) a UML use case diagram covering (ideally extensively) the use cases of your application (usually one of the first phase to be achieved). You then have to write test cases covering every use case and pass these test cases. It can be done through manual or automatic testing with well-known (and pretty expensive) tools.

For the when, we usually do these System tests after deployment (test team is using the installer provided by the development team), because we also test the installer itself, where Integration tests are before or after deployment depending on the case.

In your case, if the installer is error-free, and you're 100% sure than testing before the deployment using your PYTHONPATH variable will never bring a bug once deployed, then you can choose to test before deployment. It's pure risk-management and it's your call, because you're the one who knows the best the pro and cons of this for your applications. (Personnaly, I don't see why bugs cannot exist there, they are everywhere :-) )

Hope that helps and I'm not off-topic

Answer 4

We've faced a similar issue about deployment, and are using virtualenv to test deployments. Especially with the --no-site-packages option, it's like having a pristine install, and no mucking with PYTHONPATH beyond a well-tested 3rd-party solution. And just to be sure, we run the entire thing, virtualenv and all, in a fresh virtual machine.

btw, a helpful trick with virtualenvs is ./setup.py develop.

(From one journeyman to another...)

Answer 5

For sure you should test your program in completely accurate post-deployment scenarios.

You might consider integrating a self-test capability right into your production code, and providing a mechanism to run it on demand. It may make sense to run the self-test only during your own system testing, but your installer could run it as a final step, your users could run it from a menu item, or you even could have a "power-on self test" at startup.

There are lots of techniques for doing this. If you deploy a web service, the self-test could be invoked via an administrative URI which would make HTTP requests on itself to verify that they work. Static self-tests are probably all you need, but you can also load and run test scripts dynamically. For the latter, you can use your language's interpretation facilities, or if there are none, you can embed an interpreter into your software (eg, TCL for C/C++ and Rhino for Java) or write your own mini-interpreter.

Answer 6

Doing functional testing has limitations, as you cannot generally simulate a person using the GUI, and you won't test abnormal actions a user may do as they may just click or type things that wouldn't make sense.

The best thing to do is to have as little business logic in your view part of the application, then you can better test, as you can test the controllers on down, or if you have it, your web/REST services on down.

You need to try to be as complete in testing, test both positive and negative cases, to ensure that you know how the application will work regardless of what is sent by the user.

I have found NUnit and junit to be useful for doing testing this way, but for functional tests, ultimately I end up resorting to just going through a test script manually, in most cases. I tend to have a way to run all the tests for the application by using something like ant, to help make certain that as much can be tested as possible, then I would go home, as the full suite of tests could take a long time to run.