A testing solution depends on the functionality of the application and the resources at your disposal. Your first question should be what platform am I writing on and for? The Operating System platform will determine the default tool set you have.
What is the scope of the application?
-Are you testing graphical elements?
-GUI IO?
-Computational functionality?
Will you have access to:
- Virtual Machine(s) - one computer emulating other environments
- Use Multiple local machines and a KVM Switch, more raw processing power, easy access, works well for GUI testing
- Use multiple remote machines via SSH or some other server-client arrangement, a client like VNC or a form of remote desktop might be required .
You will require an automated method of application installation, activation, reset and reporting. Can you run scripts or batchfiles? Must the app run in a GUI mode, does it have a CLI? Do you have access to other services such as web servers, can you open ports to send/receive status reports? Can you remotely login to the test machines, launch and reset the browsers from the command line?. Can you reset a hung Application?
Failure and success can be reported in various ways such as the creation of a local file, the downloading of an image from a server or a ping on a port. Essentially testing will require more work and time because you will need to design, implement and perhaps ironically test a testing suite and apparatus.