NetBeans Platform Application: How to create test environment for first automated end-to-end test through GUI?

Question

How to write first end-to-end automated test for NetBeans Platform Application on Java (not just capture & replay) which suppose run the application and test if click on the Main Menu -> File -> Exit menu item will close it?

Given: installed jdk1.6.0_17 and apache-maven-2.2.1

Then:

mvn archetype:generate -DarchetypeGroupId=org.codehaus.mojo.archetypes -DarchetypeArtifactId=netbeans-platform-app-archetype -DarchetypeVersion=1.1 -DgroupId=stackoverflow.com -DartifactId=netbeans-app -Dversion=1.0-SNAPSHOT

Then:

mvn install
cd netbeans-app\application\target\foo\bin\
foo.exe

And see that default GUI application is successfully running.

Which open-source GUI testing framework would you recommend for completing this task? Abbot/Costello, Jelly/Jemmy, UISpec4J, Marathon, FEST-Swing or WindowLicker?

Answer 1

I would say:

Jemmy is probably out because it works best as an Eclipse plug in. (not sure about Jelly)

If you have custom components then Abbot/Costello is out, I found that it reacts unpredictably to those. With custom components, it works like a charm though.

If running the tests from within your Netbeans IDE is very important to you, then Marathon is out as well, because you will need to the tests from the command line (or its standalone application of course). It has, IMO, the best support for custom components though.

That leaves FEST really. FEST integrates really well into running from Netbeans, because it can be run the same way as JUnit, with its test results getting displayed right in Netbean's IDE's "Test Results" panel. And it comes with a really neat and simple plugin that makes it a no-brainer to set up the FEST tests: Geertjan's NetBeans plug-in for FEST-Swing. The main drawback of FEST is that it's replay only, no record, and test cases have to be written manually.

I personally had to make a similar decision on choosing test tool for an automated testing framework several months ago - I went with Marathon, because custom component testing in my target application was of utmost importance. Other factors played out in the decision as well: Ability to integrate its tests into a continuous integration system, such as cruise control; and testers with limited programming ability were to write/ record the test scripts; was also important in our case.

For your purposes, I would say for your specific question FEST seems like the most logical choice with the information known now.

However, I suggest that you spend a day on each test tool, attempting to hook each one up and take notes about the pros and cons. AT the end of it, you might find that you have answered your own question better than I have because it is likely that you will discover more requirements than you yourself are aware of right now - I know I did!

HTH

---

Disclaimer: I have no experience with UISpec4J, Jelly and WindowLicker, so my answer may be inomplete.

---

Edit: Just to add on, FEST can test the application when exiting. It fixed that recently. Marathon cannot do the same - and that affects my testing for test cases which involve testing the closing of the application.
As far as I am aware, Abbot/Costello also is affected by the same problem (but I have not kept up to date on this), and for the other test tools, I have not tried testing the exit of the application.