Using JUnit as an acceptance test framework

Question

OK, so I work for a company who has openly adopted agile practices for development in recent years. Our unit tests and code quality are improving. One area we still are working on is to find what works best for us in the automated acceptance test arena. We want to take our well formed user stories and use these to drive the code in a test driven manner. This will also give us acceptance level tests for each user story which we can then automate.

To date, we've tried Fit, Fitnesse and Selenium. Each have their advantages, but we've also had real issues with them as well. With Fit and Fitnesse, we can't help but feel they overcomplicate things and we've had many technical issues using them. The business haven't fully bought in these tools and aren't particularly keen on maintaining the scripts all the time (and aren't big fans of the table style). Selenium is really good, but slow and relies on real time data and resources.

One approach we are now considering is the use of the JUnit framework to provide similiar functionality. Rather than testing just a small unit of work using JUnit, why not use it to write a test (using the JUnit framework) to cover an acceptance level swath of the application? I.e. take a new story ("As a user I would like to see basic details of my policy...") and write a test in JUnit which starts executing application code at the point of entry for the policy details link but covers all code and logic down to the stubbed data access layer and back to the point of forwarding to the next page in the application, asserting on what data the user should see on that page.

This seems to me to have the following advantages:

• Simplicity (no additional frameworks required)
• Zero effort to integrate with our Continuous Integration build server (since it already handles our JUnit tests)
• Full skillset already present in the team (its just a JUnit test after all)

And the downsides being:

• Less customer involvement (though they are heavily involved in writing the user stories in the first place from which the acceptance tests will be written)
• Perhaps more difficult to understand (or make understood) the user story and acceptance criteria in a JUnit class verses a freetext specification ala Fit or Fitnesse

So, my question is really, have you ever tried this method? Ever considered it? What are your thoughts? What do you like and dislike about this approach? Finally, please only mention alternative frameworks if you can say why you like or dislike them more than this approach.

Answer 1

IMHO Not a good idea. Apart from the 2 cons that you mentioned, how about the vast amount of effort required to build an acceptance test framework that will deal all kinds of web/UI/glue scenarios that you might have? Your point "Simplicity (no additional frameworks required)" is incorrect as your organization would have to invest on building a more elaborate framework on top of JUnit. It may be more than JUnit.

Answer 2

You mentioned some good points about the pros and cons. Can I add:

Upsides: I like the idea of being able to test everything without having to have a database, a web server. This helps enormously in understanding the code, and in refactoring.

Downsides:

The JUnit code will be complex and hard to maintain. You will have to reproduce all of the interactions between code and user within the Junit test, which very quickly becomes complex. If all you are doing is testing one single entry point, then this could well work, but if you are testing multiple screens (a wizard or something like that), then it can quickly become fragile and unmanageable. The problem in my experience is almost always the plumbing code between pages, or the setup code required for a page.

One other thing to consider is the flexibility of what you're trying to do. With Fitnesse, it is fairly easy to add another test if you discover a bug. In fact, this is the idea behind Fitnesse: the 'user' can add another test without having to change the code. Don't underestimate the value of this.

So I would suggest two things:

1) Try it. Take a user story (not too complex, not too simple), and do it in JUnit. Compare what you've done with the Selenium/Fitnesse etc, and see if it is worthwhile. See if you 1. actually find bugs, 2. produce brittle, hard to manage code.

2) Could you use the data created by the Fitnesse as input to your JUnit tests (thus removing the need for a web server etc). You could possibly read the data and call the methods in the Java classes directly. Again, you may have problems with setup & database code.

Answer 3

Hey.

Business and UAT
Business most of the time will loose interest in any testing tool sooner or later. They are Business not testers after all, so they won't commit much to writing/maintaining test scripts. They are Business, they want to do business. From their point of view testers/developers/other IT guys are responsible for delivering them software of certain quality.
Even if Agile is your way, and you get some level of commitment from business don't expect them to behave like tester with every sprint/iteration or whatever you have. It is really not their job.
One digression: User Acceptance Tests are be manual tests executed by the user(s), so he(they) can say is product what he(they) asked for.
So whenever feature (or set of features) is ready, do presentation for business, let them play with it, let them say is it what they need. Don't force them to check this feature with every iteration.

Now back to your acceptance tests. Whenever you have story from client to do, implement test(s) for it in your test infrastructure. You write it, you run it, you maintain it. It may be wise to have test for feature and drive development with ADD/BDD/TDD or whatever you want to call it. It should be obvious that this test(s) than contribute to your regression tests suite in next iterations.

So my point of view is, Business won't be keen on writing/maintaining tests, especially when set of features grows. Don't fight with it, adapt to it. Let them do just (manual) User Acceptance Tests at the end of iteration for new features. For your own sake create tests for features they want have. Create feature acceptance tests for each feature. Let those tests become your Regression Testing Suit. Accept that t is your responsibility to maintain it.

Acceptance testing Framework
JUnit, huh? Then try with WatiJ for web, and Abbot for desktop. Keep in mind that those test won't be simple Unit Tests. You want something that will tests features of real application, you want Test Script to execute specific Scenario/Business Process. This means you need to approach writing those tests like writing application they are testing: DRY,KISS, YAGNI, Design Patterns - everything applies. In the end, it will be writing software, that just happens to test other software you write.

Will those test become large and complex with this approach? Well, larger and more complicated than unit tests. But they are not testing single unit but whole application. Besides they will be far more simpler and smaller than application you are writing in the first place.

Will you bee writing testing framework? If you want to succeed with this approach, than yes, you will be writing testing framework - collection of test classes and helper classes that have some knowledge about your application implemented. But you won't be extending JUnit. JUnit here is just some API you use in your framework.