Software Test Automation - Masters Thesis

Question

Hi,

i am trying to write a thesis about Software Test Automation. I plan to compare the two Approaches of Recording and Programming of Test Scripts, and to discuss about several Automation Frameworks, for example Abbot, Selenium, Yemmy, FEST, etc ... Also in my Thesis will be a short overview about Softwaretesting Techniques and maybe a comparison of automated testing to software testing.

EDIT: I am planning to the aspects of testing an Application over it's GUI. So my Tests would be mostly on the Blackbox Side of the testing world. I have not planned to write about Unit Testing.

At the Moment i read pretty much about the different Automation Frameworks, but i may not have the time to review all of them. So i plan to read about them and make the Thesis more literature - based.

• Do you think this Topic could be successful?
• Do you have any other Ideas concerning this Topic?
• Can you reccommend Literature?
• What is your Opinion about this Topic?

Answer 1

I don't know about literature, but I think the ACM publications in your school library would probably produce results. Particularly the SIG* newsletters. (Perhaps SIGSOFT?)

It does sound like a good Master's thesis to me. Of course your thesis advisor is the final word on that. You should go talk to them.

Answer 2

As a literature-based review, this makes an excellent topic; there's plenty of material out there. Obviously I'm not going to start to get into all the details of that, since that's your job as the author. :-)

However, though I'm not familiar with the original research requirements for a master's thesis, this certainly wouldn't be enough for a PhD thesis. I would look for original work you could add to this. One idea would be a taxonomy of testing methods and systems. You might also examine the role of testing as compared to formal verification.

Answer 3

Software Test Automation is a big topic, and you may want to narrow your focus rather than attempt to cover a mix of frameworks, playback/record, overview of techniques, automated vs. not.

Entire books have been written about software test automation:

• As a general topic
• Focusing on functional/feature tests (FIT)
• Focusing on unit tests
• Focusing on unit tests using one particular language and framework

Frameworks are aimed at different types of testing:

• Unit testing
  • Test-Driven Development
  • Behavior-Driven Development
• Feature/Functional testing
• GUI testing (Windows, Java GUIs, X Windows, etc.)
• Web testing
• Performance Testing
• Security Testing

I would consider focusing on frameworks (or techniques, or whatever) in one of these areas rather than trying to cover them all. Or pick a couple of these areas and contrast them.

The issue of playback/record vs. handwritten tests seems old to me. In the 1980's vendors liked to push playback/record for Windows GUI automation. It made for great demos and high hopes. But it also made for brittle tests and shelfware. Playback/record is nice to get you started with a tool, but to be maintainable, you generally need scripts written at a higher level. That ushered in a new era of spreadsheet and keyword-based approaches, and eventually FIT/FitNesse.

Answer 4

A survey of the literature should be a fine focus for a MS thesis. It sounds like you want to just talk about black-box GUI-driving customer-facing tools, which is a reasonably small niche.

You /might/ want to have a page or two on the whole world of test tools - unit testing, security, load, etc, as someone mentioned above. But I think you targeted your niche pretty well.

I would think with a 6-credit thesis you should have plenty of time to explore and try out some of the bigger commercial and open-source tools as well as survey the literature. I would encourage you to look into both commerical tools (quick test pro, test complete) and also keyword-driven automation - selenium RC, for example. Someone else mentioned testing "behind the GUI" eg FIT/Fitnesse, it might be worth discussing and evaluating.

I cover black-box, functional test automation in my monthly column in the December 2008 issue of software test and performance magazine:

http://www.stpmag.com/issues/stp-2008-12.pdf (page 7)

That's the one page scratch-the-surface introduction. The five-sentence introduction is that screen record/playback tools compare everything, so if your GUI changes at all, in any way (even if you just change the screen resolution) that can come back as a false error. Keyword-driven tools only check what you tell them to check - they miss if a button is suddenly disabled for no good reason or an icon is not transparent.

Only a human is good at checking that hidden assertion at the end of every test case "... and nothing else strange happened."

So computer-based test execution and evaluation can have some value, but it should be part of a balanced breakfast.

Other things to look into:

• James Bach's "Software Test Automation Snake Oil"
• Kaner, Bach and Pettichord's book "Lessons Learned in Software Testing"
• My blog post on test Frameworks - http://xndev.blogspot.com/2007/09/whats-test-framework.html (it's the number 4 google result for "what is a test framework", so I'm comfortable recommending it)
• The minefield analogy ( http://www.testingperspective.com/tpwiki/doku.php?id=minefield )
• The papers of Doug Hoffman on test automation: http://www.softwarequalitymethods.com/H-Papers.html
• The classic "shelfware" problem of test automation
• The anti-intellectualism pushed by some proponents of the blackbox test automation community
• Kaner's Black Box Software Testing Course
• James Bach's work on /cognitive/ testing
• Context Driven Software Testing
• Jon Kohl's work on "Man and Machine", or the cyborg approach (instead of computer-alone test execution and evaluation)

I hope that helps.

Answer 5

An excellent book on test automation has just been published this year: “Implementing Automated Testing”, Elfriede Dustin, Thom Garrett & Bernie Gauf, Addison Wesley.