Understanding how software testing works and what to test.

Question

Intro:
I've seen lots of topics here on SO about software testing and other terms I don't understand.

Problem:
As a beginner developer I, unfortunately, have no idea how software testing works, not even how to test a simple function. This is a shame, but thats the truth. I also hope this question can help others beginners developers too.

Question:
Can you help me to understand this subject a little bit more?

Maybe some questions to start would help:

• When I develop a function, how should I test it? For example: when working with a sum function, should I test every input value possible or just some limits? How about testing functions with strings as parameters?
• In a big program, do I have to test every single piece of code of it? When you guys program do you test every code written?
• How automated test works and how can I try one? How tools for automated testing works and what they do?
• I`ve heard about unit testing. Can I have a brief explanation on this?
• What is a testing framework?

If possible please post some code with examples to clarify the ideas.

Any help on this topic is very welcome! Thanks.

Answer 1

Well, Usually, there're three kinds of tests. Unit Tests, System tests and QA tests. Unit tests, as the name suggests, test small units - separate functions and classes.

For all modern development environments there're unit test frameworks. There's Nunit for .net, as well as MS unit test framework in Visual Studio, CPPUnit for C++, JUnit, etc. All meant for one thing: connect to parts of your program, run your pre-defined scripts and report error.

CPPUnit, for example, is based on macros like CPPUNIT_ASSERT_EQUAL, meant to be used as something like this: CPPUNIT_ASSERT_EQUAL(sum(arr), 17). And it would say if it's not equal, in which case the test will be considered failed.

You are supposed to develop the tests for every function, and after that - you are not afraid to change and optimize the code. It is generally referred to a "repeatability" - ability to do a complex action, such as full testing of all codebase, with a single click.

Unit tests are required for every moden software development, because the experience shows that they improve development speed. It is also suggested that unit test code may serve as a kind of "documentation" for the library.

System tests are automated tests of larger, high-level functionality. The idea of system tests is to feed clean input (such as databases, user input, etc) to test the whole thing, validating the output against pre-defined resutls. It is essential that the system ouput is deterministic, and depends only on the input. Every time the system changes, the system tests change also.

TDD is a cool-sounding bad idea, suggesting that you should not develop anything before implementing the proper automated tests, and then writing code to satisfy the tests. It is regardedd as failure, because changes in the design are inevitable during development, and a small design change usually causes drastic changes in the unit tests.

Manual QA is the final, and most important type of software testing. The idea is to prepare a test plan, whcih is done during design and coding phases, collecting all ideas developers had during coding of every if statement, how to actually make this particular if statement run along the less expected code path. The QA personnel, meant to be capable of anything that can be done with the program without development environment, can follow the resulting test procedure and their own ideas, to find more bugs.

Answer 2

I found the book 'JUnit Pocket Guide' by Kent Beck as an excellent (and cheap and compact!) introduction to Unit Testing: the book is split roughly into sections on the merits of test-driven programming and general testing techniques and then goes into the specifics of the JUnit framework (which he co-created).

http://www.amazon.co.uk/JUnit-Pocket-Guide-Kent-Beck/dp/0596007434/ref=sr_1_7?ie=UTF8&s=books&qid=1276811600&sr=8-7

With regard to your request for some illustrative examples of Unit Testing; this JUnit Primer isn't bad:

http://www.clarkware.com/articles/JUnitPrimer.html#testcase

Answer 3

You can divide the tests into three big branches (actually there are more, but if you are a beginner you have to understand the base first): Alpha, Beta, Full working code.

1) In alpha you try every single value for every single piece of code.
2) In beta you give the presumably working application to the world, in order to get a feedback from testers and improve it.
3) When your app is fully working you just have to wait for the error-report (it always comes) of a user who discovered that issue and fix it with a patch.

When I develop a function, how should I test it? For example: when working with a sum function, should I test every input value possible or just some limits? How about testing functions with strings as parameters?

In alpha you have to think about ALL the possibilities your code can be dial with. In the case of a sum() function, they are a few, but if you are developing let's say a big database server application, you need to take in account every possible input and handle every possible error, think of the end user as a stupid person or even as a malicious cracker.

n a big program, do I have to test every single piece of code of it? When you guys program do you test every code written?

I do, you should, micro$oft doesn't. (catch the joke ;) )

How automated test works and how can I try one? How tools for automated testing works and what they do?

No automation can overcome human intelligence and a good debugger (that's the only tool you really need!)

I`ve heard about unit testing. Can I have a brief explanation on this?

http://en.wikipedia.org/wiki/Unit_testing

What is a testing framework?

en.wikipedia.org/wiki/Test_automation_framework

Answer 4

Let's start with the obvious:

How does testing work ? In test-driven development you first think about the functionality you want to implement and then write a test for it. In the example you gave of a sum function it is quite obvious what it should do. You then write a test that makes sure that the summation worked.

Unit tests should be as lightweight as possible so you can run them each time you hit the build button. When you do this now in this example your test will fail, because you have not yet implemented a sum function.

Now you write the actual function and continue debugging and implementing until the test passes. Then you are sure you have implemented the feature you wanted.

Now how should you design your test ? You cannot test all, that is impossible. As an example lets say you take user input you have to validate. Then it would be a natural thing to write at least two test cases for that validation code: one that makes sure that valid input is parsed as such. The second test case gets invalid input and you make sure that it fails, raises an exception or whatever behavior you desired. So in this case it is good to have a positive test which is expected to pass and a negative test which checks if invalid input is not validated.

When should one test? As I mentioned before the test should be lightweight enough so that they can be run on each build. And yes, run all of them. This makes sure that you do not miss a dependency in your code that breaks things far away from the point you edited.

Can anything be tested ? Well, usually methods that rely on external ressources are hard to test. What I mean by that are databases, network connections or specific hardware and drivers. It can be done but then you have to set up a larger test setup.

Bugfixing and tests A typical scenario where test get really useful is if you are squeezing bugs. Not literally of course. If you have a bug you have to fix, first try to write a test for it. And then fix your code until your test passes. From this point on this test "watches over your code" that this bug will never come back again.

What do you gain by testing ? In my opinion there are many things

1. More modular, easier to maintain code because it has to be testable
2. Confidence. Having a code base that is largely tested gives you the confidence that it works as expected and stays like this.
3. You find bugs early. This means you can fix them more easily.

It takes some effort to get used to using tests but I think it is worth it. Especially if you are writing libraries of some kind.

Answer 5

What to test

There are a couple of really useful concepts that helped me work this out:

• http://en.wikipedia.org/wiki/Equivalence_partitioning
• http://en.wikipedia.org/wiki/Boundary-value_analysis

Equivalence partitioning says "Running the test in this context with these events is equivalent to this other test over here, so I don't need to do both."

Boundary analysis says "Here's the point at which a change in context or a different event causes a different outcome, so I will want tests on either side of this boundary."

Thinking about these can really help to minimize the number of tests you need to write.

When to test

I always manually test my code to make sure that it works. I often write an additional unit test which covers every line of code I'm about to write. The unit test fails before I write the code to make it pass. If it's something a user will do, then I often write an acceptance test or automated scenario, too.

Automated testing

Automated testing means either scripting or recording a test so that you don't have to do it manually - your script or code will do it for you. There are two kinds of automation tools: BDD- or English-readable acceptance-testing fixtures which let you write the scripts, and automation wrappers which let you automate more easily. So for instance, you might choose to use GivWenZen or Fitnesse as your fixture, and Selenium as your web automation tool.

If you've got Visual Studio 2008+ you could download the source code from here and try out the scenario to see it run:

http://code.google.com/p/wipflash/

Unit testing

In its strictest sense, Unit Testing means testing an element of code in isolation from all other elements of code. We use mocks in place of real pieces of code that our Code Under Test requires. In reality we are often pragmatic about the nature of "units" - I don't mock out domain objects, for instance.

However, unit testing in the way that most people use it isn't really about testing. Mostly it's a mechanism for achieving clean design and separation of responsibilities, because you're forced into this by trying to test it.

I like to think of unit testing as "writing an example of how I'm going to use the code I'm about to write". The example happens to reside in a unit-testing framework and be executable.

Testing frameworks

• Unit-testing frameworks help you to run a lot of tests together.
• Acceptance-testing frameworks help you to run a larger scenario and are often English-readable.
• Automation frameworks usually sit underneath your English-readable frameworks and help you to interact with an application.

Good luck!