unit testing for an application server

Question

Hi, I wrote an application server (using python & twisted) and I want to start writing some tests. But I do not want to use Twisted's Trial due to time constraints and not having time to play with it now. So here is what I have in mind: write a small test client that connects to the app server and makes the necessary requests (the communication protocol is some in-house XML), store in a static way the received XML and then write some tests on those static data using unitest.

My question is: Is this a correct approach and if yes, what kind of tests are covered with this approach?

Also, using this method has several disadvantages, like: not being able to access the database layer in order to build/rebuild the schema, when will the test client going to connect to the server: per each unit test or before running the test suite?

Answer 1

haven't used twisted before, and the twisted/trial documentation isn't stellar from what I just saw, but it'll likely take you 2-3 days to implement correctly the test system you describe above. Now, like I said I have no idea about Trial, but I GUESS you could probably get it working in 1-2 days, since you already have a Twisted application. Now if Trial gives you more coverage in less time, I'd go with Trial.

But remember this is just an answer from a very cursory look at the docs

Answer 2

"My question is: Is this a correct approach?"

It's what you chose. You made a lot of excuses, so I'm assuming that your pretty well fixed on this course. It's not the best, but you've already listed all your reasons for doing it (and then asked follow-up questions on this specific course of action). "correct" doesn't enter into it anymore, so there's no answer to this question.

"what kind of tests are covered with this approach?"

They call it "black-box" testing. The application server is a black box that has a few inputs and outputs, and you can't test any of it's internals. It's considered one acceptable form of testing because it tests the bottom-line external interfaces for acceptable behavior.

If you have problems, it turns out to be useless for doing diagnostic work. You'll find that you need to also to white-box testing on the internal structures.

"not being able to access the database layer in order to build/rebuild the schema,"

Why not? This is Python. Write a separate tool that imports that layer and does database builds.

"when will the test client going to connect to the server: per each unit test or before running the test suite?"

Depends on the intent of the test. Depends on your use cases. What happens in the "real world" with your actual intended clients?

You'll want to test client-like behavior, making connections the way clients make connections.

Also, you'll want to test abnormal behavior, like clients dropping connections or doing things out of order, or unconnected.

Answer 3

I think you chose the wrong direction. It's true that the Trial docs is very light. But Trial is base on unittest and only add some stuff to deal with the reactor loop and the asynchronous calls (it's not easy to write tests that deal with deffers). All your tests that are not including deffer/asynchronous call will be exactly like normal unittest.

The Trial command is a test runner (a bit like nose), so you don't have to write test suites for your tests. You will save time with it. On top of that, the Trial command can output profiling and coverage information. Just do Trial -h for more info.

But in any way the first thing you should ask yourself is which kind of tests do you need the most, unit tests, integration tests or system tests (black-box). It's possible to do all with Trial but it's not necessary allways the best fit.

Answer 4

You should use Trial. It really isn't very hard. Trial's documentation could stand to be improved, but if you know how to use the standard library unit test, the only difference is that instead of writing

import unittest

you should write

from twisted.trial import unittest

... and then you can return Deferreds from your test_ methods. Pretty much everything else is the same.

The one other difference is that instead of building a giant test object at the bottom of your module and then running

python your/test_module.py

you can simply define your test cases and then run

trial your.test_module

If you don't care about reactor integration at all, in fact, you can just run trial on a set of existing Python unit tests. Trial supports the standard library 'unittest' module.