Using SimpleHTTPServer for unit testing

Question

Hello,

I'm writing a Python module that wraps out a certain web service API. It's all REST, so relatively straightforward to implement.

However, I found a problem when it comes to unit testing: as I don't run the services I made this module for, I don't want to hammer them, but at the same time, I need to retrieve data to run my tests. I looked at SimpleHTTPServer, which would be OK.

I solved part of the issue I had, but now, since I can't seem to terminate the thread, I get issues with "address already in use" when launching the test application more than once.

Here's some sample code

PORT = 8001

handler = SimpleHTTPServer.SimpleHTTPRequestHandler

httpd = SocketServer.TCPServer(("", PORT), handler)

httpd_thread = threading.Thread(target=httpd.serve_forever)
httpd_thread.setDaemon(True)
httpd_thread.start()

api_data = urllib.urlopen("http://localhost:8001/post/index.json")
print "Data start:"
print json.load(api_data)

Where "index.json" is a mock JSON file I made which substitutes the real thing. How can I clean things gracefully after the program terminates?

Answer 1

SimpleHTTPServer sounds like the right tool for the job, if you want it to run in the background you can use the threading module.

If you want us to solve the 'trouble' you're having, please post some sample code and explain what goes wrong with it...

Answer 2

We use a server built on wsgiref. http://docs.python.org/library/wsgiref.html

It's very easy to add features to this server as we add unit tests.

We start the server with subprocess. http://docs.python.org/library/subprocess.html?highlight=subprocess#module-subprocess

We do not use threads for this kind of testing. Why? (1) Our unit test server is rather complex and we'd like to keep it completely isolated from client applications. (2) Our client applications will be separate processes on separate hardware, we need to be sure that we have realistic performance expectations for that configuration. (3) It's simpler. (4) It's portable across all platforms. (5) It's trivial to change from stand-alone unit testing to integration testing with a production-like server that's already running.

We actually have a small WSGI application that makes the server shutdown in a reasonably controlled manner so that the logs are shutdown properly.

Answer 3

Try using a subclass of TCPServer with allow_reuse_address set True:

class TestServer(SocketServer.TCPServer):
    allow_reuse_address = True

...
httpd = TestServer(("", PORT), handler)