What's a server equivalent of django.test.client to test external URL fetch?

Question

I'd like to make sure a given view in my test is fetching an external URL properly. It uses urllib2 (but this shouldn't matter, since it's blackbox testing). Perhaps a temporary local server of some kind? Is there an app that does this already? (My Googling has failed me.)

Answer 1

Your module under test presumably imports urllib2. You can monkey-patch the module in your test harness to point to your own fake urllib2 - for instance, this needn't be a module, it could be a class instance which has an urlopen method which checks that the correct url is called and returns a suitable response.

Update: Here's a skeleton of the approach. Let's assume your module under test is called mymodule. In your test module (or a separate utility module), you could have:

import urllib2 # the real urllib2

class Urllib2Simulator: # This is a sort of Mock
    #In practice, you could supply additional parameters to this to tell it
    #how to behave when e.g. urlopen is classed
    def __init__(self):
        self.urls_opened = []

    def urlopen(self, url, data=None): # this method simulates urlopen
        self.urls_opened.append((url, data)) # remember what was opened
        #Now, you can either delegate to the real urllib2 (simplest)
        #Or completely simulate what it does (that's more work)
        #Let's keep it simple for this answer.
        #Our class instance will be acting a bit like a proxy.
        return urllib2.urlopen(url, data)

    #similarly define any other urllib2 functions that mymodule calls

and then, in your test code:

class MyModuleTest(unittest.TestCase):

    def test_url_retrieval(self): # use whatever name is best
        real_urllib2 = mymodule.urllib2 #remember it so we can restore it
        simulator = Urllib2Simulator()
        mymodule.urllib2 = simulator # the monkey-patch is here
        # here, invoke your mymodule functionality which is supposed to
        # retrieve URLs using urllib2.urlopen
        mymodule.do_something_which_fetches_urls()
        #restore the previous binding to urllib2
        mymodule.urllib2 = real_urllib2 # restored - back to normal
        #Now, check that simulator.urls_opened contains the correct values

I've used this technique with some success. (It's particularly useful when you want to simulate time passing.) In a unit-test scenario it's less work than setting up a real server. For integration testing I'd probably use a real server, as S. Lott's answer suggests. But this approach allows you to easily simulate different conditions, without having a whole server-based test framework (for example, you can set things up so that the server appears to return particular errors, to test how your code would handle them, or configurable delays in responding, so you can test timeouts, or malformed responses, etc.)

Answer 2

You can use SimpleHTTPServer to rig up a fake web server for testing.

We use the WSGI reference implementation, wsgiref, to rig up a fake web server for testing, also. We like wsgiref because it's a very clean way to create an extensible mock web server. Further, we have status and shutdown WSGI applications that we use to make sure that everything happened properly from the web site's point of view.