How can one mock/stub python module like urllib

Question

I need to test a function that needs to query a page on an external server using urllib.urlopen (it also uses urllib.urlencode). The server could be down, the page could change; I can't rely on it for a test.

What is the best way to control what urllib.urlopen returns?

Answer 1

Probably the best way to handle this is to split up the code, so that logic that processes the page contents is split from the code that fetches the page.

Then pass an instance of the fetcher code into the processing logic, then you can easily replace it with a mock fetcher for the unit test.

e.g.

class Processor(oject):
    def __init__(self, fetcher):
        self.m_fetcher = fetcher

    def doProcessing(self):
        ## use self.m_fetcher to get page contents

class RealFetcher(object):
    def fetchPage(self, url):
        ## get real contents

class FakeFetcher(object):
    def fetchPage(self, url):
        ## Return whatever fake contents are required for this test

Answer 2

The simplest way is to change your function so that it doesn't necessarily use urllib.urlopen. Let's say this is your original function:

def my_grabber(arg1, arg2, arg3):
    # .. do some stuff ..
    url = make_url_somehow()
    data = urllib.urlopen(url)
    # .. do something with data ..
    return answer

Add an argument which is the function to use to open the URL. Then you can provide a mock function to do whatever you need:

def my_grabber(arg1, arg2, arg3, urlopen=urllib.urlopen):
    # .. do some stuff ..
    url = make_url_somehow()
    data = urlopen(url)
    # .. do something with data ..
    return answer

def test_my_grabber():
    my_grabber(arg1, arg2, arg3, urlopen=my_mock_open)

Answer 3

Another simple approach is to have your test override urllib's urlopen() function. For example, if your module has

import urllib

def some_function_that_uses_urllib():
    ...
    urllib.openurl()
    ...

You could define your test like this:

import mymodule

def dummy_urlopen(url):
    ...

mymodule.urllib.urlopen = dummy_urlopen

Then, when your tests invoke functions in mymodule, dummy_urlopen() will be called instead of the real urlopen(). Dynamic languages like Python make it super easy to stub out methods and classes for testing.

See my blog posts at http://visionandexecution.org for more information about stubbing out dependencies for tests.

Answer 4

Did you give Mox a look? It should do everything you need? Here is a simple interactive session illustrating the solution you need:

>>> import urllib
>>> # check that it works
>>> urllib.urlopen('http://www.google.com/')
<addinfourl at 3082723820L ...>
>>> # check what happens when it doesn't
>>> urllib.urlopen('http://hopefully.doesnotexist.com/')
#-- snip --
IOError: [Errno socket error] (-2, 'Name or service not known')

>>> # OK, let's mock it up
>>> import mox
>>> m = mox.Mox()
>>> m.StubOutWithMock(urllib, 'urlopen')
>>> # We can be verbose if we want to :)
>>> urllib.urlopen(mox.IgnoreArg()).AndRaise(
...   IOError('socket error', (-2, 'Name or service not known')))

>>> # Let's check if it works
>>> m.ReplayAll()
>>> urllib.urlopen('http://www.google.com/')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.5/site-packages/mox.py", line 568, in __call__
    raise expected_method._exception
IOError: [Errno socket error] (-2, 'Name or service not known')

>>> # yay! now unset everything
>>> m.UnsetStubs()
>>> m.VerifyAll()
>>> # and check that it still works
>>> urllib.urlopen('http://www.google.com/')
<addinfourl at 3076773548L ...>

Answer 5

And what if I want to mock out the Response class of urllib2?

Answer 6

I am using Mock's patch decorator:

from mock import patch

[...]

@patch('urllib.urlopen')
def test_foo(self, urlopen_mock):
    urlopen_mock.return_value = MyUrlOpenMock()