Unit testing functions that access files

Question

i have two functions one that builds path to set of files and another that reads the files the function as below

def pass_file_name(self):
    self.log_files= []
    file_name = self.path+"\\access_"+self.appliacation+".log"
    if os.path.isfile(file_name):
        self.log_files.append(file_name)
    for i in xrange(7):
         file_name = self.path+"\\access_"+self.appliacation+".log"+"."+str(i+1)
         if os.path.isfile(file_name):
            self.log_files.append(file_name)
    return self.log_files


def read_log_files (self, log_file_names): 
    self.log_entrys = []
    self.log_line = []
    for i in log_file_names:
        self.f = open(i)
        for line in self.f:
            self.log_line = line.split(" ")
            #print self.log_line
            self.log_entrys.append(self.log_line)
    return self.log_entrys

what would be the best way to unit test these two functions ?

Answer 1

Bind the open name in the module to a function that mocks the file opening.

Answer 2

Personally, I'd build a test harness that set up the required files before testing those two functions.

For each test case (where you expect the file to be present - remember to test failure cases too!), write some known logs into the appropriately named files; then call the functions under test and check the results.

Answer 3

You have two units here:

• One that generate file paths
• Second that reads them

Thus there should be two unit-test-cases (i.e. classes with tests). First would test only file paths generation. Second would test reading from predefined set of files you prepared in special subdirectory of tests directory, it should test in isolation from first test case.

In your case, you could probably have very short log files for tests. In this case for better readability and maintenance it is good idea to embed them right in test code. But in this case you'll have to improve your reading function a bit so it can take either file name or file-like object:

from cStringIO import StringIO

# ...
def test_some_log_reading_scenario(self):
    log1 = '\n'.join([
        'log line',
        'another log line'
    ])
    log2 = '\n'.join([
        'another log another line',
        'lala blah blah'
    ])
    # ...
    result = myobj.read_log_files([StringIO(log1), StringIO(log2)])
    # assert result

Answer 4

I'm no expert but I'll give it a go. First a bit of refactoring: make them functional (remove all class stuff), remove unneeded things. This should make it much easier to test. You can always make the class call these functions if you really want it in a class.

def pass_file_name(base_filename, exists):
    """return a list of filenames that exist
       based upon `base_filename`.
       use `os.path.isfile` for `exists`"""

    log_files = []
    if exists(base_filename):
        log_files.append(base_filename)
    for i in range(1, 8):
         filename = base_filename + "." + str(i)
         if exists(filename):
             log_files.append(filename)
    return log_files

def read_log_files (self, log_files):
    """read and parse each line from log_files
       use `pass_file_name` for `log_files`"""

    log_entrys = []
    for filename in log_files:
        with open(filename) as myfile:
            for line in myfile:
                log_entrys.append(line.split())
    return log_entrys

Now we can easily test pass_file_name by passing in a custom function to exists.

class Test_pass_file_name(unittest.TestCase):
    def test_1(self):
        """assume every file exists
           make sure all logs file are there"""
        exists = lambda _: True
        log_files = pass_file_name("a", exists)
        self.assertEqual(log_files,
                    ["a", "a.1", "a.2", "a.3", 
                     "a.4", "a.5", "a.6", "a.7"])

    def test_2(self):
        """assume no files exists
           make sure nothing returned"""
        exists = lambda _: False
        log_files = pass_file_name("a", exists)
        self.assertEqual(log_files, [])

    # ...more tests here ...

As we assume os.path.isfile works we should have got pretty good testing of the first function. Though you could always have the test actually create some files then call pass_file_name with exists = os.path.isfile.

The second one is harder to test; I have been told that the best (unit)tests don't touch the network, databases, GUI or the hard-drive. So maybe some more refactoring would make it easier. Mocking open could work; or would could actually write some long file in the test function and read them in.

http://stackoverflow.com/questions/1289894/how-do-i-mock-an-open-used-in-a-with-statement-using-the-mock-framework-in-pytho