Using Python, what's the best way to create a set of files on disk for testing?

Question

I'm looking for a way to create a tree of test files to unit test a packaging tool. Basically, I want to create some common file system structures -- directories, nested directories, symlinks within the selected tree, symlinks outside the tree, &c.

Ideally I want to do this with as little boilerplate as possible. Of course, I could hand-write the set of files I want to see, but I'm thinking that somebody has to have automated this for a test suite somewhere. Any suggestions?

Answer 1

Automated in what way?

You could write a simple format to define a basic file structure using nested dictionaries:

## if you saved this as tree.py
## you could use it by doing:
# from tree import *
## then following the examples at the bottom of this file

import os, shutil, time

class Node:
    def __init__(self, name):
        self.name = name
        self.parent = None

    def setParent(self, parent):
        self.parent = parent

    def resolve(self):
        if self.parent: assert self.parent.exists()
        self.create()

    def exists(self):
        if os.path.exists(self.getPath()):
            return True
        else:
            return False

    def getPath(self): # you can nest things in symlinks
        if self.parent:
            return os.path.join(self.parent.getPath(), self.name)
        else:
            return self.name

    def create(self):
        raise NotImplemented, 'you must subclass node with your file type'

    def __repr__(self):
        return '<Node: %s>' % self.name

class Symlink(Node):
    def __init__(self, target, name):
        self.name = name
        self.target = target
        self.parent = None

    def create(self, basePath=None):
        os.symlink(self.target, self.getFilePath())

        assert 'symlink' in dir(os), "tried to create a symlink, but operating system doesn't support it"

class Folder(Node):
    def create(self):
        ## swap os.mkdir() for os.makedirs() if you want parent
        ## directories to be created if they don't already exist

        # os.makedirs(self.getPath()
        os.mkdir(self.getPath())

class File(Node):
    def __init__(self, name, contents=''):
        self.name = name
        self.contents = contents
        self.parent = None

    def create(self):
        f = open(self.getPath(), 'wb')
        f.write(self.contents)
        f.close()

def createAll(tree, parent=None):
    for node in tree:
        next = None
        if type(tree) == dict:
            if tree[node]:
                next = tree[node]

        if type(node) in (str, unicode):
            # coerce string to folder
            node = Folder(node)

        if parent:
            node.setParent(parent)

        node.resolve()
        if next: createAll(next, node)

if __name__ == '__main__':

    for name in ('src', 'src2', 'src3'):
        if os.path.exists(name):
            shutil.rmtree(name)

    time.sleep(0.1) # give it time to delete, took a second in one of my tests and denied access to creation

    empty = None # probably better than using None syntactically to indicate closed nodes of the tree

    test = {
        Folder('src'): {
            # if you *know* your folder won't contain any more levels, you can use a list instead of a dict
            # which means you don't need to specify None as the value for the folder key
            Folder('test'): [
                Symlink('..', 'recursive'),
                Symlink('..', 'still recursive'),
                Symlink('..', 'another recursion'),
            ],
            Folder('whee'): {
                Folder('nested'): {
                    Folder('nested'): {
                        Folder('done'): empty,
                        Symlink('recursive', '..'): empty,
                    }
                }
            }
        }
    }

    # the same structure expressed in a cleaner way, made possible by coercing strings to folder nodes:
    test2 = {
        'src2': {
            File('blank'): empty,
            File('whee.txt', 'this file is named whee.txt'): empty,
            # see above comment about using list as a container
            'test': [
                Symlink('..', 'recursive'),
                Symlink('..', 'still recursive'),
                Symlink('..', 'another recursion'),
            ],
            'whee': {
                'nested': {
                    'nested': {
                        'done': empty,
                        Symlink('..', 'recursive'): empty,
                    }
                }
            }
        }
    }

    test3 = {
        'src2': {
            File('blank'): empty,
            File('whee.txt', 'this file is named whee.txt'): empty,
            # see above comment about using list as a container
            'test': [
                File('file1.txt', 'poor substitute for a symlink'),
                File('file2.txt', 'I wish I could be a symlink'),
                File('file3.txt', "I'm hungry"),
            ],
            'nest': {
                'nested': {
                    'nested': {
                        'done': empty,
                        File('rawr.txt', 'I like pie.'): empty,
                    }
                }
            }
        }
    }

    if 'symlink' in dir(os): # these tests are no good if the OS doesn't support symlinks
        createAll(test)
        createAll(test2)

    createAll(test3)

You could also zip the required files and have the script unzip them when it runs.

Answer 2

I do this type of thing for testing Unix user creation and home directory copies. The Zip suggestion is a good one.

I personally keep two directory structures -- one is a source and one becomes the test structure. I just sync the source to the destination via shutil.copytree as part of the test setup.

That makes it easy to change the test structure on the fly (and not have to unzip).