How can I process this text file and parse what I need?

Question

I'm trying to parse ouput from the Python doctest module and store it in an HTML file.

I've got output similar to this:

**********************************************************************
File "example.py", line 16, in __main__.factorial
Failed example:
    [factorial(n) for n in range(6)]
Expected:
    [0, 1, 2, 6, 24, 120]
Got:
    [1, 1, 2, 6, 24, 120]
**********************************************************************
File "example.py", line 20, in __main__.factorial
Failed example:
    factorial(30)
Expected:
    25252859812191058636308480000000L
Got:
    265252859812191058636308480000000L
**********************************************************************
1 items had failures:
   2 of   8 in __main__.factorial
***Test Failed*** 2 failures.

Each failure is preceded by a line of asterisks, which delimit each test failure from each other.

What I'd like to do is strip out the filename and method that failed, as well as the expected and actual results. Then I'd like to create an HTML document using this (or store it in a text file and then do a second round of parsing).

How can I do this using just Python or some combination of UNIX shell utilities?

EDIT: I formulated the following shell script which matches each block how I'd like,but I'm unsure how to redirect each sed match to its own file.

python example.py | sed -n '/.*/,/^\**$/p' > `mktemp error.XXX`

Answer 1

You can write a Python program to pick this apart, but maybe a better thing to do would be to look into modifying doctest to output the report you want in the first place. From the docs for doctest.DocTestRunner:

                                  ... the display output
can be also customized by subclassing DocTestRunner, and
overriding the methods `report_start`, `report_success`,
`report_unexpected_exception`, and `report_failure`.

Answer 2

This is a quick and dirty script that parses the output into tuples with the relevant information:

import sys
import re

stars_re = re.compile('^[*]+$', re.MULTILINE)
file_line_re = re.compile(r'^File "(.*?)", line (\d*), in (.*)$')

doctest_output = sys.stdin.read()
chunks = stars_re.split(doctest_output)[1:-1]

for chunk in chunks:
    chunk_lines = chunk.strip().splitlines()
    m = file_line_re.match(chunk_lines[0])

    file, line, module = m.groups()
    failed_example = chunk_lines[2].strip()
    expected = chunk_lines[4].strip()
        got = chunk_lines[6].strip()

    print (file, line, module, failed_example, expected, got)

Answer 3

I wrote a quick parser in pyparsing to do it.

from pyparsing import *

str = """
**********************************************************************
File "example.py", line 16, in __main__.factorial
Failed example:
    [factorial(n) for n in range(6)]
Expected:
    [0, 1, 2, 6, 24, 120]
Got:
    [1, 1, 2, 6, 24, 120]
**********************************************************************
File "example.py", line 20, in __main__.factorial
Failed example:
    factorial(30)
Expected:
    25252859812191058636308480000000L
Got:
    265252859812191058636308480000000L
**********************************************************************
"""

quote = Literal('"').suppress()
comma = Literal(',').suppress()
in_ = Keyword('in').suppress()
block = OneOrMore("**").suppress() + \
        Keyword("File").suppress() + \
        quote + Word(alphanums + ".") + quote + \
        comma + Keyword("line").suppress() + Word(nums) + comma + \
        in_ + Word(alphanums + "._") + \
        LineStart() + restOfLine.suppress() + \
        LineStart() + restOfLine + \
        LineStart() + restOfLine.suppress() + \
        LineStart() + restOfLine + \
        LineStart() + restOfLine.suppress() + \
        LineStart() + restOfLine  

all = OneOrMore(Group(block))

result = all.parseString(str)

for section in result:
    print section

gives

['example.py', '16', '__main__.factorial', '    [factorial(n) for n in range(6)]', '    [0, 1, 2, 6, 24, 120]', '    [1, 1, 2, 6, 24, 120]']
['example.py', '20', '__main__.factorial', '    factorial(30)', '    25252859812191058636308480000000L', '    265252859812191058636308480000000L']

Answer 4

This is probably one of the least elegant python scripts I've ever written, but it should have the framework to do what you want without resorting to UNIX utilities and separate scripts to create the html. It's untested, but it should only need minor tweaking to work.

import os
import sys

#create a list of all files in directory
dirList = os.listdir('')

#Ignore anything that isn't a .txt file.
#
#Read in text, then split it into a list.
for thisFile in dirList:
    if thisFile.endswith(".txt"):
        infile = open(thisFile,'r')

        rawText = infile.read()

        yourList = rawText.split('\n')

        #Strings
        compiledText = ''
        htmlText = ''

        for i in yourList:

            #clunky way of seeing whether or not current line  
            #should be included in compiledText

            if i.startswith("*****"):
                compiledText += "\n\n--- New Report ---\n"

            if i.startswith("File"):
                compiledText += i + '\n'

            if i.startswith("Fail"):
                compiledText += i + '\n'

            if i.startswith("Expe"):
                compiledText += i + '\n'

            if i.startswith("Got"):
                compiledText += i + '\n'

            if i.startswith(" "):
                compiledText += i + '\n'


    #insert your HTML template below

    htmlText = '<html>...\n <body> \n '+htmlText+'</body>... </html>'


    #write out to file
    outfile = open('processed/'+thisFile+'.html','w')
    outfile.write(htmlText)
    outfile.close()