ansaurus

Question

Parsing broken XML with lxml.etree.iterparse

Answer 1

A:

Edit your question, stating what happens (exact error message and traceback (copy/paste, don't type from memory)) to make you think that "bad unicode" is the problem.

Get chardet and feed it your MySQL dump. Tell us what it says.

Show us the first 200 to 300 bytes of your dump, using e.g. print repr(dump[:300])

Update You wrote """As you can see, chardet thinks it is an ascii file, but there is a "\x1e" right in the middle of this example which is making lxml raise an exception."""

I see no "bad unicode" here.

chardet is correct. What makes you think that "\x1e" is not ASCII? It is an ASCII character, a C0 control character named "RECORD SEPARATOR".

The error message says that you have an invalid character. That is also correct. The only control characters that are valid in XML are "\t", "\r" and "\n". MySQL should be grumbling about that and/or offering you a way of escaping it e.g. _x001e_ (yuk!)

Given the context, it looks like that character could be deleted with no loss. You may wish to fix your database or you may wish to remove suchlike characters from your dump (after checking that they are all vanishable) or you may wish to choose a less picky and less volumnious output format than XML.

Update 2 You presumably want to user iterparse() not because it's your end goal but because you want to save memory. If you used a format like CSV you wouldn't have a memory problem.

John Machin 2010-02-28 22:50:13

I've added that info to my question... Thanks for your help!

erikcw 2010-02-28 23:13:58

Thanks for the info! I'd still like to try to figure out a way to make lxml.etree.iterparse fix the broken XML just like lxml.etree.parse(filename, lxml.etree.XMLParser(recover=True)) is able to.

erikcw 2010-02-28 23:57:43

@John Machin: RE: "Update 2" -- Yes, I'm using iterparse() to save memory. It's true that in this case I could dump the MySQL db into CSV. That is a very practical solution. However, I'd still like to know how to handle this issue with lxml since processing large XML documents is something I'll likely have to do in the future. It's a very generic issue don't you think?

erikcw 2010-03-03 03:15:34

@ericw: How to handle this issue with lxml: (1) you change the lxml code to do what you want (2) you pay someone else (e.g. the lxml author) to do what you want (3) you sit on the beach until someone else changes lxml to do what you want -- all conditional on whether it's possible/sensible for lxml to be so changed. The principled solution for processing large XML documents is to make them valid XML before you attempt to parse them; that way the recovery is under your control.

John Machin 2010-03-03 03:43:22

Answer 2

+1 A:

I solved the problem by creating a class with a File like object interface. The class' read() method reads a line from the file and replaces any "bad characters" before returning the line to iterparse.

#psudo code

class myFile(object):
    def __init__(self, filename):
        self.f = open(filename)

    def read(self, size=None):
        return self.f.next().replace('\x1e', '').replace('some other bad character...' ,'')


#iterparse
context = lxml.etree.iterparse(myFile('bigfile.xml', tag='RECORD')

I had to edit the myFile class a few times adding some more replace() calls for a few other characters that were making lxml choke. I think lxml's SAX parsing would have worked as well (seems to support the recover option), but this solution worked like a charm!

erikcw 2010-03-05 20:33:35

ansaurus

tags:

views:

answers:

Parsing broken XML with lxml.etree.iterparse

related questions