Python: Reading part of a text file

Question

HI all

I'm new to python and programming. I need to read in chunks of a large text file, format looks like the following:

<word id="8" form="hibernis" lemma="hibernus1" postag="n-p---nb-" head-"7" relation="ADV"/>

I need the form, lemma and postag information. e.g. for above I need hibernis, hibernus1 and n-p---nb-.

How do I tell python to read until it reaches form, to read forward until it reaches the quote mark " and then read the information between the quote marks "hibernis"? Really struggling with this.

My attempts so far have been to remove the punctuation, split the sentence and then pull the info I need from a list. Having trouble getting python to iterate over whole file though, I can only get this working for 1 line. My code is below:

f=open('blank.txt','r')
quotes=f.read()
noquotes=quotes.replace('"','')
f.close()

rf=open('blank.txt','w')
rf.write(noquotes)
rf.close()   

f=open('blank.txt','r')
finished = False
postag=[]
while not finished:
   line=f.readline()
   words=line.split()
   postag.append(words[4])
   postag.append(words[6])
   postag.append(words[8])              
   finished=True

Would appreciate any feedback/criticisms

thanks

Answer 1

I'd suggest using the regular expression module: re

Something along these lines perhaps?

#!/usr/bin/python
import re

if __name__ == '__main__':
    data = open('x').read()
    RE = re.compile('.*form="(.*)" lemma="(.*)" postag="(.*?)"', re.M)
    matches = RE.findall(data)
    for m in matches:
        print m

This does assume that the <word ...> lines are each on a single line and that each part is in that exact order, and that you don't need to deal with full xml parsing.

Answer 2

Is your file proper XML? If so, try a SAX parser:

import xml.sax
class Handler (xml.sax.ContentHandler):
   def startElement (self, tag, attrs):
       if tag == 'word':
           print 'form=', attrs['form']
           print 'lemma=',attrs['lemma']
           print 'postag=',attrs['postag']

ch = Handler ()
f = open ('myfile')
xml.sax.parse (f, ch)

(this is rough .. it may not be entirely correct).

Answer 3

In addition to the usual RegEx answer, since this appears to be a form of XML, you might try something like BeautifulSoup ( http://www.crummy.com/software/BeautifulSoup/ )

It's very easy to use, and find tags/attributes in things like HTML/XML, even if they're not "well formed". Might be worth a look.

Answer 4

Parsing xml by hand is usually the wrong thing. For one thing, your code will break if there's an escaped quote in any of the attributes. Getting the attributes from an xml parser is probably cleaner and less error-prone.

An approach like this can also run into problems parsing the entire file if you have lines that don't match the format. You can deal with this either by creating a parseline method (something like

def parse (line):
      try: 
          return parsed values here
        except:

You can also simplify this with filter and map functions:

lines = filter( lambda line: parseable(line), f.readlines())
values = map (parse, lines)

Answer 5

Just to highlight your problem:

finished = False
counter = 0
while not finished:
   counter += 1
   finished=True
print counter

Answer 6

With regular expressions, this is the gist (you can do the file.readline() part):

import re
line = '<word id="8" form="hibernis" lemma="hibernus1" postag="n-p---nb-" head-"7" relation="ADV"/>'
r = re.compile( 'form="([^"]*)".*lemma="([^"]*)".*postag="([^"]*)"' )
match = r.search( line )
print match.groups()

>>> 
('hibernis', 'hibernus1', 'n-p---nb-')
>>>

Answer 7

First, don't spend a lot of time rewriting your file. It's generally a waste of time. The processing to clean up and parse the tags is so fast, that you'll be perfectly happy working from the source file all the time.

source= open( "blank.txt", "r" )
for line in source:
    # line has a tag-line structure
    # <word id="8" form="hibernis" lemma="hibernus1" postag="n-p---nb-" head-"7" relation="ADV"/>
    # Assumption -- no spaces in the quoted strings.
    parts = line.split()
    # parts is [ '<word', 'id="8"', 'form="hibernis"', ... ]
    assert parts[0] == "<word"
    nameValueList = [ part.partition('=') for part in parts[1:] ]
    # nameValueList is [ ('id','=','"8"'), ('form','=','"hibernis"'), ... ]
    attrs = dict( (n,eval(v)) for n, _, v in nameValueList )
    # attrs is { 'id':'8', 'form':'hibernis', ... }
    print attrs['form'], attrs['lemma'], attrs['posttag']

Answer 8

wow, you guys are fast :) If you want all attributes of a list (and the ordering is known), then you can use something like this:

import re
print re.findall('"(.+?)"',INPUT)

INPUT is a line like:

<word id="8" form="hibernis" lemma="hibernus1" postag="n-p---nb-" head="7" relation="ADV"/>

and the printed list is:

['8', 'hibernis', 'hibernus1', 'n-p---nb-', '7', 'ADV']

Answer 9

If it's XML, use ElementTree to parse it:

from xml.etree import ElementTree

line = '<word id="8" form="hibernis" lemma="hibernus1" postag="n-p---nb-" head="7" relation="ADV"/>'

element = ElementTree.fromstring(line)

For each XML element you can easily extract the name and all the attributes:

>>> element.tag
'word'
>>> element.attrib
{'head': '7', 'form': 'hibernis', 'postag': 'n-p---nb-', 'lemma': 'hibernus1', 'relation': 'ADV', 'id': '8'}

So if you have a document with a bunch of word XML elements, something like this will extract the information you want from each one:

from xml.etree import ElementTree

XML = '''
<words>
    <word id="8" form="hibernis" lemma="hibernus1" postag="n-p---nb-" head="7" relation="ADV"/>
</words>'''

root = ElementTree.fromstring(XML)

for element in root.findall('word'):
    form = element.attrib['form']
    lemma = element.attrib['lemma']
    postag = element.attrib['postag']

    print form, lemma, postag

Use parse() instead of fromstring() if you only have a filename.