Reading and Grouping a List of Data in Python

Question

I have been struggling with managing some data. I have data that I have turned into a list of lists each basic sublist has a structure like the following

<1x>begins
<2x>value-1
<3x>value-2
<4x>value-3
 some indeterminate number of other values
<1y>next observation begins
<2y>value-1
<3y>value-2
<4y>value-3
 some indeterminate number of other values

this continues for an indeterminate number of times in each sublist

EDIT I need to get all the occurrences of <2,<3 & <4 separated out and grouped together I am creating a new list of lists [[<2x>value-1,<3x>value-2, <4x>value-3], [<2y>value-1, <3y>value-2, <4y>value-3]]

EDIT all of the lines that follow <4x> and <4y> (and for that matter <4anyalpha> have the same type of coding and I don't know a-priori how high the numbers can go-just think of these as sgml tags that are not closed I used numbers because my fingers were hurting from all the coding I have been doing today.

The solution I have come up with finally is not very pretty

 listINeed=[]
 for sublist in biglist:
    for line in sublist:
        if '<2' in line:
            var2=line
        if '<3' in line:
            var3=line
        if '<4' in line:
            var4=line
            templist=[]
            templist.append(var2)
            templist.append(var3)
            templist.append(var4)
            listIneed.append(templist)
            templist=[]
            var4=var2=var3=''

I have looked at ways to try to clean this up but have not been successful. This works fine I just saw this as another opportunity to learn more about python because I would think that this should be processable by a one line function.

Answer 1

You're off to a good start by noticing that your original solution may work but lacks elegance.

You should parse the string in a loop, creating a new variable for each line. Here's some sample code:

import re

s = """<1x>begins
<2x>value-1
<3x>value-2
<4x>value-3
 some indeterminate number of other values
<1y>next observation begins
<2y>value-1
<3y>value-2
<4y>value-3"""
firstMatch = re.compile('^\<1x')
numMatch = re.compile('^\<(\d+)')
listIneed = []
templist = None
for line in s.split():
        if firstMatch.match(line):
                if templist is not None: 
                        listIneed.append(templist)
                templist = [line]
        elif numMatch.match(line):
            #print 'The matching number is %s' % numMatch.match(line).groups(1)
            templist.append(line)
if templist is not None: listIneed.append(templist)

print listIneed

Answer 2

If you want to pick out the second, third, and fourth elements of each sublist, this should work:

listINeed = [sublist[1:4] for sublist in biglist]

Answer 3

itertools.groupby() can get you by.

itertools.groupby(biglist, operator.itemgetter(2))

Answer 4

If I've understood your question correctly:

import re
def getlines(ori):
    matches = re.finditer(r'(<([1-4])[a-zA-Z]>.*)', ori)
    mainlist = []
    sublist = []
    for sr in matches:
        if int(sr.groups()[1]) == 1:
            if sublist != []:
                mainlist.append(sublist)
            sublist = []
        else:
            sublist.append(sr.groups()[0])
    else:
        mainlist.append(sublist)
    return mainlist

...would do the job for you, if you felt like using regular expressions.

The version below would break all of the data down into sublists (not just the first four in each grouping) which might be more useful depending what else you need to do to the data. Use David's listINeed = [sublist[1:4] for sublist in biglist] to get the first four results from each list for the specific task above.

import re
def getlines(ori):
    matches = re.finditer(r'(<(\d*)[a-zA-Z]>.*)', ori)
    mainlist = []
    sublist = []
    for sr in matches:
        if int(sr.groups()[1]) == 1:
            print "1 found!"
            if sublist != []:
                mainlist.append(sublist)
            sublist = []
        else:
            sublist.append(sr.groups()[0])
    else:
        mainlist.append(sublist)
    return mainlist