Appending lists from files to a single list in Python

Question

I'm trying to write a function that reads files from a "deferred" directory which contains files that contain lists. Here's what the files in the deferred folder contain:

'173378981', '45000', '343434', '3453453', '34534545', '3452342', '234234', '42063008', 'Exempted', '10000'
'1000014833', '0', '0', '0', '0', '0', '0', '0', 'Exempted', '0'
'1000009598', '0', '0', '0', '0', '0', '0', '0', 'Exempted', '0'
'279483421', '0', '0', '0', '0', '0', '0', '0', 'Exempted', '0'
'1000009600', '0', '0', '0', '0', '0', '0', '0', 'Exempted', '0'
'389453080', '0', '0', '0', '0', '0', '0', '0', 'Exempted', '0'
'1000009602', '0', '0', '0', '0', '0', '0', '0', 'Exempted', '0'

The function used to write the file(s):

def storeDeferredRecords(records):
 """docstring for createFile"""
 now = datetime.datetime.now()
 filename = deferredDir + '/' + now.strftime("%Y%m%d-%H%M%S")
 f = open(filename, 'w')
 newlist = map(lambda(x): str(x)[1:-1], records)
 for item in newlist:
  f.write("%s\n" % item)
 f.close

I need help with the function used to read the file. I was only able to write this:

def getDeferredRecords():
     """docstring for getDeferredRecords"""
     infiles = [infile for infile in glob.glob(deferredDir + '/*')]
                <code to read the contents of each file here>

Can someone help me out? I need to read the lines and insert them into a list. This list will then be merged with records from separate CSV file.

Answer 1

See the csv module:

BigList = []
for filename in glob.glob(deferredDir + '/*'):
    PartList = csv.reader(open(filename))
    BigList.extend(PartList)

Is that what you had in mind?

Answer 2

The Python cvs module is likely a good answer:
http://docs.python.org/library/csv.html

Question:

glob.glob() returns an iterable already, so I do not see the point here...

[infile for infile in glob.glob(deferredDir + '/*')]

Rather:

BigList = []
for filename in glob.glob(deferredDir + '/*'):
    #CVS read code here
    #add to BigList

Food for thought.

Answer 3

First, the last line in the store function needs to be like this f.close()

Your store function saves the values in a newline-separated manner. To read all the files, should be enough:

def getDeferredRecords():
    """docstring for getDeferredRecords"""
    return dict((infile, list(iter(file(infile)))) 
                     for infile in glob.glob(deferredDir + '/*'))

Explanation: a file is iterable, so you can do for line in file: print line for example. With list(iter(file)) you have the lines of a file in a list. dict((a, b) for a, b in foo) returns a dictionary with {a: b} pairs. The return value of the function is a dictionary with the format {filename: list_of_lines_in_file}. Keep in mind that the list elements are strings with a trailing newline.

Answer 4

Incorporating ideas from Tim Pietzcker, here are the re-written functions:

def storeDeferredRecords(records):
 """docstring for createFile"""
 now = datetime.datetime.now()
 filename = deferredDir + '/' + now.strftime("%Y%m%d-%H%M%S")
 f = csv.writer(open(filename, 'w'), delimiter=',')
 f.writerows(records)

def getDeferredRecords():
 """docstring for getDeferredRecords"""
 for filename in glob.glob(deferredDir + '/*'):
  def_records = csv.reader(open(filename,'r'))
  records.extend(def_records)

I used csv.writer instead of using the previous code block:

f = open(filename, 'w')
newlist = map(lambda(x): str(x)[1:-1], records)
for item in newlist:
        f.write("%s\n" % item)
f.close

Thanks to all those who replied!