[Python] Best strategy for dealing with incomplete lines of data from a file.

Question

I use the following block of code to read lines out of a file 'f' into a nested list:

for data in f:
     clean_data = data.rstrip()
     data = clean_data.split('\t') 
     t += [data[0]]
     strmat += [data[1:]]

Sometimes, however, the data is incomplete and a row may look like this:

['955.159', '62.8168', '', '', '', '', '', '', '', '', '', '', '', '', '', '29', '30', '0', '0']

It puts a spanner in the works because I would like Python to implicitly cast my list as floats but the empty fields '' cause it to be cast as an array of strings (dtype: s12).

I could start a second 'if' statement and convert all empty fields into NULL (since 0 is wrong in this instance) but I was unsure whether this was best.

1. Is this the best strategy of dealing with incomplete data?
2. Should I edit the stream or do it post-hoc?

Answer 1

The way how you should deal with incomplete values depends on the context of your application (which you haven't mentioned yet).

For example, you can simply ignore missing values

>>> l = ['955.159', '62.8168', '', '', '', '', '', '', '', '', '', '', '', '', '', '29', '30', '0', '0']
>>> filter(bool, l) # remove empty values
['955.159', '62.8168', '29', '30', '0', '0']
>>> map(float, filter(bool, l)) # remove empty values and convert the rest to floats
[955.15899999999999, 62.816800000000001, 29.0, 30.0, 0.0, 0.0]

Or alternatively, you might want to replace them with NULL as you mentioned:

>>> map(lambda x: x or 'NULL', l)
['955.159', '62.8168', 'NULL', 'NULL', 'NULL', 'NULL', 'NULL', 'NULL', 'NULL', 'NULL', 'NULL', 'NULL', 'NULL', 'NULL', 'NULL', '29', '30', '0', '0']

As you can see, there are many different strategies of dealing with incomplete data. Anyway, the example snippets here might help you to choose the right one for your task. And as you can see, I prefer the functional programming like build-ins for doing stuff like this, because it's often the shortest and easiest way to do it (and I don't think there will be any noticeable differences in the execution time).