Python DictReader - Skipping rows with missing columns?

Question

heya,

I have a Excel .CSV file I'm attempting to read in with DictReader.

All seems to be well, except it seems to omit rows, specifically those with missing columns.

Our input looks like:

mail,givenName,sn,lorem,ipsum,dolor,telephoneNumber
[email protected],ian,bay,3424,8403,2535,+65(2)34523534545
[email protected],mike,gibson,3424,8403,2535,+65(2)34523534545
[email protected],ross,martin,,,,+65(2)34523534545
[email protected],david,connor,,,,+65(2)34523534545
[email protected],chris,call,3424,8403,2535,+65(2)34523534545

So some of the rows have missing lorem/ipsum/dolor columns, and it's just a string of commas for those.

We're reading it in with:

def read_gd_dump(input_file="blah 20100423.csv"):
    gd_extract = csv.DictReader(open('blah 20100423.csv'), restval='missing', dialect='excel')
    return dict([(row['something'], row) for row in gd_extract])

And I checked that "something" (the key for our dict) isn't one of the missing columns, I had originally suspected it might be that. It's one of the columns after that.

However, DictReader seems to completely skip over the rows. I tried setting restval to something, didn't seem to make any difference. I can't seem to find anything in Python's CSV docs (http://docs.python.org/library/csv.html) that would explain this behaviour, but I may have misread something.

Any ideas?

Thanks, Victor

EDIT:

It turns out I was pretty stupid - I was indexing the dict on a column ("something") that was empty for some rows in the input CSV file, a fact I didn't even notice in the mass of data (basically there were two ID columns, and I was using the wrong one).

Hence, Alex was right, there were duplicates in "something", and hence each subsequent entry with an empty "something" was overwriting the previous one.

I've awarded the answer to Alex Martelli.

Answer 1

Can't reproduce your problem -- when I save that data and then assign list(gd_extract), I see:

[{'telephoneNumber': '+65(2)34523534545', 'ipsum': '8403', 'sn': 'bay', 'dolor': '2535', 'mail': '[email protected]', 'givenName': 'ian', 'lorem': '3424'}, {'telephoneNumber': '+65(2)34523534545', 'ipsum': '8403', 'sn': 'gibson', 'dolor': '2535', 'mail': '[email protected]', 'givenName': 'mike', 'lorem': '3424'}, {'telephoneNumber': '+65(2)34523534545', 'ipsum': '', 'sn': 'martin', 'dolor': '', 'mail': '[email protected]', 'givenName': 'ross', 'lorem': ''}, {'telephoneNumber': '+65(2)34523534545', 'ipsum': '', 'sn': 'connor', 'dolor': '', 'mail': '[email protected]', 'givenName': 'david', 'lorem': ''}, {'telephoneNumber': '+65(2)34523534545', 'ipsum': '8403', 'sn': 'call', 'dolor': '2535', 'mail': '[email protected]', 'givenName': 'chris', 'lorem': '3424'}]

five dicts, including those with missing ipsum etc. I fear that in your laudable attempt at simplifying the problem you've simplified it excessively, so that your bug has gone away.

If you have duplicates in column something (can't check, since you don't have that column in your sample data) that would of course explain the "apparently missing" rows -- they're not missing from the csv reader's returned stream, they get "overwritten" in the dict you're returning. Could that be the issue?

Answer 2

This may be nothing to do with your problem, and Alex's analysis is quite plausible given the lack of information, but you should ALWAYS open a csv file with "rb" or "wb" mode (assuming Python 2.X). If you don't, you run the risk of various mysterious happenings. A csv file is not a text file, it's a BINARY file.

In any case, please edit your question to show:
(1) (a) a sample file (b) a script (c) output -- which together demonstrate the alleged problem
(2) what version of Python you are running
(3) what OS

Update: For Python 3.X, do as the blessed manual says: """If csvfile is a file object, it should be opened with newline=''. Although this advice is included only with csv.reader, it applies equally to csv.writer, csv.DictReader, and csv.DictWriter.