ansaurus

Question

Answer 1

+1 A:

if Python is not a must

$ gawk '{b[$1]++;c[++d,$1]=$0}END{for(i in b){if(b[i]>=4){for(j=1;j<=d;j++){print c[j,i]}}}}' file

And yes, 70MB file is fine.

ghostdog74 2010-07-27 00:15:33

The first part appears to select all records which end with the digit 4 followed by optional spaces an/or tabs. The OP appears to want records for names which appear at least 4 times. Your solution **accidentally** produces the same output as the OP's required output. Add another record `tom pqrs 4`; your solution prints that, contrary to the OP's requirements.

John Machin 2010-07-27 00:59:39

what is this? unix?

I__ 2010-07-27 01:07:59

Unix and windows (GNU win32)

ghostdog74 2010-07-27 01:14:01

@john, yes, i think i misread the requirement.

ghostdog74 2010-07-27 01:14:53

Answer 2

+3 A:

Clearly, you cannot decide which rows are interesting until you've seen all rows (since the very last row might be the one turning some count from three to four and thereby making some previously seen rows interesting, for example;-). So, unless your CSV file is horribly huge, suck it all into memory, first, as a list...:

import csv

with open('thefile.csv', 'rb') as f:
  data = list(csv.reader(f))

then, do the counting -- Python 2.7 has a better way, but assuming you're still on 2.6 like most of us...:

import collections
counter = collections.defaultdict(int)
for row in data:
    counter[row[0]] += 1

and finally do the selection loop...:

for row in data:
    if counter[row[0]] >= 4:
        print row

Of course, this prints each interesting row as a roughly-hewed list (with square brackets and quotes around the items), but it will be easy to format it in any way you might prefer.

Alex Martelli 2010-07-27 02:04:32

its 70megs, is that ok?

I__ 2010-07-27 03:41:39

@I___, yep, it will be just fine. A typical modern machine has no less than a GB (a thousand megs), so it will have ample space to read in and process your seventy-megs file (even though the data typically takes up a bit more space in memory than it does on disk).

Alex Martelli 2010-07-27 04:26:15

@I__: try it; it's only 10 lines of code and it won't break your computer if it runs out of memory ... if it does, then you can write some code to implement the outline that I gave you in a comment to your non-question.

John Machin 2010-07-27 04:27:27

hi alex, this is not working for me, it's returning the wrong subset

I__ 2010-07-27 20:50:06

@I__, if your delimiter, as it appears, is a tab (rather than a comma, the default delimiter from which Comma Separated Variable files, aka CSV files, are named), of course you have to make you reader slightly different, e.g. `data = list(csv.reader(f, delimiter='\t'))` rather than the default way I built it. Apart from that obvious issue, the code, applied to your example input file, gives the expected output.

Alex Martelli 2010-07-28 00:04:15

ansaurus

tags:

views:

answers:

python csv: getting subset

related questions