tags:

views:

188

answers:

3

Is there a better way to read lines two at a time from a file in python than:

with open(fn) as f:
    for line in f:
        try:
            line2 = f.next()
        except StopIteration:
            line2 = ''
        print line, line2 # or something more interesting

I'm in 2.5.4. Anything different in newer versions?

EDIT: a deleted answer noted: in py3k you'd need to do next(f) instead of f.next(). Not to mention the print change

+6  A: 
import itertools

with open(fn) as f:
  for line, line2 in itertools.izip_longest(f, f, fillvalue=''):
    print line, line2

Alas, izip_longest requires Python 2.6 or better; 2.5 only has izip, which would truncate the last line if f has an odd number of lines. It's quite easy to supply the equivalent functionality as a generator, of course.

Here's a more general "N at a time" iterator-wrapper:

def natatime(itr, fillvalue=None, n=2):
  return itertools.izip_longest(*(iter(itr),)*n, fillvalue=fillvalue)

itertools is generally the best way to go, but, if you insisted on implementing it by yourself, then:

def natatime_no_itertools(itr, fillvalue=None, n=2):
  x = iter(itr)
  for item in x:
    yield (item,) + tuple(next(x, fillvalue) for _ in xrange(n-1))

In 2.5, I think the best approach is actually not a generator, but another itertools-based solution:

def natatime_25(itr, fillvalue=None, n=2):
  x = itertools.chain(iter(itr), (fillvalue,) * (n-1))
  return itertools.izip(*(x,)*n)

(since 2.5 doesn't have the built-in next, as well as missing izip_longest).

Alex Martelli
+1: suggestion for generator in Py2.5
Jarret Hardie
never paid attention to the `default` parameter of `next`, thanks for pointing that out.
SilentGhost
try/except seems simple and clear (and wouldn't be need if we could guarantee an even number of lines). why would you prefer writing a generator here? Wouldn't the generator require a try/except?
foosion
Unless I remove that last argument, (the empty string) I get ValueError: too many values to unpack. What are you trying to accomplish with that?
recursive
@recursive, I had forgotten to give the `fillvalue=` explicitly (what I'm trying to accomplish is to use your specified fill value, an empty string, instead of the default None!-), edited and fixed now.
Alex Martelli
@foosion, if you can guarantee an even number of lines then itertools.izip (available in 2.5 and earlier) is perfectly fine. Generators are the "one obvious way" to encapsulate complex looping logic in Python, when you can't get itertools to perform it for you: if you need "two items at a time" from an iterator in one spot of your software system, you'll likely need it in other spots, and repetition of what could be properly encapsulated is evil (that's why Python offers generators: to let you encapsulate looping logic!-).
Alex Martelli
@foosion again, yep, try/except is ok, BUT -- "flat is better than nested", and with itertools (or other approaches I've shown) you avoid the nesting that try/except requires. Also, if you can avoid raising and catching exceptions, you'll generally get much faster code (esp. with speed-demon `itertools`) -- so why go out of your way to catch an exception when that can easily be avoided by using the standard Python library? (In the end I thought of a good itertools non-generator for 2.5 too, but generators can still have reasonably good performance characteristics when they're needed).
Alex Martelli
@Alex - I agree completely if izip_longest is available or if we could guarantee an even number of lines. The only issue then is 2.5 with the possibility of an odd number of lines. natatime_25 avoids try/except at the possible expense of clarity. I can understand my code (and your izip code) at a glance, but it takes me more than a glance to understand natatime_25, which makes it a trade-off between "Flat is better than nested" and "Readability counts" (and "Simple is better than complex" and a few other Zen lines).
foosion
@foosion, if you find code using itertools complex or unreadable, I beseech you to go on an itertools quest: the code is really linear and direct, and itertools' returns are just **huge** in terms of performance, scalability, and productivity through thinking at an higher level of abstractions. Neither `chain` nor `izip` are at all complex (indeed if you like `izip_longest`, `izip` is even simpler than that!-), and `chain` just does two iterables one after the other... what could be simpler?-).
Alex Martelli
@Alex, it's more having to go through the code step by step. (1) natatime gets passed f, which iterates over an open file, (2) first argument to chain iterates over f, (3) second argument to chain ..., all of which takes more thought than (a) read one line, (b) read another line if it's there and (c) repeat until done. In any event, thanks!
foosion
@foosion, I see your point, but it really boils down to familiarity, I think -- you're familiar with looping on a file, "know" that a file is its own iterator so calling next on it "advances" it (so the `for` will not read that line again), know about StopIteration, etc; and you're not as familiar with itertools, so you find it less direct... but it's not intrinsically so, speaking as one equally familiar with both sets of concepts and with teaching them to experienced programmers that are not familiar with either. Still, whatever floats your boat, of course!-)
Alex Martelli
@Alex - I agree completely. Things move from "what??" to "oh that makes sense" to "of course" as one gets more experienced. Readability is good, but expanding the set of what is readable is better.
foosion
+1  A: 

for small to medium sized files,

>>> data=open("file").readlines()
>>> for num,line in enumerate(data[::2]):
...  print ''.join(data[num:num+2])
ghostdog74
+2  A: 

You could possibly make it more clear with a generator:

def read2(f):
    for line in f:
        try:
            line2 = f.next()
        except StopIteration:
            line2 = ''

        yield line, line2

with open(fn) as f:
    for line1, line2 in read2(f):
        print line1
        print line2
Cixate