trouble writing to file in "for line in" iterator in Python script

Question

This seems to be an embarrassingly simple question, but, after a day of reading over How-To's and manuals, it must be asked.

I'm writing many lines to a few files using a few nested loops, inserting some static strings and copying lines over from other files over and over again. The output appears to be a single copy of the static strings and all of the lines I want to copy from other files, instead of multiple copies of the strings combined with the copied lines.

I made a test script that I thought would mimic the behaviour, but it behaves perfectly:

for i in range(10):
    f = open('output.txt','w')

    f.write( "---------------------------\n" )

    FILE1 = open('test1.txt','r')
    for line in FILE1:
        f.write( "... compliments of loop #1 ...\n" )
        f.write( line )
    FILE1.close()

    f.write( "\n##########################\n" )

    FILE2 = open('test2.txt','r')
    for line in FILE2:
        f.write( "... compliments of loop #1 ...\n" )
        f.write( line )
    FILE2.close()

    f.write( "\n++++++++++++++++++++++++++\n" )

The output is as expected: the static strings interleaved with copied strings. My real, ugly script, however, doesn't do this. I don't want to paste the whole thing in here, but will include as much as I think is relevant (and I'll probably get that wrong, too, 'cause I don't know what's going on). It references an array of objects - I won't include the class, as it seems to behave.

for i in range(10):
    print "script-%s-%i-%s%s" % (cities[i].user,i,cities[i].name,cities[i].coords)

    f = open("script-%s-%i-%s%s" % (cities[i].user,i,cities[i].name,cities[i].coords),'w')

    f.write( "//\n" )
    f.write( "//\n// %s - %s            %s\n" % (cities[i].user,cities[i].name,cities[i].coords) )
    f.write( "//\n" )
    f.write( "//\n" )

    npc10 = open("script-%s-npc10-%i.txt" % (cities[i].user,i),'r')
    for line in npc10:
        f.write( "ifgosub ( m_city.AnyIdleHero(%s) == false ) wait_for_big_hero\n" % (cities[i].hero) )
        f.write( "ifgosub ( m_city.IsArmyReady(a:%i,s:%i,w:%i,wo:%i) == false ) gosub check_npc10\n" % (lvl10.arch,lvl10.scout,lvl10.warr,lvl10.work) )
        f.write( "ifgosub ( m_city.IsArmyReady(a:%i,s:%i,w:%i,wo:%i) == false ) farm_npc5\n" % (lvl10.arch,lvl10.scout,lvl10.warr,lvl10.work) )
        f.write( "ifgosub ( m_city.AnyIdleHero(%s) == false ) wait_for_big_hero\n" % (cities[i].hero) )
        f.write( line )
    npc10.close()

    f.write( "\n//\n" )
    f.write( "label farm_npc5\n" )

    npc5 = open("script-%s-npc5-%i.txt" % (cities[i].user,i),'r')
    for line in npc5:
        f.write( "sleep 5\n" )
        f.write( line )
    npc5.close()

    f.write( "\n//\n" )
    # ... 107 lines of static f.write's

    f.close()

Sample of one of the input files ( script-%s-npc10-%i.txt" % (cities[i].user,i) ) - they are all very similar:

attack 456,357 Alfred a:9215,t:185,wo:200,w:2000,s:200 //Distance: 1 Mission time: 8m 52s
attack 159,357 Alfred a:9215,t:185,wo:200,w:2000,s:200 //Distance: 1 Mission time: 8m 52s
attack 159,215 Alfred a:9215,t:185,wo:200,w:2000,s:200 //Distance: 1 Mission time: 12m 34s

Sample of one of the output files
( "script-%s-%i-%s%s" % (cities[i].user,i,cities[i].name,cities[i].coords) ):

//
// user1 - cityname1 (456,456)
//
//
ifgosub ( m_city.AnyIdleHero(Alfonso) == false ) wait_for_big_hero
ifgosub ( m_city.IsArmyReady(a:92150,s:2000,w:2000,wo:2000) == false ) gosub check_npc10
ifgosub ( m_city.IsArmyReady(a:92150,s:2000,w:2000,wo:2000) == false ) farm_npc5
ifgosub ( m_city.AnyIdleHero(Alfonso) == false ) wait_for_big_hero
attack 456,357 Alfred a:9215,t:185,wo:200,w:2000,s:200 //Distance: 1 Mission time: 8m 52s
attack 159,357 Alfred a:9215,t:185,wo:200,w:2000,s:200 //Distance: 1 Mission time: 8m 52s
attack 159,215 Alfred a:9215,t:185,wo:200,w:2000,s:200 //Distance: 1 Mission time: 12m 34s
//
label farm_npc5
sleep 5
attack 354,159 Alfred b:50,t:40 //Distance: 1 Mission time: 13m 20s
attack 789,654 Alfred b:50,t:40 //Distance: 2 Mission time: 26m 40s
attack 125,456 Alfred b:50,t:40 //Distance: 2 Mission time: 29m 48s
//
[...]

What's the difference? Why don't the static strings get repeated, while the copied lines do?

ANSWER: I wrote the source files in OSX TextEdit, so their newline character was '\r', as pointed out by gnibbler. Following his lead led me to 6 PEP 278: Universal Newline Support. Using the file mode 'rU' squared everything.

Thanks folks!

Answer 1

It appears that the for loop is only executing once.

I'm not sure why it would do that - perhaps there is a problem with the line endings so the whole file is being read in as a single line.

For example, if the lines of the input script file has \r line ending and python is expecting \n line endings.

Try using 'rU' as the mode instead of 'r'

Answer 2

my guess is that when you iterate over the lines in the input files (in the inner loops), the whole file is read as a single line, therefore there is only a single iteration.

Not sure, but a possible cause might be mixing Windows/Linux files and python interpreters ("\r\n" vs "\n" End-Of-Line encodings), but someone could correct me here.

General tip when mixing Linux / Windows text files: convert your files with dos2unix or unix2dos command-line tools.

Answer 3

Maybe you mixed up tabs and spaces in your file and therefore the indentation isn't like it appears to be. Try python's -t flag to check for inconsistent usage of tabs:

python -t script.py

Answer 4

Using the sample you gave as an input file and a made up cities+lvl10 variable, the script works as expected: For each of the "attack" lines in the input, it prints multiple "ifgosub" lines first and then the "attack" line.

The most likely cause of your problem is that iteration over a file doesn't give you the lines separately, but all lines at once. In my experiments that happened if I chose to use CR (\r) as line end characters (that convention was used by MacOS up to 9.x). Maybe some stange character encodings for the input files may cause a similar problem.

To test this assumption, just try to iterate over one of your files, maybe like this:

i = 0
npc10 = open("script-%s-npc10-%i.txt" % (cities[i].user,i),'r')
lines = [line for line in npc10]
npc10.close()

after that you can check len(lines) if it is larger than 1.

BTW, since Python 2.6 (or 2.5 + from __future__ import with_statement) you can use context managers to ensure that files are closed:

with open("script-%s-npc10-%i.txt" % (cities[i].user,i),'r') as npc10:
    for line in npc10:
        #....
    # the file gets closed automatically after the with-statement