Python - How can I open a file and specify the offset in bytes?

Question

I'm writing a program that will parse an Apache log file periodically to log it's visitors, bandwidth usage, etc..

The problem is, I don't want to open the log and parse data I've already parsed. For example:

line1
line2
line3

If I parse that file, I'll save all the lines then save that offset. That way, when I parse it again, I get:

line1
line2
line3 - The log will open from this point
line4
line5

Second time round, I'll get line4 and line5. Hopefully this makes sense...

What I need to know is, how do I accomplish this? Python has the seek() function to specify the offset... So do I just get the filesize of the log (in bytes) after parsing it then use that as the offset (in seek()) the second time I log it?

I can't seem to think of a way to code this >.<

Answer 1

If your logfiles fit easily in memory (this is, you have a reasonable rotation policy) you can easily do something like:

log_lines = open('logfile','r').readlines()
last_line = get_last_lineprocessed() #From some persistent storage
last_line = parse_log(log_lines[last_line:])
store_last_lineprocessed(last_line)

If you cannot do this, you can use something like (see accepted answer's use of seek and tell, in case you need to do it with them) http://stackoverflow.com/questions/136168/get-last-n-lines-of-a-file-with-python-similar-to-tail

Answer 2

You can manage the position in the file thanks to the seek and tell methods of the file class see http://docs.python.org/library/stdtypes.html#file-objects

The tell method will tell you where to seek next time you open

Answer 3

If you're parsing your log line per line, you could juste save line number from the last parsing. You would juste have then to start read it from the good line the next time.

Seeking is more usefull when you have to be in a very specific place in the file.

Answer 4

Easy but not recommended :):

last_line_processed = get_last_line_processed()    
with open('file.log') as log
    for record_number, record in enumerate(log):
        if record_number >= last_line_processed:
            parse_log(record)

Answer 5

log = open('myfile.log')
pos = open('pos.dat','w')
print log.readline()
pos.write(str(f.tell())
log.close()
pos.close()

log = open('myfile.log')
pos = open('pos.dat')
log.seek(int(pos.readline()))
print log.readline()

Of course you shouldn't use it like that - you should wrap the operations up in functions like save_position(myfile) and load_position(myfile), but the functionality is all there.

Answer 6

Note that you can seek() in python from the end of the file:

f.seek(-3, os.SEEK_END)

puts the read position 3 lines from the EOF.

However, why not use diff, either from the shell or with difflib?

Answer 7

Here is code proving using the length sugestion of yours and the tell methond:

beginning="""line1
line2
line3"""

end="""- The log will open from this point
line4
line5"""

openfile= open('log.txt','w')
openfile.write(beginning)
endstarts=openfile.tell()
openfile.close()

open('log.txt','a').write(end)
print open('log.txt').read()

print("\nAgain:")
end2 = open('log.txt','r')
end2.seek(len(beginning))

print end2.read()  ## wrong by two too little because of magic newlines in Windows
end2.seek(endstarts)

print "\nOk in Windows also"
print end2.read()
end2.close()