Reading files in python

Question

Trying to understand how you're supposed to read files in python. This is what I've done and it isn't working quite properly:

import os.path

filename = "A 180 mb large file.data"
size = os.path.getsize(filename)

f = open(filename, "r")
contents = f.read()
f.close()

print "The real filesize is", size
print "The read filesize is", len(contents)

f = open(filename, "r")

size = 0

while True:
    contents = f.read(4)
    if not contents: break
    size += len(contents)

f.close()

print "this time it's", size

Outputs:

The real filesize is 183574528
The read filesize is 10322
this time it's 13440

Somebody knows whats going on here? :)

Answer 1

If your file confuses the C libraries, then your results are expected.

The OS thinks it's 180Mb.

However, there are null bytes scattered around, which can confuse the C stdio libraries.

Try opening the file with "rb" and see if you get different results.

Answer 2

The first is the filesize in bytes, the other times you read the file as text and count characters. Change all open(filename, "r") to open(filename, "rb") and it works.

Answer 3

This is not about strings : Python is perfectly happy with null bytes in strings.

This is because you are on Windows and you open the file in text mode, so it converts all "\n" into "\r\n", thus destroying all your binary data.

Open your file in binary mode with mode "rb"