How can I alter a file and write only the changes to disk - basically, sed (python)?

Question

Let's say I have a file /etc/conf1

it's contents are along the lines of

option = banana
name = monkey
operation = eat

and let's say I want to replace "monkey" with "ostrich". How can I do that without reading the file to memory, altering it and then just writing it all back? Basically, how can I modify the file "in place"?

Answer 1

You should look at the fileinput module:

http://docs.python.org/library/fileinput.html

There's an option to perform inplace editing via the input method:

http://docs.python.org/library/fileinput.html#fileinput.input

UPDATE - example code:

import fileinput
import re
import sys

for line in fileinput.input(inplace=True):
    sys.stdout.write(re.sub(r'monkey', 'ostrich', line))

Using sys.stdout.write so as not to add any extra newlines in.

Answer 2

You can't. "ostrich" is one letter more than "monkey", so you'll have to rewrite the file at least from that point onwards. File systems do not support "shifting" file contents upwards or downwards.

If it's just a small file, there's no reason to bother with even this, and you might as well rewrite the whole file.

If it's a really large file, you'll need to reconsider the internal design of the file's contents, for example, with a block-based approach.

Answer 3

It depends on what you mean by "in place". How can you do it if you want to replace monkey with supercalifragilisticexpialidocious? Do you want to overwrite the remaining file? If not, you are going to have to read ahead and shift subsequent contents of the file forwards.

Answer 4

CPU instructions operate on data which come from memory.

The portion of the file you wish to read must be resident in memory before you can read it; before you write anything to disk, that information must be in memory.

The whole file doesn't have to be there at once, but to do a search-replace on an entire file, every character of the file will pass through RAM at some point.

What you're probably looking for is something like the mmap() system call. The above fileinput module sounds like a plausible thing to use.

Answer 5

In-place modifications are only easy if you don't alter the size of the file or only append to it. The following example replaces the first byte of the file by an "a" character:

fd = os.open("...", os.O_WRONLY | os.O_CREAT)
os.write(fd, "a")
os.close(fd)

Note that Python's file objects don't support this, you have to use the low-level functions. For appending, open file file with the open() function in "a" mode.

Answer 6

sed -i.bak '/monkey$/newword/' file