Best way to modify a file when using pipes?

Question

I often have shell programming tasks where I run into this pattern:

cat file | some_script > file

This is unsafe - cat may not have read in the entire file before some_script starts writing to it. I don't really want to write the result to a temporary file (its slow, and I don't want the added complication of thinking up a unique new name).

Perhaps, there is there is a standard shell command that will buffer a whole stream until EOF is reached? Something like:

cat file | bufferUntilEOF | script > file

Ideas?

Answer 1

Using a temporary file is IMO better than attempting to buffer the data in the pipeline.

It almost defeats the purpose of pipelines to buffer them.

Answer 2

You're looking for sponge.

Answer 3

Using a temporary file is the correct solution here. When you use a redirection like '>', it is handled by the shell, and no matter how many commands are in your pipeline, the shell is free to delete and overwrite the output file before any command is executed (during pipeline setup).

Answer 4

Using mktemp(1) or tempfile(1) saves you the expense of having to think up unique filename.

Answer 5

I think that the best way is to use a temp file but if you want another approach, you can use something like awk to buffer up the input into memory before your application starts receiving input. The following script will buffer the all of the input into the lines array before it starts to output it to the next consumer in the pipeline.

{ lines[NR] = $0; }
END {
    for (line_no=1; line_no<NR; ++line_no) {
        print lines[line_no];
    }
}

You can collapse it into a one-liner if you want:

cat file | awk '{lines[NR]=$0;} END {for(i=1;i<NR;++i) print lines[i];}' > file

With all of that, I would still recommend using a temporary file for the output and then overwriting the original file with it.

Answer 6

Like many others, I like to use temporary files. I use the shell process-id as part of the temporary name so that if multiple copies of the script are running at the same time, they won't conflict. Finally, I then only overwrite the original file if the script succeeds (using boolean operator short circuiting - it's a little dense but very nice for simple command lines). Putting that altogether, it would like like

some_script < file > smscrpt.$$ && mv smscrpt.$$ file

This will leave the temporary file if the command fails. If you want to clean up on error, you can change that to:

some_script < file > smscrpt.$$ && mv smscrpt.$$ file || rm smscrpt.$$

BTW, I got rid of the poor use of cat and replaced it will input redirection.