Unable to use SED to edit files fast

Question

The file is initially

$cat so/app.yaml 
application: SO
...

I run the following command. I get an empty file.

$sed s/SO/so/ so/app.yaml > so/app.yaml 
$cat so/app.yaml 
$

How can you use SED to edit the file and not giving me an empty file?

Answer 1

$ sed -i -e's/SO/so/' so/app.yaml

The -i means in-place.

Answer 2

I believe that redirecting output into the same file you are editing is causing your problem.

You need redirect standard output to some temporary file and when sed is done overwrite the original file by the temporary one.

Answer 3

The > used in piping will open the output file when the pipes are all set up, i.e. before command execution. Thus, the input file is truncated prior to sed executing. This is a problem with all shell redirection, not just with sed.

Sheldon Young's answer shows how to use in-place editing.

Answer 4

You are using the wrong tool for the job. sed is a stream editor (that's why it's called sed), so it's for in-flight editing of streams in a pipe. ed OTOH is a file editor, which can do everything sed can do, except it works on files instead of streams. (Actually, it's the other way round: ed is the original utility and sed is a clone that avoids having to create temporary files for streams.)

ed works very much like sed (because sed is just a clone), but with one important difference: you can move around in files, but you can't move around in streams. So, all commands in ed take an address parameter that tells ed, where in the file to apply the command. In your case, you want to apply the command everywhere in the file, so the address parameter is just , because a,b means "from line a to line b" and the default for a is 1 (beginning-of-file) and the default for b is $ (end-of-file), so leaving them both out means "from beginning-of-file to end-of-file". Then comes the s (for substitute) and the rest looks much like sed.

So, your sed command s/SO/so/ turns into the ed command ,s/SO/so/.

And, again because ed is a file editor, and more precisely, an interactive file editor, we also need to write (w) the file and quit (q) the editor.

This is how it looks in its entirety:

ed -- so/app.yaml <<-HERE
    ,s/SO/so/
    w
    q
HERE

See also my answer to a similar question.

What happens in your case, is that executing a pipeline is a two-stage process: first construct the pipeline, then run it. > means "open the file, truncate it, and connect it to filedescriptor 1 (stdout)". Only then is the pipe actually run, i.e. sed is executed, but at this time, the file has already been truncated.

Some versions of sed also have a -i parameter for in-place editing of files, that makes sed behave a little more like ed, but using that is not advisable: first of all, it doesn't support all the features of ed, but more importantly, it is a non-standardized proprietary extension of GNU sed that doesn't work on many non-GNU systems. It's been a while since I used a non-GNU system, but last I used one, neither Solaris nor OpenBSD nor HP-UX nor IBM AIX sed supported the -i parameter.