This is a very basic concept, but something I have never been able to articulate that well. and I would like to try to spell it and see where I go wrong.

If I have to, how would I define a "newline character". say if I create a new file in unix(or windows), then does the file store the "end of line" information by inserting a special character in the file called as "new line character". If so, what is its ascii value? I remember that in C programs, I have checked for the read character against the value '\n' . And why this confusing 2 characters to represent end of line characters..

bash$ cat states

Say, I want to insert one line space between the lines and want an output of the form: Desired output:




bash$sed -e 's/\n/\n\n/g' states  does not work.

Why can't I treat "new line character" here just as I would treat any other character and run something like above command. (I understand that one might say that this is a matter of syntax of sed, but could one please explain the intuition behind not allowing this, so that I can get rid of my confusion.

Similarly, inside the vim editor, I can not use :%s/\n/\n\n/g . Why so?

Do I need to further escape \n by using a backslash in sed and from within vim?.



+6  A: 

NewLine (\n) is 10 (0xA) and CarriageReturn (\r) is 13 (0xD).

Different operating systems picked different end of line representations for files. Windows uses CRLF (\r\n). Unix uses LF (\n). Older Mac OS versions use CR (\r), but OS X switched to the Unix character.

Here is a relatively useful FAQ.

OS 9 uses `\r`; they dropped it in OS X and switched to matching Unix
Michael Mrozek
+1 @Michael, OS X certainly does not use `\r`.
Carl Norum
Right, fixed that. You know, ya'll can edit answers too. :)
+1. But you've mentioned LF but not actually clarified what it is or even what it stands for. :)
+7  A: 

From the sed man page:

Normally, sed cyclically copies a line of input, not including its terminating newline character, into a pattern space, (unless there is something left after a "D" function), applies all of the commands with addresses that select that pattern space, copies the pattern space to the standard output, appending a newline, and deletes the pattern space.

It's operating on the line without the newline present, so the pattern you have there can't ever match. You need to do something else - like match against $ (end-of-line) or ^ (start-of-line).

Here's an example of something that worked for me:

$ cat > states
$ sed -e 's/$/\
> /' states



I typed a literal newline character after the \ in the sed line.

Carl Norum
`\n` *does* work in `sed`, so you could also use just`sed 's/$/\n/' states`
Jabir Ali Ouassou
+1 for mentioning pattern spaces btw :-)
Jabir Ali Ouassou
@Jabir, not on my machine.
Carl Norum
+3  A: 
sed 's/$/\n/' states

Try this:

$ sed -e $'s/\n/\n\n/g' states
+5  A: 

Escape characters are dependent on whatever system is interpreting them. \n is interpreted as a newline character by many programming languages, but that doesn't necessarily hold true for the other utilities you mention. Even if they do treat \n as newline, there may be some other techniques to get them to behave how you want. You would have to consult their documentation (or see other answers here).

For DOS/Windows systems, the newline is actually two characters: Carriage Return (ASCII 13, AKA \r), followed by Line Feed (ASCII 10). On Unix systems (including Mac OSX) it's just Line Feed. On older Macs it was a single Carriage Return.

Cogwheel - Matthew Orlando
+1  A: 

I think this post by Jeff Attwood addresses your question perfectly. It takes you through the differences between newlines on Dos, Mac and Unix, and then explains the history of CR (Carriage return) and LF (Line feed).

That post has the basic jist of the issue, but it also has some factual errors and half-truths. You might be better off reading Wikipedia's newline topic.
Adrian McCarthy

There is a good explanation of newline, carriage return and line feed at this wordreference thread.