Linux command to replace string in LARGE file with another string

Question

I have a huge SQL file that gets executed on the server. The dump is from my machine and in it there are a few settings relating to my machine. So basically, I want every occurance of "c://temp" to be replace by "//home//some//blah"

How can this be done from the command line?

Answer 1

sed is a good choice for large files.

sed -i.bak -e 's%C://temp%//home//some//blah%' large_file.sql

It is a good choice because doesn't read the whole file at once to change it. Quoting the manual:

A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline). While in some ways similar to an editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient. But it is sed's ability to filter text in a pipeline which particularly distinguishes it from other types of editors.

The relevant manual section is here. A small explanation follows

-i.bak enables in place editing leaving a backup copy with .bak extension

s%foo%bar% uses s, the substitution command, which substitutes matches of first string in between the % sign, 'foo', for the second string, 'bar'. It's usually written as s// but because your strings have plenty of slashes, it's more convenient to change them for something else so you avoid having to escape them.

Example

vinko@mithril:~$ sed -i.bak -e 's%C://temp%//home//some//blah%' a.txt
vinko@mithril:~$ more a.txt
//home//some//blah
D://temp
//home//some//blah
D://temp
vinko@mithril:~$ more a.txt.bak
C://temp
D://temp
C://temp
D://temp

Answer 2

The sed command can do that. Rather than escaping the slashes, you can choose a different delimiter (_ in this case):

sed -e 's_c://temp/_/home//some//blah/_' file1.txt > file2.txt

Answer 3

Try sed? Something like:

sed 's/c:\/\/temp/\/\/home\/\/some\/\/blah/' mydump.sql > fixeddump.sql

Escaping all those slashes makes this look horrible though, here's a simpler example which changes foo to bar.

sed 's/foo/bar/' mydump.sql > fixeddump.sql

As others have noted, you can choose your own delimiter, which would prevent the leaning toothpick syndrome in this case:

sed 's|c://temp\\|home//some//blah|' mydump.sql > fixeddump.sql

The clever thing about sed is that it operating on a stream rather than a file all at once, so you can process huge files using only a modest amount of memory.

Answer 4

Just for completeness. In place replacement using perl.

perl -i -p -e 's{c://temp}{//home//some//blah}g' mysql.dmp

No backslash escapes required either. ;)

Answer 5

There's also a non-standard UNIX utility, rpl, which does the exact same thing that the sed examples do; however, I'm not sure whether rpl operates streamwise, so sed may be the better option here.

Answer 6

perl -pi -e 's#c://temp#//home//some//blah#g' yourfilename

The -p will treat this script as a loop, it will read the specified file line by line running the regex search and replace.

-i This flag should be used in conjunction with the -p flag. This commands Perl to edit the file in place.

-e Just means execute this perl code.

Good luck

Answer 7

gawk

awk '{gsub("c://temp","//home//some//blah")}1' file