How can I quickly fix EBCDIC control characters in large files using Perl?

Question

My apologies if this comes across as a newbie question. I'm not a Perl developer, but am trying to use it within an automation process, and I've hit a snag.

The following command runs quickly (a few seconds) on my Linux system (Ubuntu 9.10 x64, Perl 5.10), but is extremely slow on a Windows system (Windows 2003 x86, Strawberry Perl 5.12.1.0).

perl -pe 's/\x00\x42\x00\x11/\x00\x42\x00\xf0/sgx' inputfile > outputfile

The pattern to find/replace hex characters is intended to fix EBCDIC carriage control characters in a file that is between 500MB to 2GB in size. I'm not sure if this is even the most efficient way to do this, but it would seem to do the trick... if only it would run quickly on the Windows system it needs to run on.

Any thoughts?

Answer 1

Note that there is a distinction between text and binary files on Windows. Text files are subject to automatic EOL conversion which I assume might add to the run time as well as potentially messing up your binary substitution (presumably not the case here).

Also, there is no point using the /sx with this substitution.

I think the heart of the matter boils down to this: With the -p switch, you are supposed to be processing the input line-by-line. Where is the first EOL (as understood by perl) in the file? Are you trying to read a huge string into memory, do the s/// on it and write out?

How about using the following script:

#!/usr/bin/perl

use strict; use warnings;
$/ = "\x00\x42\x00\x11";
$\ = "\x00\x42\x00\xf0";

while ( <> ) {
    chomp;
    print;
}

Also, you absolutely need to use double-quotes on Windows. Compare and contrast:

C:\Temp> perl -pe 's/perl/merl/' t.pl
#!/usr/bin/perl
...

C:\Temp> perl -pe "s/perl/merl/" t.pl
#!/usr/bin/merl
...