tags:

views:

87

answers:

4
+2  Q: 

Why do I get extra line breaks in the web page I download with Perl?

I'm writing a simple Perl script (on Windows) to download the response of a get request to a url to a file. Pretty straight-forward. Except when it writes to the output file, I get extra line breaks. So like instead of:

<head>
  <title>title</title>
  <link .../>
</head>

I get

<head>

  <title>title</title>

  <link .../>

</head>

Here's the Perl script:

use LWP::Simple;

my $url = $ARGV[0];
my $content = get($url);

open(outputFile, '+>', $ARGV[1]);

print outputFile $content;

close(outputFile);

I suppose I could just get wget for Windows, but now this is bothering me. How do I get rid of those extra line breaks?!

A: 

chomp($content) would be my guess. as it looks like there is natively already set of \n's in it.

EDIT: Sorry I just realized that chomp won't work, unless you split the file up into lines, then chomp each line, as chomp will only chomp the end of the input string, my solution wouldn't help in this case, however, you could split it on \n\n and then join? I do like the solution to use a regex on the string returned in an answer below. however for me the minor modification of: including some additional changes, so it still separates lines but it will check for either 2+ \n's or 2+ \r's or any combination of the two. then returning a \n in it's place, that way it's only going to have one new line per line (hopefully)

$content =~ s/[\n\r]+/\n/g;

EDITED Above again, accidentally put a ! in there for some reason....not sure why

onaclov2000  2010-10-14 14:16:00
A: 

play around with

$content=~s/[\n\r]//g;
zzapper  2010-10-14 14:24:35
+3  A: 

I'm guessing that $content already includes CRLF newlines and Perl's IO layer is doing LF -> CRLF conversion. (Internally, "\n" is a single character in Perl, usually LF). I'd add

binmode(outputFile);

after the open to disable that conversion and write the results of $content directly.

Michael Carman  2010-10-14 14:27:11
+10  A: 
  1. There's no sane reason for the >+ mode in your example code. Just saying.
  2. LWP::Simple has a getstore method. If you're using LWP::Simple, why not use it?
  3. By default, open is going to push the :crlf I/O layer when running on win32, which turns \n into \r\n. But the data you're writing already has \r\n, so you're ending up with too many newlines. If you want data to be written verbatim, you should use binmode, or open the handle with :raw to begin with. LWP already does this correctly.
hobbs  2010-10-14 14:34:23

related questions

Verifying files for testing
How do you create your own moniker (URL Protocol) on Windows systems?
Windows Equivalent of 'nice'
Is this a good way to determine OS Architecture?
Dual vs. Quadcore on a Webserver
Is there a WMI Redistributable Package?
PHP Error - Uploading a file
Ruby On Rails with Windows Vista - Best Setup?
OpenVPN Config file to prevent timeout
How can I create Debian install packages in Windows for a Visual Studio project?
How do I configure and communicate with a serial port?
What's the best setup for Mono development on Windows?
Appropriate pagefile size for SQL Server
XML Editing/Viewing Software
Linux shell equivalent on IIS
What is a better file copy alternative than the Windows default?
Windows Help files - what are the options?
Heap corruption under Win32; how to locate?
Best way to access Exchange using PHP?
Get a preview jpeg of a pdf on windows?
WinForms ComboBox data binding gotcha
How do you schedule tasks in Windows?
Register Windows program with the mailto protocol programmatically
Embedding Windows Media Player for all Browsers
Best Subversion clients for Windows Vista (64bit)