I'm trying to fetch Wikipedia pages using LWP::Simple, but they're not coming back. This code: #!/usr/bin/perl use strict; use LWP::Simple; print get("http://en.wikipedia.org/wiki/Stack_overflow"); doesn't print anything. But if I use some other webpage, say http://www.google.com, it works fine. Is there some other name that I shou...
I am trying to implement a request to an unreliable server. The request is a nice to have, but not 100% required for my perl script to successfully complete. The problem is that the server will occasionally deadlock (we're trying to figure out why) and the request will never succeed. Since the server thinks it is live, it keeps the so...
How can I get LWP to verify that the certificate of the server I'm connecting to is signed by a trusted authority and issued to the correct host? As far as I can tell, it doesn't even check that the certificate claims to be for the hostname I'm connecting to. That seems like a major security hole (especially with the recent DNS vulnera...
I have a client/server system that performs communication using XML transferred using HTTP requests and responses with the client using Perl's LWP and the server running Perl's CGI.pm through Apache. In addition the stream is encrypted using SSL with certificates for both the server and all clients. This system works well, except that ...
How can I extract information from a website (http://tv.yahoo.com/listings) and then create an XML file out of it? I want to save it so to parse later and display information using JavaScrit? I am quite new to Perl and I have no idea about how to do it. ...
I would like to know whether it is possible to force LWP::UserAgent to accept an expired SSL certificate for a single, well-known server. The issue is slightly complicated by the Squid proxy in between. I went as far as to set up a debugging environment like: use warnings; use strict; use Carp; use LWP::UserAgent; use LWP::Debug qw(+);...
How can I add proxy support to this script? use LWP::Simple; $url = "http://stackoverflow.com"; $word = "how to ask"; $content = get $url; if($content =~ m/$word/) { print "Found $word"; } ...
Hi all, I need to install LWP::Parallel::UserAgent to run on Windows environment, I used ActivePerl 5.10. I search on Google but did not find any information about repo for this package Thanks, Minh. ...
Hi, I am using WWW::Mechanize and currently handling HTTP responses with the 'Content-Encoding: gzip' header in my code by first checking the response headers and then using IO::Uncompress::Gunzip to get the uncompressed content. However I would like to do this transparently so that WWW::Mechanize methods like form(), links() etc work...
I am trying to get an image from an HTTP server using Perl. I have the full URL of the file and am attempting to use my $data = LWP::Simple::get $params{URL}; my $filename = "image.jpg"; open (FH, ">$filename"); print FH $data; close (FH); Now, logically, to me at least, this should work. But the files are slightly different sizes,...
I'm building a webcrawler in Perl/LWP. How can the webcrawler follow a link in a ASP.NET grid like this: <a id="ctl00_MainContent_listResult_Top_LnkNextPage" href="javascript:__doPostBack('ctl00$MainContent$listResult$Top$LnkNextPage','')">Next</a> ...
I would like to write a script to login to a web application and then move to other parts of the application: use HTTP::Request::Common qw(POST); use LWP::UserAgent; use Data::Dumper; $ua = LWP::UserAgent->new(keep_alive=>1); my $req = POST "http://example.com:5002/index.php", [ user_name => 'username', user_password => "passwor...
I am fetching some pages over the Web using Perl's LWP::UserAgent and would like to be as polite as possible. By default, LWP::UserAgent does not seamlessly handle compressed content via gzip. Is there an easy way to make it do so, to save everyone some bandwidth? ...
I'm using WWW::Mechanize to do some standard website traversal, but at one point I have to construct a special POST request and send it off. All this requires session cookies. In the POST request I'm making, spaces are being encoded to + symbols, but I need them encoded as a %20. I can't figure out how to alter this behaviour. I realis...
I need to perform a large number of HTTP post requests, and ignore the response. I am currently doing this using LWP::UserAgent. It seems to run somewhat slow though I am not sure if it is waiting for a response or what, is there anyway to speed it up and possibly just ignore the responses? ...
When using the perl module LWP::Simple, is there a simple way to determine the speed and amount downloaded by a single getstore() invocation? This would be useful for observing the status of large file downloads. Off the top of my head, one approach would be to: store the current time (time0) run getstore in a new process poll the kn...
I'm trying to access a protected file. Server is using digest authentication - which I can see from the printed out response. Here is the sample code: use LWP; use strict; my $url = 'http://somesite.com/aa/bb/cc.html'; my $username = 'scott'; my $password = 'tiger'; my $browser = LWP::UserAgent->new('Mozilla'); $browser->credentials("...
I find that the return from LWP::UserAgent->request() contains both the header and body of a HTTP response. I just need the body of the response to do some parsing, so how can I do? ...
This is my first post on SO, so be gentle. I'm not even sure if this belongs here, but here goes. I want to access some information on one of my personal accounts. The website is poorly written and requires me to manually input the date I want the information for. It is truly a pain. I have been looking for an excuse to learn more Perl ...
I've used LWP capability to handle gzip encoded content as described here, but in some cases I randomly get unexpected results at least for the one website I've tested: $response->decoded_content could become undefined while $response->content still returns original gzip encoded response. Tried even without internal charset decoding (dec...