I have a fairly simple perl script with uses the LWP::UserAgent module to follow URLs through redirection to find the final destination URL which it then stores in our MySQL database. The problem is that from time to time the script reports warnings that look like this: Day too big - 25592 > 24855 Sec too small - 25592 < 74752 Sec too ...
The following code ... my $user_agent = LWP::UserAgent->new; my $request = HTTP::Request->new(GET => $url); my $response = $user_agent->request($request); if ($response->is_success) { print "OK\n"; } else { die($response->status_line); } .. will fail with .. 500 Can't connect to <hostname> (Bad hostname '<hostname>') .. if ...
Hey, Lets say i have this code: use strict; use LWP qw ( get ); my $content = get ( "http://www.msn.co.il" ); print STDERR $content; The error log shows something like "\xd7\x9c\xd7\x94\xd7\x93\xd7\xa4\xd7\xa1\xd7\x94" which i'm guessing it's utf-16 ? The website's encoding is with <META HTTP-EQUIV="Content-Type" CONTENT="text/h...
Example code: my $ua = LWP::UserAgent->new; my $response = $ua->get('http://example.com/file.zip'); if ($response->is_success) { # get the filehandle for $response->content # and process the data } else { die $response->status_line } I need to open the content as a file without prior saving it to the disk. How would you do ...
I'm doing some web scraping using Perl's LWP. I need to process a set of URLs, some of which may redirect (1 or more times). How can I get ultimate URL with all redirects resolved, using HEAD method? ...
#!/usr/bin/perl use LWP::UserAgent; use HTTP::Request::Common; use HTML::TreeBuilder; use Cwd 'realpath'; use warnings; use strict; my $ua = LWP::UserAgent->new(); my $response = $ua->request( POST 'http://mydomain.com/upload.php', 'Content-Type' => 'multipart/form-data', 'Content' => [...
I am trying to use LWP::Simple to make a GET request to a REST service. Here's the simple code: use LWP::Simple; $uri = "http://api.stackoverflow.com/0.8/questions/tagged/php"; $jsonresponse= get $uri; print $jsonresponse; On my local machine, running Ubuntu 10.4, and Perl version 5.10.1: farhan@farhan-lnx:~$ perl --version This...
I am having trouble with a perl screenscraper to an HTTPS site. In debugging, I ran the following: print $res->headers_as_string; and in the output, I have the following line: Client-SSL-Warning: Peer certificate not verified Is there a way I can auto-accept this certificate, or is that not the problem? #!/usr/bin/perl use LWP::...
Is it possible to read HTML Web 2.0 Source Code that is dynamically generated ? The Perl LWP with its agent->response does not pick up any dynamically generated HTML code. Many websites today are generating dynamic html. If I am shoppping for best prices, and the prices are dynamically fetched and dumped, then I am out of business. Are...
I'm requesting a web page using LWP in perl, and I'd like to be able to access the SSL certificate that the web server presents (I'm looking for an expiration date in the cert, among other things). The information I want isn't in the three headers that Crypt::SSLeay adds to the request. Is there a way that I'm overlooking with which I ...
Hey. I don't know anything about Perl but I urgently need to modify a Perl script. At some point it's downloading an about 500MB file from a server using system("lwp-download $HttpPath $Out");. Is there any way I can find out if the downloading process went correctly, e.g. check whether downloaded file has the same size as the original...
The following PHP code does exactly what I want to do. The problem is I need to re-create it in Perl and I've been playing around with the open() and sysopen() Perl functions but can't do it. Does anyone have any help or know of any links that might help? Thanks. $URL = "http://example.com/api.php?arguments=values"; echo file_get_conten...
We have a few different websites running on the same server that all access 1 particular web service with each having their own unique API key. Unfortunately the web service has a daily limit based on IP address (not API key) so while each of our sites is way under their daily limit, combined they are over the IP limit. When accessed via...
Hi. I'm trying to get Code Closure to work, but unfortunately, there's always an error thrown. Here's the code: use LWP::UserAgent; use HTTP::Request::Common; use HTTP::Response; my $name = 'test.js'; my $agent = new LWP::UserAgent(); $agent->agent("curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1....
Hello, I wrote a quick script to download files using LWP::Simple library and its getstore() function. It is working rather well, but occasionally downloaded file is not complete. I do not know what is causing this, but when I download it afterward manually using wget in command line file is OK. I would guess corrupted files are caus...
I'm trying to update my status through the Twitter API and OAuth. I get stuck on the last step, the status update. Here's my code. The header: $ua->default_header('Content-Type' => "application/x-www-form-urlencoded"); $ua->default_header('oauth_signature' => "$signature"); $ua->default_header('Authorization' => '"OAuth realm="Twitter...
I want to make a program that communicates with http://www.md5crack.com/crackmd5.php. My goal is to send the site a hash (md5) and hopefully the site will be able to crack it. After, I would like to display the plaintext of the hash. My problem is sending the data to the site. I looked up articles about using LWP however I am still lost....
Hello good evening dear stackoverflow-friends, A triple job: i have to do a job with tree task. we have three tasks: Fetch pages Parse HTML Store data... And yes - this is a true Perl-job! i have to do a parser-job on all 6000 sub-pages of a site in suisse. (a governmental site - which has very good servers ). see http://www.e...
I'm working with a web application that sends some non-standard HTTP headers in its response to a login request. The header in question is: SSO_STATUS: LoginFailed I tried to extract it with LWP::Response as $response->header('SSO_STATUS') but it does not work. It does work for standard headers such as Set-Cookie, Expires etc. Is th...