What's the encoding for the Google main page?

Question

When Google's main page communicates with Firefox or Chrome it uses a particular type of encoding (Perl says it is utf.64). However, I can't decode it using such; is it a gzipped enconding? I need to finish an app in Perl that should be able to make sense of the Google homepage using Firefox (like a proxy).

Answer 1

Using LiveHTTPHeaders:

http://www.google.com/

GET / HTTP/1.1
Host: www.google.com
User-Agent: ***
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.7,tr;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: UTF-8,*
Keep-Alive: 115
Connection: keep-alive
Cookie: ***

HTTP/1.1 200 OK
Date: Thu, 18 Mar 2010 15:29:03 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=UTF-8
Content-Encoding: gzip
Server: gws
Content-Length: 4440
X-XSS-Protection: 0

which shows that the data returned is gzipped and the character encoding used is UTF-8.

#!/usr/bin/perl

use strict; use warnings;
use LWP::UserAgent;

my $ua = LWP::UserAgent->new();
$ua->show_progress(1);

my $response = $ua->get('http://google.com/');

if ( $response->is_success ) {
    print $response->decoded_content, "\n";
}

Answer 2

Assuming you are employing LWP or something compatible, just use HTTP::Message::decoded_content. Both content encoding and character encoding is figured out automatically for you.