Python http download page source

Question

hello there i was wondering if it was possible to connect to a http host (I.e. for example google.com) and download the source of the webpage?

Thanks in advance.

Answer 1

you can use urllib2 module.

import urllib2
url="http://somewhere.com"
page =urllib2.urlopen(url)
data=page.read()
print data

see the doc for more examples

Answer 2

The documentation of httplib (low-level) and urllib (high-level) should get you started. Choose the one that's more suitable for you.

Answer 3

Using urllib2 to download a page.

Google will block this request as it will try to block all robots. Add user-agent to the request.

import urllib2
user_agent = 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.63 Safari/534.3'
headers = { 'User-Agent' : user_agent }
req = urllib2.Request('http://www.google.com', None, headers)
response = urllib2.urlopen(req)
page = response.read()

You can also use pyCurl

import sys
import pycurl

class ContentCallback:
        def __init__(self):
                self.contents = ''

        def content_callback(self, buf):
                self.contents = self.contents + buf

t = ContentCallback()
curlObj = pycurl.Curl()
curlObj.setopt(curlObj.URL, 'http://www.google.com')
curlObj.setopt(curlObj.WRITEFUNCTION, t.content_callback)
curlObj.perform()
curlObj.close()
print t.contents