Script to connect to a web page

Question

Looking for a python script that would simply connect to a web page (maybe some querystring parameters).

I am going to run this script as a batch job in unix.

Answer 1

urllib2 will do what you want and it's pretty simple to use.

import urllib
import urllib2

params = {'param1': 'value1'}

req = urllib2.Request("http://someurl", urllib.urlencode(params))
res = urllib2.urlopen(req)

data = res.read()

It's also nice because it's easy to modify the above code to do all sorts of other things like POST requests, Basic Authentication, etc.

Answer 2

What are you trying to do? If you're just trying to fetch a web page, cURL is a pre-existing (and very common) tool that does exactly that.

Basic usage is very simple:

curl www.example.com

Answer 3

You might want to simply use httplib from the standard library.

myConnection = httplib.HTTPConnection('http://www.example.com')

you can find the official reference here: http://docs.python.org/library/httplib.html

Answer 4

A simple wget called from a shell script might suffice.

Answer 5

Try this:

aResp = urllib2.urlopen("http://google.com/");
print aResp.read();

Answer 6

If you need your script to actually function as a user of the site (clicking links, etc.) then you're probably looking for the python mechanize library.

Python Mechanize