What is the quickest way to HTTP GET in Python?

Question

What is the quickest way to HTTP GET in Python if I know the Content will be a string? I am searching the docs for a quick one-liner like:

contents = url.get("http://example.com/foo/bar")

But all I can find using Google are httplib and urllib - and I am unable to find a shortcut in those libraries.

Does standard Python 2.5 have a shortcut in some form as above, or should I write a function url_get?

1. I would prefer not to capture the output of shelling out to wget or curl.

Answer 1

import urllib2
urllib2.urlopen("http://example.com/foo/bar").read()

How is that?

Answer 2

Have a look at httplib2, which - next to a lot of very useful features - provides almost exactly what you want. You do have to create a Http object first, so it's not exactly a oneliner:

import httplib2
http = httplib2.Http()

resp, content = http.request("http://example.com/foo/bar")

Where content would be the response body (as a string), and resp would contain the status and response headers.

It doesn't come included with a standard python install though (but it only requires standard python), but it's definitely worth checking out.

Answer 3

If you want solution with httplib2 to be oneliner consider instatntinating anonymous Http object

import httplib2
resp, content = httplib2.Http().request("http://example.com/foo/bar")

Answer 4

Here is a wget script in Python:

# From python cookbook, 2nd edition, page 487
import sys, urllib

def reporthook(a, b, c):
    print "% 3.1f%% of %d bytes\r" % (min(100, float(a * b) / c * 100), c),
for url in sys.argv[1:]:
    i = url.rfind("/")
    file = url[i+1:]
    print url, "->", file
    urllib.urlretrieve(url, file, reporthook)
print

Answer 5

theller's solution for wget is really useful, however, i found it does not print out the progress throughout the downloading process. It's perfect if you add one line after the print statement in reporthook.

import sys, urllib

def reporthook(a, b, c):
    print "% 3.1f%% of %d bytes\r" % (min(100, float(a * b) / c * 100), c),
    sys.stdout.flush()
for url in sys.argv[1:]:
    i = url.rfind("/")
    file = url[i+1:]
    print url, "->", file
    urllib.urlretrieve(url, file, reporthook)
print