Python Web-Scrape Loop via CSV list of URLs ???

Question

HI, I've got a list of 10 websites in CSV. All of the sites have the same general format, including a large table. I only want the the data in the 7th columns. I am able to extract the html and filter the 7th column data (via RegEx) on an individual basis but I can't figure out how to loop through the CSV. I think I'm close but my script won't run. I would really appreciate it if someone could help me figure-out how to do this. Here's what i've got:

#Python v2.6.2

import csv 
import urllib2
import re

urls = csv.reader(open('list.csv'))
n =0
while n <=10:
    for url in urls:
        response = urllib2.urlopen(url[n])
        html = response.read()
        print re.findall('td7.*?td',html)
        n +=1

Answer 1

When I copied your routine, I did get a white space / tab error error. Check your tabs. You were indexing into the URL string incorrectly using your loop counter. This would have also messed you up.

Also, you don't really need to control the loop with a counter. This will loop for each line entry in your CSV file.

#Python v2.6.2

import csv 
import urllib2
import re

urls = csv.reader(open('list.csv'))
for url in urls:
    response = urllib2.urlopen(url[0])
    html = response.read()
    print re.findall('td7.*?td',html)

Lastly, be sure that your URLs are properly formed:

http://www.cnn.com
http://www.fark.com
http://www.cbc.ca