Searching a website

Question

import urllib
import re
import os
search = (raw_input('[!]Search: '))
site = "http://www.exploit-db.com/list.php?description="+search+"&author=&platform=&type=&port=&osvdb=&cve="   
print site
source = urllib.urlopen(site).read()
founds = re.findall("href='/exploits/\d+",source)
print "\n[+]Search",len(founds),"Results\n"
if len(founds) >=1:
        for found in founds:
                found = found.replace("href='","")
                print "http://www.exploit-db.com"+found
else:
        print "\nCouldnt find anything with your search\n"

When I search the exploit-db.com site I only come up with 25 results, how can I make it go to the other page or go pass 25 results.

Answer 1

Easy to check by just visiting the site and looking at the URLs as you manually page: just put right after the ? in the URL page=1& to look at the second page of results, or page=2& to look at the third page, and so forth.

How is this a Python question? It's a (very elementary!) "screen scraping" question.

Answer 2

Apparently the exploit-db.com site doesn't allow extending the page size. You therefore need to "manually" page through the result list by repeating the urllib.urlopen() to get subsequent pages. The URL is the same as the one initially used, plus the &page=n parameter. Attention this n value appears to be 0-based (i.e. &page=1 will give the second page)