Large number of sample documents

Question

Does anyone know where I can find a huge repository of sample documents, in a variety of subjects? I'm looking for at least a few thousands documents (Office or PDF should be fine) in order to test some algorithms... The documents should have some common ground - for example, a thousand docs related to programming, another thousand related to ecology, etc...

Anyone know where I can get it?

Answer 1

On the internetzzz?

Edit: Me? Not being helpful? :)

import mechanize, urllib, os

template = r"http://www.google.com/search?q=filetype:pdf&hl=en&start=%s&sa=N"
links = []

br = mechanize.Browser()
br.set_handle_robots(False)
br.addheaders = [('User-agent', 'Firefox')]
for i in xrange(0, 30, 10):
    br.open(template % i)
    links.extend((link.url for link in br.links(url_regex="^http.+pdf$"))
for url in links:
    urllib.urlretrieve(url, os.path.basename(url))

Answer 2

Have you tried using wikipedia? Create a script that:

1. Calls http://en.wikipedia.org/wiki/Special:Random to get a random page

2. Follows the resulting redirect, appending ?printable=yes to the end of the url (so as to remove the layout crud) - use wget for these two steps, or its equivalent, it'll do the redirect for you.

3. Pipes the resulting html content through a html->pdf converter.

4. Repeat 1000 times.

That should get you a wide variety of content.

Answer 3

You can just use the advanced search on the Yahoo Search API to specify the document type you are looking for.

http://developer.yahoo.com/search/boss/boss_guide/Web_Search.html#optional_args_web

If you want a large number of word documents then specify the type of document you want then carry out a search based on some pre-selected keywords. That should give you a bunch of documents back.

You could also scrape the advanced Google search and grab the document links that way by specifying a fileType (from a random list), e.g:

http://www.google.co.in/search?q=monkey+badger+filetype%3Apdf