Unable to find an internet page blocked by robots.txt

Question

Problem: to find answers and exercises of lectures in Mathematics at Uni. Helsinki

Practical problems

1. to make a list of sites with .com which has Disallow in robots.txt
2. to make a list of sites at (1) which contain files with *.pdf
3. to make a list of sites at (2) which contain the word "analyysi" in pdf-files

Suggestions for practical problems

1. Problem 3: to make a compiler which scrapes data from pdf-files

Questions

1. How can you search .com -sites which are registered?
2. How would you solve the practical problems 1 & 2 by Python's defaultdict and BeautifulSoap?

Answer 1

If I understand your requirements, you'd essentially have to spider every possible site in order to see which one(s) match your criteria. I don't see any faster or more efficient solution, regardless of what tools you use.

Answer 2

If I understand you correctly then I don't see how this is possible without, as mentioned already, scanning the entire internet. You are looking for pages on the internet which are not on Google? There is not a database of every site on the net and if they are indexed by a search engine or not...

You would literally need to index the entire web and then go though each site and check if they are on google.

I am also confused if this relates in one site or the web since your question seems to switch between both.

Answer 3

Do you mean that you have your lectures on a web page of your University's intranet and that you would like to be able to access this page from outside your University's intranet?

I assume that in order to access your Uni's intranet you must enter a password, and that Google does not index any of the Uni's intranet pages -- which is the nature of an intranet.

If all the above assumptions are correct then you simply need to host your pdf files on a website outside your University's intranet. Simplest way is to start a blog (no cost involved and very easy and quick to do) and then post your pdf files there.

Google will then index your pages and also "scrape data" from your pdf's as you put it, which means that the text within your pdf files will be searchable.

Answer 4

I am trying to find every web site on the internet that has a pdf-file which has the word "Analyysi"

Not an answer to your question, but: PLEASE respect the site owner's wish to NOT be indexed.

Answer 5

Your questions are faulty.

With respect to (2), you are making the faulty assumption that you can find all PDF files on a webserver. This is not possible, for multiple reasons. The first reason is that not all documents may be referenced. The second reason is that even if they are referenced, the reference itself may be invisible to you. Finally, there are PDF resources which are generated on the fly. That means they do not exist until you ask for them. And since they depend on your input, there's an infinite amount of them.

Question 3 is faulty for pretty much the same reasons. In particular, the generated PDF may contain the word "analyysi" only if you used it in the query. E.g. http://example.com/makePDF.cgi?analyysi

Answer 6

I outline:

1. Law

"The problem comes with enforcing that law! In principal it is easy, in practice it is expensive!" source

"There is no law stating that /robots.txt must be obeyed, nor does it constitute a binding contract between site owner and user, but having a /robots.txt can be relevant in legal cases." source

2. Practise

disallow filetype:txt

3. Theoretically Possible?