how to get Company contact page url

Question

Hi i have csv file which contains company url list like this www.google.com,www.ibm.com.....

Here i want to get contactus or aboutus page url (example http://www.google.com/contact) for each url which is present in csv file i have one idea checking the links with the following patterns (contact us, about us, about, locations).

If you do not find any of those, flag the url and write it into a log file. If you find the pattern, just print the address (it is used for some other process)

Any one help how to do this

Thanks in Advance

Answer 1

use scrapy.

Answer 2

I'd suggest using Beautiful Soup to parse the page. Another alternative would be to setup a HIT on Mechanical Turk.