Is there any python lib to scrape search engine(S) results?

Question

I am looking for a python library to scrape results from search engines (google, yahoo, bing, etc).

I only found for google -> http://github.com/kevinw/xgoogle/tree/253db7ddc8603a9dcb038ae42684cf3499a22a4b

Does someone knows one for multiple search engines?

Answer 1

Scrapy is a pretty cool framework for scraping, but you will have code/configure it to work for the sites you want.

Answer 2

It's not too hard to write them. I usually just use php. Look into curl to retrive the page and then the dom object and dom xpath. You can use xpath to select the parts of the result you want.

Xpath is pretty simple if you install firebug and firexpath. I am working on a position checker right now. Same idea but it returns the position of a domain based on a keyword.