Python web crawling and storing to mysql

Question

Hi, Looking for few days for some simple solution for this, but I think that in this moment I am still on the beginning :)

I need good web crawler written in Python to store complete page into mysql database. Small system that I am experimenting uses now PHP Sphider to crawl and store into database. I need something that works almost exact like sphider, but writen in Python. So just storing database to into table where from other scripts taking content and doing the rest of job that I need. Sphider is slow, and want to replace it.

So, I look at scrapy and some other projects but anything didn't feet in my needs, this is my last try before I start coding myself, so if someone know what can solve me this problem please tell me.

Answer 1

http://scrapy.org/ should do what you are looking for