Creating a spider using Scrapy, Spider generation error.

Question

I just downloaded Scrapy (web crawler) on Windows 32 and have just created a new project folder using the "scrapy-ctl.py startproject dmoz" command in dos. I then proceeded to created the first spider using the command:

scrapy-ctl.py genspider myspider myspdier-domain.com

but it did not work and returns the error:

Error running: scrapy-ctl.py genspider, Cannot find project settings module in python path: scrapy_settings.

I know I have the path set right (to python26/scripts), but I am having difficulty figuring out what the problem is. I am new to both scrapy and python so there is a good possibility that I have failled to do something important.

Also, I have been using eclipse with the Pydev plugin to edit the code if that might cause some problems.

Answer 1

There is a difference between PATH and PYTHON_PATH. Is your PYTHON_PATH set correctly? This path is where python looks to include packages / modules.

Answer 2

Set PYTHONPATH environment variable to python26/scripts.

Answer 3

use the scrapy-ctl.py in the project's dir. that script will know about that project's settings. the main scrapy-ctl.py doesn't have a clue about that specific project's settings.