Hey,
I'm planning on writing a web crawler and a web-based front end for it (or, at least, the information it finds). I was wondering if it's possible to use the Django framework to let the web crawler use the same MySQL backend as the website (without making the web crawler a "website" in it's self).
Cheers,
Pete
Yes, you can use the same database.
Some people use Django on top of a PHP application for its admin functionality, or to build newer features with Django and its ORM.
What I'm trying to say is that if you're putting data from your crawl into the same place that you will let Django store its data, you can access them as long as you create Django models for each table.
However, I don't see why the crawler can't be written within Django itself. I've written some non web based apps (a crawler and an aggregator) in Django and it works quite well.
You can use Django ORM outside of an HTTP server.
Basically you need to set DJANGO_SETTINGS_MODULE environment variable. Then you can import and use your django code. Here's an article on stand-alone Django scripts.
Alternatively you can choose to interact with your Django server via custom management commands. This will be a bit more work. But in the end this method allows for a greater decoupling between the crawler and the controller (Django project).