I am looking for a good framework for building web apps in python (could also be python + php). The apps will make http requests (GET/POST) and process data, so I'd like to have threading support to make it faster. I am currently using exec(/path/to/php thread.php ... ); to run "threads" on php.

The main app will scrap tons of urls and then there are other apps like "modules" that analize the data. I mainly looking for a framework that allow me to manage threads via a web interface, those threads would run GET/POST request and retrieve data that will be stored for later use by other apps.

+2  A:

Tornado is an open source version of the scalable, non-blocking web server and and tools that power FriendFeed

+1 For Tornado - rarely gets mentioned. Very nice bit of software.
+3  A: 

I use Django a lot:

That's all python and html-templates. It's not "multi-threaded", but you can set up your server to have multiple processes handling incoming requests, so that one user doesn't hang others, if that's what you're worrying about when you ask for "multi-threaded". But that's a separate question.

I want multi threading because I will be running a "pseudo spider" bot that will request a lot of urls and having only one script doing it is slow as hell, also I would like to have an easy way to manage the web interface to interact (modify settings, start-stop, etc) with threads.
there's not much you can do for true multithreading in python. but you can do some nice work with multiprocessing. as long as you don't need extremely fast intercommunication between them, there's not a lot of difference.
or if you need the process of creating a new "thread" to be lightweight - then, also, threads make a big difference over processes.

I've been very impressed with


You'll want to check out WebOB. It uses WSGI, and can be used with threaded servers no problem. Ian Bicking, who helped with the Python WSGI standard, worte a great post you should check out: What does a WebOB application look like?

Also: Search first!

+1  A: 

I would take a look at gevent. You could use greenlets to divide up the work, and the various synchronization classes provided to enable communication. gevent also has a WSGI server, so you could implement your web interface with a simple framework like Flask,, etc.


I'm working on a somewhat similar project, but we are not worried with performance/threads yet.

For the scrapping/crawling thing there's a great Python framework called Scrappy.

As was said before, Python does not support multi-threading natively, but there is a project called Parallel Python that seems pretty easy to integrate with anything. I'm planning to use it if needed.

For the front-end I'm just using good old Django. :)

Brunno Gomes

Do you have the option of running under Jython? If so, you could e.g. build your web UI in something "normal" like Django/web2py, then have the processing handed off to run under JVM threads.

Just a thought...