Looking for a good python framework for web apps (w/threading support)

Question

I am looking for a good framework for building web apps in python (could also be python + php). The apps will make http requests (GET/POST) and process data, so I'd like to have threading support to make it faster. I am currently using exec(/path/to/php thread.php ... ); to run "threads" on php.

The main app will scrap tons of urls and then there are other apps like "modules" that analize the data. I mainly looking for a framework that allow me to manage threads via a web interface, those threads would run GET/POST request and retrieve data that will be stored for later use by other apps.

Answer 1

http://github.com/facebook/tornado

Tornado is an open source version of the scalable, non-blocking web server and and tools that power FriendFeed

Answer 2

I use Django a lot:

http://www.djangoproject.com/

That's all python and html-templates. It's not "multi-threaded", but you can set up your server to have multiple processes handling incoming requests, so that one user doesn't hang others, if that's what you're worrying about when you ask for "multi-threaded". But that's a separate question.

Answer 3

I've been very impressed with web.py

Answer 4

You'll want to check out WebOB. It uses WSGI, and can be used with threaded servers no problem. Ian Bicking, who helped with the Python WSGI standard, worte a great post you should check out: What does a WebOB application look like?

Also: Search first!

Answer 5

I would take a look at gevent. You could use greenlets to divide up the work, and the various synchronization classes provided to enable communication. gevent also has a WSGI server, so you could implement your web interface with a simple framework like Flask, web.py, etc.

Answer 6

I'm working on a somewhat similar project, but we are not worried with performance/threads yet.

For the scrapping/crawling thing there's a great Python framework called Scrappy.

As was said before, Python does not support multi-threading natively, but there is a project called Parallel Python that seems pretty easy to integrate with anything. I'm planning to use it if needed.

For the front-end I'm just using good old Django. :)

Answer 7

Do you have the option of running under Jython? If so, you could e.g. build your web UI in something "normal" like Django/web2py, then have the processing handed off to run under JVM threads.

Just a thought...