Python stack for data centric scalable REST application

Question

What Python frameworks/libraries you would use to create a data centric REST application. The application will have the following traits:

• read-biased in terms of number of individual ad-hoc requests
• write-biased in number of records added in batch feeds
• scalable (think virtual appliance, cloud)
• variety of data formats such as csv files, XML, Excel, SQL RDBMS, JSON - think uniform shell over disparate storage formats
• multiple RDBMS instances accessible at the same time
• caching - I am biased toward out of the box HTTP cache like Varnish but if someone could comment on trade-off vs. memcache that would be helpful
• search - non-Python Apache Solr (Lucene) is what I have in mind but am curious to find out about other options
• application will generally have no GUI but I would like to be able to create web based usage demos with ease
• interoperability with SOAP based web services (not a high priority)
• URL inference from model - not sure how feasible this will be, after all URL is not a query language, but would be nice if simple filtering did not involve stating URL patterns explicitly

For the main framework I have my eye on web.py, Werkzeug and Pylons. I used Django in the past and like how well supported it is. Some projects such as django-storage could be a great shortcut but I am uncertain whether it's main purpose on display/web GUI apps is the right choice for me.

Answer 1

I think the only things you need are werkzeug and SQLAlchemy.

I'm not sure that you need web.py or pylons. A simple set of WSGI application will easily handle authentication, caching and routing of requests from URL/method to a specific Python class that does the real work.

With SQLAlchemy you can have all of the database adapters wrapped in a consistent ORM.

---

"Any ideal with regards to avoiding boilerplate code for URL dispatch?"

Not sure how to answer this. It's a fundamental feature of WSGI that everything can be reused in a trivial way.

There are lots of URL-dispatching WSGI applications. Lots.

Start by reading the wsgiref module in the standard library: http://docs.python.org/library/wsgiref.html#wsgiref.util.shift_path_info. This shows the common design pattern to URL parsing in a WSGI environment: update the environment and invoke an application with the enriched environment.

Look at werkzeug's URL routing. http://werkzeug.pocoo.org/documentation/0.5.1/routing.html

Look at additional URL routing.

• http://pesto.redgecko.org/dispatch.html

• http://pypi.python.org/pypi/urlrelay/0.6

Answer 2

Rolling your own solutions with a lightweight framework is a great option - but be prepared to work on the glue code. It sounds like you have a pretty good grasp on the individual pieces you'll need, though, so you might like something like mnml or another super-micro framework to start with.

If you're liking Django, though, be sure to check out the django multi-db branch for your requirement for multiple RDBMSes. It's pretty stable and in the process of being merged to trunk. Check out django-piston to simplify the REST API boilerplate and managing serializations. Most of your variety of storage formats could be written as Django serializers for its serialization framework, too.

Django's caching framework is pretty great and wonderfully granular for everything from HTTP responses to database rows to bits of Python data, and it supports a variety of backends.

Answer 3

Werkzeug seems like the best choice for this task, especially since it has excellent URL routing. If you're only interested in providing JSON APIs, you could look at wsloader (http://code.google.com/p/wsloader), which is written using Werkzeug.