Hi Everyone,
I need your advices to choose a Python Web Framework for developing a large project:
Database (Postgresql)will have at least 500 tables, most of them with a composite primary key, lots of constraints, indexes & queries. About 1,500 views for starting. The project belongs to the financial area. Alwasy new requirements are coming.
Will a ORM be helpful?
Thanks in advance.
./erchans
Django has been used by many large organizations (Washington Post, etc.) and can connect with Postgresql easily enough. I use it fairly often and have had no trouble.
Yes. An ORM is essential for mapping SQL stuff to objects.
You have three choices.
Use someone else's ORM
Roll your own.
Try to execute low-level SQL queries and pick out the fields they want from the result set. This is -- actually -- a kind of ORM with the mappings scattered throughout the applications. It may be fast to execute and appear easy to develop, but it is a maintenance nightmare.
If you're designing the tables first, any ORM will be painful. For example, "composite primary key" is generally a bad idea, and with an ORM it's almost always a bad idea. You'll need to have a surrogate primary key. Then you can have all the composite keys with indexes you want. They just won't be "primary".
If you design the objects first, then work out tables that will implement the objects, the ORM will be pleasant, simple and will run quickly, also.
Alwasy new requirements are coming.
So what you really need is a framework that will allow you to adapt rapidly to changing specs.
From personal experience, I can only discuss django, which is great because it allows you to get up and running quickly.
If you stick to its ORM, you will have a pretty easy time getting your models fleshed out and connected in useful ways. You will need to familiarize yourself with a database migration tool, because Django does not have one built in. dmigrations seems to be a leading tool for this.
Another choice for ORM's is SQLAlchemy, which appears to be a bit more mature out of the box.
Depending on what you want to do, you actually have a few possible frameworks :
[Django] Big, strong (to the limit of what a python framework can be), and the older in the race. Used by a few 'big' sites around the world ([Django sites]). Still is a bit of an overkill for almost everything and with a deprecated coding approach.
[Turbogears] is a recent framework based on Pylons. Don't know much about it, but got many good feedbacks from friends who tried it.
[Pylons] ( which Turbogears2 is based on ). Often saw at the "PHP of Python" , it allow very quick developements from scratch. Even if it can seem inappropriate for big projects, it's often the faster and easier way to go.
The last option is [Zope] ( with or without Plone ), but Plone is way to slow, and Zope learning curve is way too long ( not even speaking in replacing the ZODB with an SQL connector ) so if you don't know the framework yet, just forget about it.
And yes, An ORM seem mandatory for a project of this size. For Django, you'll have to handle migration to their database models (don't know how hard it is to plug SQLAlchemy in Django). For turbogears and Pylons, the most suitable solution is [SQLAlchemy], which is actually the most complete ( and rising ) ORM for python. For zope ... well, nevermind
Last but not least, I'm not sure you're starting on a good basis for your project. 500 tables on any python framework would scare me to death. A boring but rigid language such as java (hibernate+spring+tapestry or so) seem really more appropriate.
Since most of your tables have composite primary keys, you'll want an ORM that supports that functionality. Django's default ORM does not support composite primary keys. SQLAlchemy does have that support (http://www.sqlalchemy.org/features.html).
The TurboGears framework uses SQLAlchemy as its default ORM. Pylons lets you use SQLAlchemy as well.
There are also ways to get Django to use SQLAlchemy, though I've not tried them myself. I prefer to use Django myself, but given your needs, I'd go with Pylons or TurboGears rather then shoe-horning a different ORM into the system.
For such horrid data-layer complexity as 500 tables with 1500 views, differently from most answers, I would personally prefer to stick with SQL (PostgreSQL offers a really excellent implementation thereof, expecially in the new 8.4 version which you should really lobby for if you have any chance); the only ORM I would [grudgingly] accept is SQLAlchemy (one of the few ORBs I don't really mind -- but the main added value is portability to different DBMS: if you're committed to just one, and in a project of this DB complexity you'd better be, then my personal opinion is that any ORM is just overhead, as the data-access layer developers will need deep familiarity with the specific DBMS to crawl towards acceptable performance).
Having picked "raw psycopg2" or SQLAlchemy as the technology for my data-access layer, that would rule out Django (which in my experience only works well with its own ORM -- but that's not suitable for a project of such DB complexity, IMNSHO). I'd go with Werkzeug, personally, as the framework most suitable for highly complex projects requiring ridiculous amounts of flexibility and power -- though Pylons and Turbogears 2 on top of it may be acceptable as a fall-back if the team just doesn't have the web app experience and skill it takes to make truly beautiful music with a flexible framework such as Werkzeug.
Last but not least, I'd strongly lobby for Dojo for the presentation layer on the client -- a rich and strongly structured Javascript framework, offering superbly designed functionality for "local data", host access, &c, optimized for the best that each of several browsers (and plug-ins such as Gears) can offer, as well as advanced UI functionality, seems likeliest to lighten the heavy development burden on the back-end team (in fact, I'd strongly recommend looking at offering an essentially RESTful interface on the server side, and delegate all presentation work to Dojo on the client -- see this site for more, except I'd be thinking of JSON rather than XML as the preferred interchange format). But, I'll readily admit to knowing far less about the UI side of things than about back-ends, business logic and overall architecture, so take this last paragraph for what it's worth!-)
There are many frameworks with different benefits and drawbacks. For big projects you need the framework to have a lot of third-party modules so you don't have to reinvent all the wheels, while the framework itself needs to be either completely overrideable so you can change even the frameworks behavior without having to resort to monkey-patching (because in complex projects, you sooner or later end up needing to modify the framework), or so minimalistic that it hardly does anything.
Minimalistic frameworks are for example Pylons, Repoze.BFG, and the latest interesting contender: Bobo (there are more, I'm sure).
Bigger frameworks that include more stuff are Django, Zope 3 and Grok, while Turbogears end up somewhere in the middle.
Zope 3 and Grok both build on the Zope Component Architecture, and the Zope Toolkit, which are a large set of components for making web applications. So are Bobo and BFG, I think, but since they are minimalistic they don't use many of the components. The benefit of the component architecture is that everything is components and you can override them with your own modified versions, so the systems are supremely customizable. The ZCA can be used with frameworks that aren't build on them as well, but of course the framework itself then isn't customizable.
I would absolutely recommend Repoze.bfg with SQLAlchemy for what you describe. I've done projects now in Django, TurboGears 1, TurboGears 2, Pylons, and dabbled in pure Zope3. BFG is far and away the framework most designed to accomodate a project growing in ways you don't anticipate at the beginning, but is far more lightweight and pared down than Grok or Zope 3. Also, the docs are the best technical docs of all of them, not the easiest, but the ones that answer the hard questions you're going to encounter the best. I'm currently doing a similar thing where we are overhauling a bunch of legacy databases into a new web deliverable app and we're using BFG, some Pylons, Zope 3 adapters, Genshi for templating, SQLAlchemy, and Dojo for the front end. We couldn't be happier with BFG, and it's working out great. BFGs classes as views that are actually zope multi-adapters is absolutely perfect for being able to override only very specific bits for certain domain resources. And the complete lack of magic globals anywhere makes testing and packaging the easiest we've had with any framework.
ymmv!