The source code behind EveryBlock.com, a major Django-powered website founded by Adrian Holovaty, one of the co-Benevolent Dictators For Life of the Django framework, was recently open-sourced. The source is available as tarballs and on github.
This large body of code from an originator of Django should have some interesting features, patterns, tricks, or techniques. What is your favorite?
The massive regular expression monster they use to geocode locations from English text. Sentences from news stories like "Shooting was heard in the area East of 3rd between Locust and Pine St" will generate a PostGIS polygon that they then match against the users' locations and layer onto maps. It is a great site, I have been excited about this but hadn't seen the announcement til now.
Very cool that they released the source. It's a nice bit of code and I think there is something to learn from checking it out.
I'm most interested in the geo/mapping functionality, but I also find the scraper routines used to harvest public data from various public sources quite interesting.
Some of the things that I noticed:
The publishing system ebpub uses custom django Authentication and user system, hence cannot use django-admin.
Altho' it uses Relational Database PostgreSQL, the data items for various data entries are stored in a single table, with types of fields defined in another table, for scalability. (An alternative to key-value pair storing systems, CouchDB)
The system uses custom database backend, so that such a modified form database can accessed with convenience in views.
The blog application is very small and sweet; Just 1 Entry model and no views, Only generic views.
Some of the bots present could be used for multiple purposes, with little tweaking.
I see that they have one scraper per site per city.
I found this to be too much work as there are better way of using one generic scraper with various directives which will be site specific. This is the way I have written my scraper for newjoblist.com
I like the look & feel for sure.
Map is too washed out
I think the source is good to look at and learn what to do or not do. A great reference.