Best Practices for Architecting Large Systems in a Dynamic Language

Question

From my experiences building non-trivial applications in Java and C#, I know that using good modular design based on known patterns and "coding to interfaces" are keys to success.

What are the architecture best practices when building large systems in a dynamic language like python or ruby?

Answer 1

IMO, common practices are not depend on a language. You still need to separate your code, obey design principles like ISP, SOC, etc, etc. I've developed large system in Perl and can say that there was no major differences from C++/C# in term of high level design.

Dynamic language can simplify development, but will it have a strong influence on common design principles? I'm not sure. This topic is too large. Maybe you should select some pattern or practice and we will try to answer how to implement it in dynamic language in the best way?

Answer 2

From my experience most applications that were intended to be small grow over time. Therefore, it is important to develop code ready for further changes from the very beginning. I found it easier to extend applications when they are written without excessive faith in dynamic goodness. It is beyond all doubts required for the program to be written in "Object Oriented" manner, and both words are important here - some programs do use objects but they fail to follow the object orientation guidelines. Coding against interfaces is good practice, so avoiding excessive inheritance is. In PHP language I use type hinting in method signatures, e.g. public function setAsProfilePicture(Picture $p) to prevent silly mistakes (note the Picture class name preceeding $p parameter). This is particularly useful when you work as a team, as the code is getting self-documented (learn more on Coding without comments on Jeff Atwood's blog). Another useful technique you may employ is Programming by contract. It is sometimes worth being strict in dynamic languages as things may go wrong in many more ways than in static-typed languages. Returning null where an object should be returned (or passed) is not an uncommon situation and explicit result type checking may save you hours.

My bottom line would be: give away some of the freedom the dynamic languages offer, it may save you effort of finding bugs that are possible only in these languages.

Answer 3

Would it be fair to say the enforcement of coding to interfaces and programming by contract would not be in the code. Perhaps in the tests to ensure an object meets its operational requirements?

What of languages in which you can add methods at runtime, how does that enter the mix?

Answer 4

The best practice would probably be not to develop large systems in a dynamic language.

Update: the very fact that such question arises means that language choice could be better. Language is not all that important in a really big long term project - the correct way is to first choose your practices and architecture, and then choose language that suits them best.

Answer 5

@Jeff V,

What of languages in which you can add methods at runtime, how does that enter the mix?

From my practical experience good practice would be to not use obscure things such as addition of methods in runtime. For large systems, simplicity is one of the major keys to success. It's fine to use dynamic typing, lambda expressions, etc., but you should try to keep code clean, simple and understandable. As I said I was involved in development of pretty sophisticated system written in Perl. Main lesson I learned from this project is to fight with temptation of using of all those nice dynamic languages features and write code in more traditional way.

Answer 6

i've not seen or had experience with large systems in ruby, but there are some rules i learned for python development,

1. write to documented interfaces, ex. zope.interface
2. don't push into trunk without complete unit test coverage
3. make sure your development cycle includes code review and enforces the unit test rule, ex. divmod's UQDS

Answer 7

As others have replied, many architecture considerations (avoid tight coupling, provide adequate documentation, specify behavior clearly) are language independent. Several additional considerations are prominent for dynamic languages:

1. Aggressively unit test. This is really language independent as well, but many view this as especially important when flying without a static type checking copilot. The reason for emphasizing this here, however, is that with dynamic languages testing is so easy. Compared to many static languages, creating mocks/stubs, writing automated test suites, etc., is almost trivial. This means that the cost/benefit is overwhelmingly in favor of thorough testing.
2. Respect the language. Your design and implementation should work with the language, rather than against it (this also impacts how you choose the implementation language for a project). Building a custom reflection API in C++ is really painful, and writing code that relies heavily on type information in Python is not a good idea. Write code that, as much as possible, lives within the paradigm of the language. This way you leverage the incredible work that the language designers (and interpreter writers) have already done.
3. Be careful with magic. Dynamic languages offer extremely powerful (and often useful) facilities for things like metaclass programming, dynamic class modification, decorator tricks, and so on. There's no need to be afraid of these outright, but don't use them unless you need them. Why? The are difficult to get right, and require a lot of expertise as a result. This means that they are harder to write, harder to maintain, and can be a source of subtle bugs. (Google monkeypatching or look up Jeff Atwood's article on this).

Answer 8

I have built several large systems in dynamic languages including Smalltalk and Ruby. The most important part of building such a system is to clearly define what the system needs to do. In modern development methods that means BDD as in rspec, but it can also mean a clear statement of what the system should do.

Once you know what it should do the next thing is to allocate that what to classes (the who). That allocation needs to be explicit. This can be a class comment up to BDD specs.

The only place where dynamic languages differ from other systems is that you have the option of letting a class satisfy more than one expectation with minimal effort. To take advantage of this the expectations should be smaller and the classes combine them, rather than having the "interfaces" be monolithic and exclusionary. Try to break up the problem into very small units of behavior/data and then combine them to make actual classes. For example in a mail system the minimal aspects of a mail attachment might be to return a B64 encoding and a mime type. It would be tempting to place that logic in a message class, but placing it on the attachment would allow anything to meet that expectation. The more flexibiilty you introduce by using modular expectations and not assuming an object has a huge set of behaivors, the more adaptable the system will be over time.

Adding dynamic behavior to a class/object is fine if it is apparent to the reader that it is happening, and doing so does not violate a reader's intuition. For example if every object in a collection received special behavior by virtue of being in the collection and only the collection used that behavior it would be fine. But, if the behaivor became relied upon by other parts of the system that were not directly tied to the collection, then you are in for a world of hurt.

Dynamic systems key advantage is that the programmer can make the rules on when an object can be suitable for a purpose. The best way to use such systems is to make those rules visible and clear to any reader/weriter within the system.

Answer 9

Take advantage of the dynamism of the language to save development time and add lots of clever and useful features. Use the best algorithms so performance problems are minimized.

Answer 10

RSWF is a fairly complex project written in Ruby (there are a few hundred lines of C for speed, but that is just an alternate version of Ruby code) from which I learned quite a lot. Here are three things that come to mind when I look back on my experiences with it over the past two or three years:

1. Comprehensive automated test suites are about the most important thing you can have when working in a dynamic language. This lets you make small changes without fear, and enables you to make large changes. RSWF has had two massive revisions where I changed the internal model of how the system worked and touched pretty much everything in one huge commit; I would never have attempted something like this without a large suite of unit and functional tests.*

2. Metaprogramming is good. Ruby, for example, lets you do a lot of clever domain-specific-language-type stuff that you can't do in Java: take advantage of this! RSWF was particularly suitable to this, but I reckon my code is about a quarter of the linecount I would have had in Java, and that translates directly into dollars and time saved on maintenance and extensions.

3. One particular thing I've noticed with this project over time is that I wish I'd started out using a lot more immutable objects (or "value objects," as the OO guys call them). Most changes I make these days happen to turn a few mutable objects into immutable ones.

(* In case it's of interest, within the library itself about 30% of the code is unit tests, and the code for the functional test suite is about 20-25% of the size of the library.)

Answer 11

If its a really large or even moderately large project that is going to involve several people, go back to Java or your other typed language of choice.

As time progresses there will be needs to refactor and a necessity for help like intellisense, both of which arent really possible with dynamic languages. I wont elaborate on when to select a static as opposed to a dynamic language. The smaller the task the more sense it makes to use a dynamic language - ie you dont want to stuff around declaring types - the task is simple and you just want to get it done. However the more complete the problem, this aide and many others become more evident and necessarily.

Perhaps some interesting points of consider include:

• Why is there a predominance of statically vs dynamic type out there in the big world ?
• Why did Microsoft concentrate on delivering the CLR as a statically typed platform first ?

Answer 12

These are a few simple ground rules which should get you started. Please note that the vast majority of them are true for all large systems, regardless of the language used.

• Unit tests. Lots and lots of unit tests. This way, you can always be certain that you did not break any of the functionality. In addition, the unit tests are your safety net in which you must perform type-safety checks (since the language does not do it for you)
• Do not be tempted to over-use the 'dynamic' features of the language. Decorators are fine, hot-patching class methods during runtime is not. If it feels "dirty", it probably is.
• Do not use the same variable for two different kinds of data.

Code like:

if x > 5:
    y = 5
else:
    y = "foo"

will quickly become unmaintainable.

• Always clearly document what you are doing. This is even more important in a language where types are not easily inferred.
• Sanitize any and all function parameters. Since you are not constrained by types, it's easy to inadvertently break code - and since this is a runtime constraint violation, you will only know about it when it's too late
• Adopt and use a lint tool. It will help you find most common mistakes. For Python, there's PyLint.
• Finally, don't be afraid. It will not blow up in your face.

We have a lot of Python-based systems, and these rules have kept us running safe and sane. Good luck.

Answer 13

Here's what I have found to be most important:

1. Modularity
2. Documented interfaces
3. Whenever you want to use a complicated type, implement it as an object (rather than, for example, a list containing different data types in each element). That's an easy way to accomplish the above two points.