How to properly organize a package/module dependency tree ?

Question

Good morning,

I am currently writing a python library. At the moment, modules and classes are deployed in an unorganized way, with no reasoned design. As I approach a more official release, I would like to reorganize classes and modules so that they have a better overall design. I drew a diagram of the import dependencies, and I was planning to aggregate classes by layer level. Also, I was considering some modification to the classes so to reduce these dependencies.

What is your strategy for a good overall design of a potentially complex and in-the-making python library? Do you have interesting suggestions ?

Thanks

Update:

I was indeed looking for a rule of thumb. For example, suppose this case happens (init.py removed for clarity)

foo/bar/a.py
foo/bar/b.py
foo/hello/c.py
foo/hello/d.py

now, if you happen to have d.py importing bar.b and a.py importing hello.c, I would consider this a bad setting. Another case would be

foo/bar/a.py
foo/bar/baz/b.py
foo/bar/baz/c.py

suppose that both a.py and b.py import c. you have three solutions: 1) b imports c, a import baz.c 2) you move c in foo/bar. a.py imports c, b.py imports .c 3) you move c somewhere else (say foo/cpackage/c.py) and then both a and b import cpackage.c

I tend to prefer 3), but if c.py has no meaning as a standalone module, for example because you want to keep it "private" into the bar package, I would preferentially go for 1).

There are many other similar cases. My rule of thumb is to reduce the number of dependencies and crossings at a minimum, so to prevent a highly branched, highly interweaved setup, but I could be wrong.

Answer 1

The question is very vague.

You can achieve this by having base/core things that import nothing from the remainder of the library, and concrete implementations importing from here. Apart from "don't have two modules importing from each-other at import-time", you should be fine.

module1.py:

import module2

module2.py:

import module1

This won't work!

Answer 2

"I drew a diagram of the import dependencies, and I was planning to aggregate classes by layer level."

Python must read like English (or any other natural language.)

An import is a first-class statement that should have real meaning. Organizing things by "layer level" (whatever that is) should be clear, meaningful and obvious.

Do not make arbitrary technical groupings of classes into modules and modules into packages.

Make the modules and package obvious and logical so that the list of imports is obvious, simple and logical.

"Also, I was considering some modification to the classes so to reduce these dependencies."

Reducing the dependencies sounds technical and arbitrary. It may not be, but it sounds that way. Without actual examples, it's impossible to say.

Your goal is clarity.

Also, the module and package are the stand-alone units of reuse. (Not classes; a class, but itself isn't usually reusable.) Your dependency tree should reflect this. You're aiming for modules that can be imported neatly and cleanly into your application.

If you have many closely-related modules (or alternative implementations) then packages can be used, but used sparingly. The Python libraries are relatively flat; and there's some wisdom in that.

---

Edit

One-way dependency between layers is an essential feature. This is more about proper software design than it is about Python. You should (1) design in layers, (2) design so that the dependencies are very strict between the layers, and then (3) implement that in Python.

The packages may not necessarily fit your layering precisely. The packages may physically be a flat list of directories with the dependencies expressed only via import statements.

Answer 3

It depends on the project, right? For example, if you are using a model-view-controller design, then your package would be structured in a way that makes the 3 groups of code independent.

If you need some ideas, open up your site-packages directory, and look through some of the code in those modules to see how they are set up.

There is no correct way without knowing more about the module; as Ali said, this is a vague question. You really just need to analyze what you have in front of you, and figure out what might work better.