Mapping module imports in Python for easy refactoring

Question

I have a bunch of Python modules I want to clean up, reorganize and refactor (there's some duplicate code, some unused code ...), and I'm wondering if there's a tool to make a map of which module uses which other module.

Ideally, I'd like a map like this:

main.py
 -> task_runner.py
  -> task_utils.py
  -> deserialization.py
   -> file_utils.py
 -> server.py
  -> (deserialization.py)
  -> db_access.py

checkup_script.py
re_test.py
main_bkp0.py
unit_tests.py

... so that I could tell which files I can start moving around first (file_utils.py, db_access.py), which files are not used by my main.py and so could be deleted, etc. (I'm actually working with around 60 modules)

Writing a script that does this probably wouldn't be very complicated (though there are different syntaxes for import to handle), but I'd also expect that I'm not the first one to want to do this (and if someone made a tool for this, it might include other neat features such as telling me which classes and functions are probably not used).

Do you know of any tools (even simple scripts) that assist code reorganization?

Do you know of a more exact term for what I'm trying to do? Code reorganization?

Answer 1

Python's modulefinder does this. It is quite easy to write a script that will turn this information into an import graph (which you can render with e.g. graphviz): here's a clear explanation. There's also snakefood which does all the work for you (and using ASTs, too!)

You might want to look into pylint or pychecker for more general maintenance tasks.

Answer 2

Writing a script that does this probably wouldn't be very complicated (though there are different syntaxes for import to handle),

It's trivial. There's import and from module import. Two syntax to handle.

Do you know of a more exact term for what I'm trying to do? Code reorganization?

Design. It's called design. Yes, you're refactoring an existing design, but...

Rule One

Don't start a design effort with what you have. If you do, you'll only "nibble around the edges" making small and sometimes inconsequential changes.

Rule Two

Start a design effort with what you should have had if you'd only been smarter. Think broadly and clearly about what you're really supposed to be doing. Ignore what you did.

Rule Three

Design from the ground up (or de novo as some folks say) with the correct package and module architecture.

Create a separate project for this.

Rule Four

Test First. Write unit tests for your new architecture. If you have existing unit tests, copy them into the new project. Modify the imports to reflect the new architecture and rewrite the tests to express your glorious new simplification.

All the tests fail, because you haven't moved any code. That's a good thing.

Rule Five

Move code into the new structure last. Stop moving code when the tests pass.

You don't need to analyze imports to do this, BTW. You're just using grep to find modules and classes. The old imports and the tangled relationships among the old imports doesn't matter, and doesn't need to be analyzed. You're throwing it away. You don't need tools smarter than grep.

If feel an urge to move code, you must be very disciplined. (1) you must have test(s) which fail and then (2) you can move some code to pass the failing test(s).