My django application takes forever to load so I'd like to find a way to measure the import latencies so I can find the offending module and make it load lazily.
Is there an obvious way to do this? I was considering tweaking the import statement itself to produce the latencies but I'm not sure exactly how to do that. I guess that it would be ideal to have the import tree (or DAG?) along with the latencies but just a list of latencies and import statements would be enough.
You can redefine the __import__ built-in function to log the start and end times (delegating everything else to the original built-in __import__). That's exactly the way to "tweak the import statement" in Python: redefine __import__!
Edit: here's a simple example...:
import sys
import __builtin__
_orgimp = __builtin__.__import__
import logging
FORMAT = "%(asctime)-15s %(message)s"
logging.basicConfig(format=FORMAT, level=logging.INFO)
def __import__(name, *a):
r = sys.modules.get(name)
if r is not None: return r
logging.info('import bgn %s', name)
r = _orgimp(name, *a)
logging.info('import end %s', name)
return r
__builtin__.__import__ = __import__
import pyparsing
This shows, on my system:
2010-08-24 08:36:39,649 import bgn pyparsing
2010-08-24 08:36:39,652 import bgn weakref
2010-08-24 08:36:39,652 import bgn _weakref
2010-08-24 08:36:39,653 import end _weakref
2010-08-24 08:36:39,653 import end weakref
2010-08-24 08:36:39,654 import bgn copy
2010-08-24 08:36:39,655 import bgn org.python.core
2010-08-24 08:36:39,656 import end copy
2010-08-24 08:36:39,675 import end pyparsing
Of course you can parse this log to show the nesting (what module first-imported what other modules) and "attempted imports" that failed (here, that of org.python.core from copy, no doubt in a try/except statement) as well as the time at which each import begins and concludes (the latter only if it does conclude, of course, not if it fails -- easy to tweak with try/finally or whatever to get the exact behavior you prefer!-).