django/python imports perfomance

Question

Hello,

Can somebody please proof why it's a bad practice to use solution like this:

In django views in 98% cases you need to use

  from django.http import HttpResponseRedirect
  from django.core.urlresolvers import reverse
  from django.utils.translation import ugettext as _

anyway in my project my every view has these imports and everything is used almost in every second function of a view:

from datetime import datetime
from django.conf import settings
from django.contrib.auth.models import User
from django.core.urlresolvers import reverse
from django.core import paginator
from django.db import connection
from django.db.models import Q
from django.http import HttpResponseRedirect, Http404, HttpResponse
from django.shortcuts import render_to_response, get_object_or_404
from django.template import RequestContext
from django.utils.translation import ugettext as _

Now add some models and forms, and I have 50 lines of bullshit that is impossible to read at all.

First thing that came to my head is of course to make more views, to split some operation and etc and etc.. but still about 30 lines of imports killing my orientation in code.

Then I just decided to put everything that being used in views by 95% of a time, to directory /project/app/imports/view.py. Now I have all common stuff just with ONE import, but my co-worker attacked me, that it's highly hard to read this kind of code, because you can't see what is imported, and why the hell it's so hard to open one more tab in your IDE..??? [especially this goes to vim users, they have FRAMES, and he is using vim]

I did the same with models, my models has it's own dir, because it's over 50 them in there, and those files are not small - about 150 lines each.. Even these files have few models inside.. so I'm just doing something like :

from myapp.models.mymodel import *

and there are some places where I just doing: from myapp.models import * [init.py of myapp/imports dir takes place in here]

Problems:

1) ok so first problem is namespace, this kind of model importing is maybe really ridiculous.. but decision with views and forms, is just nothing but lazziness to open one more tab in your IDE

2) performance problem? my co-worker really arguing a lot with this argument, that "every import takes 256kb of ram"?? (by running compiled .pyc file? no i don't believe that ;)

The question in fact is about performance problem because of imports.

p.s. I'm really new in python (just 3 month), and I open to OBJECTIVE arguments for all probs and cons about this solution.

UPDATE

Once I asked question about how to move imports to standalone file so nobody complained about this =) question is here

Answer 1

1) it's nothing but laziness to not prefix your imported names with the module it came from. It's nothing but laziness to not be willing to scroll past the imports to the code. How exactly does having that mess of imports in another file make it any easier to read through? I would leave it in the original file where they are actually used. This improves readability because if I need to know where something came from, then I can just go to the top of the file and check it out (using the emacs mark ring to go right back). It also makes it easier to maintain the list because I just have to do a quick search to see where something is used (or not used).

2) On my machine, it takes ~812 microseconds to import a module.

$ python -mtimeit -s'import os' 'reload(os)'
1000 loops, best of 3: 808 usec per loop

This will of course vary greatly with where on your PYTHONPATH it is. If performance is that tight, you might be able to squeeze some out by juggling that around. YMMV.

I'm not sure where your coworker is getting the 256kb from. That would depend on the size of the code objects involved.

>>> import sys
>>> sys.getsizeof(sys)
24
>>> sys.getsizeof(sys.modules['__main__'])
24

As you can see, the actual module object only takes 24 bytes on my 32 bit machine. I have a feeling that will be system dependent though.

>>> def sizeofmodule(mod):
...     return sum(sys.getsizeof(getattr(mod, o)) for o in dir(mod))
... 
>>> sizeofmodule(itertools)
8662
>>> sizeofmodule(sys)
10275
>>> sizeofmodule(operator)
5230

Answer 2

Some points to consider:

The time it takes to import a module is, in almost all cases, completely irrelevant: it only happens once. Your Django view module isn't being imported and re-evaluated for every request; it's loaded once and then reused. If your modules are being reloaded constantly, something is catastrophically wrong.

Every import is not taking 256kb of memory. Perhaps each individual file, loaded once, is (though I'd doubt that as well), but importing the same file repeatedly is not taking 256kb each and every time; it's merely creating a reference. If memory use is in question, simply profile it--load 10000 of something and see how much memory is used.

For Django modules, you don't always need to create a directory for each; I import each model class from models/__init__.py (eg. from Customer import Customer), so I can say from myapp.models import Profile, Customer, Book, ....

---

The tendency for Django views to need a dozen lines of imports at the top really is a problem. It turns into boilerplate, code which you copy and paste every time you start a new file. Source code requiring boilerplate is a major flaw.

At the same time, I strongly advise against what some people might recommend: import django and then using fully-qualified module names. The result is typing things like django.core.urlresolvers.reverse. When you find yourself regularly copying and pasting function names because they're so long, something has gone wrong.

There's no single, clear, obviously-correct solution to this. Putting the things which are consistently used in another module is a valid solution, but there are legitimate problems with it: it's hard to see what's being imported, and where the results of the import are used.

You'll probably find fewer gut-reaction objections if you import the "collection of modules" module itself--import djangohelpers or import djangohelpers as dh. Then, you're writing things like dh.paginator. It gives the names a clear scope, and makes it much easier to see where it's being used and where particular function names are coming from, which you lose with "import *".

(You probably do want to import things like Q and _ as bare names, though.)

Answer 3

Keep in mind that you can import a set of subpackages. So

from django.conf import settings
from django.contrib.auth.models import User
from django.core.urlresolvers import reverse
from django.core import paginator
from django.db import connection
from django.db.models import Q
from django.http import HttpResponseRedirect, Http404, HttpResponse
from django.shortcuts import render_to_response, get_object_or_404
from django.template import RequestContext
from django.utils.translation import ugettext as _

can become

from django import conf, contrib, db, http, shortcuts, template, utils
from django.core import urlresolvers, paginator

which is brief, lets you avoid writing django everywhere, and leaves it pretty obvious where something like urlresolvers.reverse is coming from. This also has the advantage of not mapping general names like reverse to highly specific functionality, leaving you with more readable code.

Answer 4

Hello,

Suppose we have file called 'project/imports/views.py' with all imports mostly used in application views:

from datetime import datetime
import string 
from random import choice
from django.conf import settings
from django.contrib.auth.models import User
from django.contrib.auth.decorators import login_required, user_passes_test
from this author is using thom django.core.files.base import ContentFile
from django.core.urlresolvers import reverse
from django.core import paginator, serializers
from django.db import connection
from django.db.models import Q
from django.http import HttpResponseRedirect, Http404, HttpResponse, HttpResponseForbidden
from django.shortcuts import render_to_response, get_object_or_404
from django.template import loader, Context, RequestContext
from django.utils.translation import ugettext as _
from project.forms import *
from project.models import *
from project.lib import send_templated_mail, query_to_dict, apply_query, safe_template_context

If I create views file in application and just import all objects from this file like this (application/views.py):

from project.imports.views import *

It will not decrease project performance if I will use in one view only 20% of these objects, in another just 80%? And what will be if I will have 150-300 models (it's not joke), more than 20 forms objects. It will not affect any performance?