how to not spend a lot memory in django model manipulation

Question

Hello guys !

I have the following model:

model rank(Models.model):
    username = models.CharField(max_length=200, default=0)
    points = models.IntegerField(default=0)
    rank = models.IntegerFiel(default=0)

what i want is, based on 'points' recalc this model, setting the result in 'rank' field.

i=0
user_list = db.Rank.objects.get.all().order_by('-points')

for user in user_list:
   user.rank = i
   user.save()
   i += 1

this works fine, but whit 1800 users i spend 130 MB of MEMORY !!!! Webfaction want kills me :)

how can i do this, without spent a lot of memory ?

Answer 1

One way to do this would be to use a custom query to calculate rank on the fly. This way you are leaving rank calculation to the database. This tutorial has a nice explanation of how to go about it.

Here is the the query from the linked website modified to suit your model:

SELECT r1.username, r1.points, COUNT(r2.points) rank
FROM app_rank r1, app_rank r2 
WHERE r1.points <= r2.points or (r1.points = r2.points and r1.username = r2.username) 
GROUP BY r1.username, r1.points
ORDER BY r1.points DESC, r1.username DESC;

On a side note, you can re-write the loop without using the counter variable i.

for index, user in enumerate(user_list):
   user.rank = index + 1
   user.save()

Answer 2

The immediate cause of the memory leak is probably that you have DEBUG set to True. With debug on, Django keeps all the queries it runs in memory. Try it with DEBUG off and you should see usage go down considerably.