Caching of querysets and re-evaluation

Question

Hi,

I'm going to post some incomplete code to make the example simple. I'm running a recursive function to compute some metrics on a hierarchical structure.

class Category(models.Model):

parent = models.ForeignKey('self', null=True, blank=True, related_name='children', default=1)

def compute_metrics(self, shop_object, metric_queryset=None, rating_queryset=None)
    if(metric_queryset == None):
        metric_queryset = Metric.objects.all()
    if(rating_queryset == None):
        rating_queryset = Rating.objects.filter(shop_object=shop_object)

    for child in self.children.all():
        do stuff
        child_score = child.compute_metrics(shop_object, metric_queryset, rating_queryset)

    metrics_in_cat = metric_queryset.filter(category=self) 
    for metric in metrics_in_cat
          do stuff

I hope that's enough code to see what's going on. What I'm after here is a recursive function that is only going to run those queries once each, then pass the results down. That doesn't seem to be happening right now and it's killing performance. Were this PHP/MySQL (as much as I dislike them after working with Django!) I could just run the queries once and pass them down.

From what I understand of Django's querysets, they aren't going to be evaluated in my if queryset == None then queryset=stuff part. How can I force this? Will it be re-evaluated when I do things like metric_queryset.filter(category=self)?

I don't care about data freshness. I just want to read from the DB once for each of metrics and rating, then filter on them later without hitting the DB again. It's a frustrating problem that feels like it should have a very simple answer. Pickling looks like it could work but it's not very well explained in the Django documentation.

Answer 1

I think the problem here is you are not evaluating the queryset until after your recursive call. If you use list() to force the evaluation of the queryset then it should only hit the database once. Note you will have to change the metrics_in_cat line to a python level filter rather than using queryset filters.

parent = models.ForeignKey('self', null=True, blank=True, related_name='children', default=1)

def compute_metrics(self, shop_object, metric_queryset=None, rating_queryset=None)
    if(metric_queryset is None):
        metric_queryset = list([Metric.objects.all())
    if(rating_queryset is None):
        rating_queryset = list(Rating.objects.filter(shop_object=shop_object))

    for child in self.children.all():
        # do stuff
        child_score = child.compute_metrics(shop_object, metric_queryset, rating_queryset)

    # metrics_in_cat = metric_queryset.filter(category=self)
    metrics_in_cat = [m for m in metric_queryset if m.category==self]
    for metric in metrics_in_cat
        # do stuff