A better way to loop through multiple lists in a django template

Question

I have some big data sets that I am looping through to display a table of data. The trouble is the looping takes a ton of time, which is okay at the moment as this is an internal tool but I would like to improve it.

The model:

class Metric_Data(models.Model):
  metric = models.ForeignKey(Metric)
  count = models.IntegerField()
  start_date = models.DateField()

I am displaying a table where the first column is the dates then each following column is a metric listing the count for that date. Like so:

Dates Metric Metric Metric ...
10/11     10     11     12
11/11     22    100   1000
...      ...    ...    ...

I tried looping over the data in the view and creating the table out of lists and passing this to the template for rendering but with several metrics and thousands of data points per metric this was rather slow. I have since switched to a template tag:

def getIndex(parser, token):
  try:
    tag_name, a_list, index = token.split_contents()
  except ValueError:
    raise template.TemplateSyntaxError, "%r tag requires exactly two arguments" % token.contents.split()[0]
  return GetIndexNode(a_list, index)

class GetIndexNode(template.Node):
  def __init__(self, a_list, index):
    self.the_list = template.Variable(a_list)
    self.index = template.Variable(index)

  def render(self, context):
    try:
      the_list = self.the_list.resolve(context)
      i = self.index.resolve(context)
      return the_list[i]
    except template.VariableDoesNotExist:
      return ''

Which is still rather slow, which just could because it is my first time writing a template tag and I've done something wrong.

EDIT: I am fetching the data in the view like so:

def show_all(request):
  metrics = Metric.objects.all()
  dates = Metric_Data.objects.all().values_list('start_date',flat=True).distinct().order_by('start_date')

  data = []

  for metric in metrics:
    data.append(Metric_Data.objects.filter(metric=metric).order_by('start_date').values_list('count', flat=True))

  return render_to_response('metric/show_all.html', {'dates': dates,
                                                   'metrics': metrics,
                                                   'data': data})

Edit: And the template

<table id="theTable" class="paginate-5">
<thead> 
<tr>
<th>Dates</th>
 {% for metric in metrics %}
  <th>{{ metric.name }}</th>
 {% endfor %}
</tr>
</thead>
<tbody> 
{% for date in dates %}
 <tr>
 <td>{{date}}</td>
 {% for metric in data %}
  <td>{% get_index metric forloop.parentloop.counter0 %}</td>
 {% endfor %}
 </tr>
{% endfor %}
</tbody>

I am thinking the best place to fix this problem might be in the model but I'm not sure how to go about it. Create a table for the dates perhaps and do the query on that table?

Ideas much appreciated thanks!

Answer 1

If you find your view to be slow, the problem is often in the database. Are you sure you know what queries are going to the database? It's possible you can make a small change that would greatly reduce the db traffic.