How to perform this sql in django model?

Question

SELECT *, SUM( cardtype.price - cardtype.cost ) AS profit  
FROM user  
LEFT OUTER JOIN  card ON (  user.id =  card.buyer_id )  
LEFT OUTER JOIN  cardtype ON (  card.cardtype_id =  cardtype.id )  
GROUP BY user.id  
ORDER BY profit DESC

I tried this:

User.objects.extra(select=dict(profit='SUM(cardtype.price-cardtype.cost)')).annotate(sum=Sum('card__cardtype__price')).order_by('-profit')

But Django automatically added SUM( cardtype.price ) to the GROUP BY clause, and the SQL doesn't run.

Can this be done without raw SQLs?

---

Provide the model, never mind these Chinese characters :)

class User(models.Model):

    class Meta:
        verbose_name = "用户"
        verbose_name_plural = "用户"
        ordering = ['-regtime']

    user_status= (
        ("normal", "正常"),
        ("deregistered", "注销"),
        ("locked", "锁定"),
    )

    name = models.CharField("姓名", max_length=20, db_index=True)
    spec_class = models.ForeignKey(SpecClass, verbose_name="专业班级")
    idcard = models.CharField("身份证号", max_length=18)
    mobileno = models.CharField("手机号", max_length=11)
    password = models.CharField("密码", max_length=50) # plain
    address = models.CharField("住址", max_length=100)
    comment = models.TextField("备注")
    certserial = models.CharField("客户证书序列号", max_length=100)
    regtime = models.DateTimeField("注册时间", default=datetime.datetime.now)
    lastpaytime = models.DateTimeField("上次付款时间", default=datetime.datetime.now)
    credit = models.FloatField("信用额度", default=100)
    money = models.FloatField("余额", default=0)
    use_password = models.BooleanField("使用密码")
    use_fetion = models.BooleanField("接收飞信提示")
    status = models.CharField("账户状态", choices = user_status, default="normal", max_length=20, db_index=True)

    def __unicode__(self):
        return self.name

class CardType(models.Model):

    class Meta:
        verbose_name = "点卡类型"
        verbose_name_plural = "点卡类型"
        ordering = ['name']

    name = models.CharField("类型名称", max_length=20, db_index=True)
    note = models.CharField("说明", max_length=100)
    offcial = models.BooleanField("官方卡", default=True)
    available = models.BooleanField("可用", default=True, db_index=True)
    payurl = models.CharField("充值地址", max_length=200)
    price = models.FloatField("价格")
    cost = models.FloatField("进货价格")

    def __unicode__(self):
        return u"%s(%.2f元%s)" % (self.name, self.price, u", 平台卡" if not self.offcial else "")

    def profit(self):
        return self.price - self.cost
    profit.short_description = "利润"


class Card(models.Model):

    class Meta:
        verbose_name = "点卡"
        verbose_name_plural = "点卡"
        ordering = ['-createtime']

    card_status = (
        ("instock", "未上架"),
        ("available", "可用"),
        ("sold", "已购买"),
        ("invalid", "作废"),
        ("returned", "退卡"), # sell to the same person !
        ("reselled", "退卡重新售出"),
    )

    cardtype = models.ForeignKey(CardType, verbose_name="点卡类型")
    serial = models.CharField("卡号", max_length=40)
    password = models.CharField("卡密", max_length=20)
    status = models.CharField("状态", choices = card_status, default="instock", max_length=20, db_index=True)
    createtime = models.DateTimeField("入库时间")
    buytime = models.DateTimeField("购买时间", blank=True, null=True)
    buyer = models.ForeignKey(User, blank=True, null=True, verbose_name="买家")

    def __unicode__(self):
        return u'%s[%s]' % (self.cardtype.name, self.serial)

Answer 1

Well, I found this
http://stackoverflow.com/questions/2424471/sum-computed-column-in-django-queryset

Have to use raw SQL now...
Thank you two!

PS: This is my first post on stackoverflow, hope I didn't break any rules.

Answer 2

First, one of the outer joins appears to be a bad idea for this kind of thing. Since you provided no information on your model, I can only guess.

Are you saying that you may not have a CARD for each user? That makes some sense.

Are you also saying that some cards don't have card types? That doesn't often make sense. You haven't provided any details. However, if a Card doesn't have a Card Type, I'll bet you have either problems elsewhere in your application, or you've chosen really poor names that don't provide the least clue as to what these things mean. You should fix the other parts of your application to assure that each card actually does have a card type. Or you should fix your names to be meaningful.

Clearly, the ORM statement uses inner joins and your SQL uses outer joins. What's the real question? How to do outer joins correctly? If you take the time to search for [Django] and Left Outer Join, you'll see that the Raw SQL is a terrible idea.

Or is the real question how to do the sum correctly? From your own answer it appears that the SQL is wrong and you're really having trouble with the sum. If so, please clean up the SQL to be correct.

If the outer joins are part of the problem -- not just visual noise -- then you have to do something like this for an outer join with a sum.

def user_profit():
    for u in User.objects.all():
        profit = sum[ t.price - t.cost
            for c in u.card_set.all()
                for t in c.cardtype_set.all() ]
        yield user, profit

In your view function, you can then provide the value of function to the template to render the report. Since it's a generator, no huge list is created in memory. If you need to paginate, you can provide the generator to the paginator and everything works out reasonably well.

This is often of comparable speed to a complex raw SQL query with a lot of outer joins.

If, indeed, the card to card-type relationship is not actually optional, then you can shorten this, somewhat. You still have an outer join to think about.

def user_profit():
    for u in User.objects.all():
        profit = sum[ c.cardtype.price - c.cardtype.cost
            for c in u.card_set.all() ]
        yield user, profit