In Django, getting a "Error: Unable to serialize database" when trying to dump data?

Question

Hello.

I'm getting an error when I'm trying to dump data to a JSON fixture in Djanog 1.2.1 on my live server. On the live server it's running MySQL Server version 5.0.77 and I imported a lot of data to my tables using the phpMyAdmin interface. The website works fine and Django admin responds as normal. But when I try and actually dump the data of the application that corresponds to the tables I get this error:

$ python manage.py dumpdata --indent=2 gigs > fixtures/gigs_100914.json 
/usr/local/lib/python2.6/site-packages/MySQLdb/__init__.py:34: DeprecationWarning: the sets module is deprecated
  from sets import ImmutableSet
Error: Unable to serialize database: Location matching query does not exist.

My Django model for 'gigs' that I'm trying to dump from looks like this in the models.py file:

from datetime import datetime
from django.db import models

class Location(models.Model):
    name = models.CharField(max_length=120, blank=True, null=True)

    class Meta:
        ordering = ['name']

    def __unicode__(self):
        return "%s (%s)" % (self.name, self.pk)

class Venue(models.Model):
    name = models.CharField(max_length=120, blank=True, null=True)
    contact = models.CharField(max_length=250, blank=True, null=True)
    url = models.URLField(max_length=60, verify_exists=False, blank=True, null=True) # because of single thread problems, I left this off (http://docs.djangoproject.com/en/dev/ref/models/fields/#django.db.models.URLField.verify_exists)

    class Meta:
        ordering = ['name']

    def __unicode__(self):
        return "%s (%s)" % (self.name, self.pk)

class Gig(models.Model):
    date = models.DateField(blank=True, null=True)
    details = models.CharField(max_length=250, blank=True, null=True)
    location = models.ForeignKey(Location)
    venue = models.ForeignKey(Venue)

    class Meta:
        get_latest_by = 'date'
        ordering = ['-date']

    def __unicode__(self):
        return u"%s on %s at %s" % (self.location.name, self.date, self.venue.name)

Like I say, Django is fine with the data. The site works fine and the relationships seem to operate absolutely fine. When a run the command to get what SQL Django is using:

$ python manage.py sql gigs
/usr/local/lib/python2.6/site-packages/MySQLdb/__init__.py:34: DeprecationWarning: the sets module is deprecated
  from sets import ImmutableSet
BEGIN;CREATE TABLE `gigs_location` (
    `id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY,
    `name` varchar(120)
)
;
CREATE TABLE `gigs_venue` (
    `id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY,
    `name` varchar(120),
    `contact` varchar(250),
    `url` varchar(60)
)
;
CREATE TABLE `gigs_gig` (
    `id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY,
    `date` date,
    `details` varchar(250),
    `location_id` integer NOT NULL,
    `venue_id` integer NOT NULL
)
;
ALTER TABLE `gigs_gig` ADD CONSTRAINT `venue_id_refs_id_3d901b6d` FOREIGN KEY (`venue_id`) REFERENCES `gigs_venue` (`id`);
ALTER TABLE `gigs_gig` ADD CONSTRAINT `location_id_refs_id_2f8d7a0` FOREIGN KEY (`location_id`) REFERENCES `gigs_location` (`id`);COMMIT;

I've triple checked the data, gone through to make sure all the relationships and data is ok after importing. But I'm still getting this error, three days on... I'm stuck with what to do about it. I can't imagine the "DeprecationWarning" is going to be a problem here. I really need to dump this data back out as JSON.

Many thanks for any help at all.

Answer 1

Could be something similar to this.

Run it with:

python manage.py dumpdata --indent=2 -v 2 --traceback gigs 

To see the underlying error.

Answer 2

I once ran in a similar problem where the error message was as mesmerizing as yours. The cause was a lack of memory on my server. It seems that generating dumps in json is quite memory expensive. I had only 60meg of memory (at djangohosting.ch) and it was not enough to get a dump for a mysql DB for which the mysql dump was only 1meg.

I was able to find out by watching the python process hit the 60meg limit using the top command in a second command line while running manage.py dumpdata in a first one.

My solution : get the mysql dump and then load it on my desktop pc, before generating the json dump. That said, for backup purposes, the mysql dumps are enough.

The command to get a mysql dump is the following :

mysqldump -p"password" -uusername database_name > dump_file_name.sql

That said, your problem could be completely different. You should really look at every table that has a foreign key to your Location table, and check if there is no field pointing to a previously deleted location. Unfortunately MySQL is very bad at maintaining Referential integrity, and you cannot count on it.