views:

1140

answers:

3

I was wondering if the following migration is possible with Django south and still retain data.

Before:

I currently have two apps, one called tv, one called movies, each with a VideoFile model (simplified here):

tv/models.py:

class VideoFile(models.Model):
    show = models.ForeignKey(Show, blank=True, null=True)
    name = models.CharField(max_length=1024, blank=True)
    size = models.IntegerField(blank=True, null=True)
    ctime = models.DateTimeField(blank=True, null=True)

movies/models.py:

class VideoFile(models.Model):
    movie = models.ForeignKey(Movie, blank=True, null=True)
    name = models.CharField(max_length=1024, blank=True)
    size = models.IntegerField(blank=True, null=True)
    ctime = models.DateTimeField(blank=True, null=True)

After:

Because the two videofile objects are so similar I want to get rid of duplication and create a new model in a separate app called media that contains a generic VideoFile class and use inheritance to extend it:

media/models.py:

class VideoFile(models.Model):
    name = models.CharField(max_length=1024, blank=True)
    size = models.IntegerField(blank=True, null=True)
    ctime = models.DateTimeField(blank=True, null=True)

tv/models.py:

class VideoFile(media.models.VideoFile):
    show = models.ForeignKey(Show, blank=True, null=True)

movies/models.py:

class VideoFile(media.models.VideoFile):
    movie = models.ForeignKey(Movie, blank=True, null=True)

So my question is, how can I accomplish this with django-south and still maintain existing data?

All three these apps are already managed by south migrations and according to the south documentation it is bad practice to combine a schema and data migration and they recommend it should be done in a few steps.

I think it could be done using separate migrations like this (assuming media.VideoFile is already created)

  1. Schema migration to rename all fields in tv.VideoFile and movies.VideoFile that will move to the new media.VideoFile model, maybe to something like old_name, old_size, etc
  2. Schema migration to tv.VideoFile and movies.VideoFile to inherit from media.VideoFile
  3. Data migration to copy old_name to name, old_size to size, etc
  4. Scheme migration to remove old_ fields

Before I go through all that work, do you think that will work? Is there a better way?

If you're interested, the project is hosted here: http://code.google.com/p/medianav/

A: 

Abstract Model

class VideoFile(models.Model):
    name = models.CharField(max_length=1024, blank=True)
    size = models.IntegerField(blank=True, null=True)
    ctime = models.DateTimeField(blank=True, null=True)
    class Meta:
        abstract = True

May be generic relation will be useful for you too.

Oduvan
If I define the parent class as an abstract model, can I then skip the whole migration process? Will south still be in sync?
Andre Miller
Abstract Model doesn't sync. Only it's "children".
Oduvan
O.. sorry, I doesn't use south before.
Oduvan
I can see how I can do it manually, but I already have an existing user base and would like to use south to automate the migration and not have them lose any data.
Andre Miller
If you use abstract inheritance like this, I don't think you need any migration at all. So that's a plus. The negative side is that you still have duplication at the database level, and you can't query all VideoFiles at once.
Carl Meyer
+1  A: 

I did a similar migration and I chose to do it in multiple steps. In addition to creating the multiple migrations, I also created the backward migration to provide a fallback if things went wrong. Then, I grabbed some test data and migrated it forward and backwards until I was sure it was coming out correctly when I migrated forwards. Finally, I migrated the production site.

scompt.com
Do you have an example of what your migrations looked like? How did you copy the data from the old to the new schema?
Andre Miller
@Andre have you looked at the South docs for data migrations? It's pretty much just like using the ORM normally, except you do it through the "orm" parameter passed to your backwards/forwards method (so you'll always have the right version of your models for running that migration, no matter the current state of your models).
Carl Meyer
I did see it and have played with it. I was just wondering if the steps I mentioned above, renaming the fields first so there isn't a clash is the simplest way of doing it.
Andre Miller
+16  A: 

This seemed like an interesting problem, and I'm becoming a big fan of South, so I decided to look into this a bit. I built a test project on the abstract of what you've described above, and have successfully used South to perform the migration you are asking about. Here's a couple of notes before we get to the code:

  • The South documentation recommends doing schema migrations and data migrations separate. I've followed suit in this.

  • On the backend, Django represents an inherited table by automatically creating a OneToOne field on the inheriting model

  • Understanding this, our South migration needs to properly handle the OneToOne field manually, however, in experimenting with this it seems that South (or perhaps Django itself) cannot create a OneToOne filed on multiple inherited tables with the same name. Because of this, I renamed each child-table in the movies/tv app to be respective to it's own app (ie. MovieVideoFile/ShowVideoFile).

  • In playing with the actual data migration code, it seems South prefers to create the OneToOne field first, and then assign data to it. Assigning data to the OneToOne field during creation cause South to choke. (A fair compromise for all the coolness that is South).

So having said all that, I tried to keep a log of the console commands being issued. I'll interject commentary where necessary. The final code is at the bottom.

Command History

django-admin.py startproject southtest
manage.py startapp movies
manage.py startapp tv
manage.py syncdb
manage.py startmigration movies --initial
manage.py startmigration tv --initial
manage.py migrate
manage.py shell          # added some fake data...
manage.py startapp media
manage.py startmigration media --initial
manage.py migrate
# edited code, wrote new models, but left old ones intact
manage.py startmigration movies unified-videofile --auto
# create a new (blank) migration to hand-write data migration
manage.py startmigration movies videofile-to-movievideofile-data 
manage.py migrate
# edited code, wrote new models, but left old ones intact
manage.py startmigration tv unified-videofile --auto
# create a new (blank) migration to hand-write data migration
manage.py startmigration tv videofile-to-movievideofile-data
manage.py migrate
# removed old VideoFile model from apps
manage.py startmigration movies removed-videofile --auto
manage.py startmigration tv removed-videofile --auto
manage.py migrate

For space sake, and since the models invariably look the same in the end, I'm only going to demonstrate with 'movies' app.

movies/models.py

from django.db import models
from media.models import VideoFile as BaseVideoFile

# This model remains until the last migration, which deletes 
# it from the schema.  Note the name conflict with media.models
class VideoFile(models.Model):
    movie = models.ForeignKey(Movie, blank=True, null=True)
    name = models.CharField(max_length=1024, blank=True)
    size = models.IntegerField(blank=True, null=True)
    ctime = models.DateTimeField(blank=True, null=True)

class MovieVideoFile(BaseVideoFile):
    movie = models.ForeignKey(Movie, blank=True, null=True, related_name='shows')

movies/migrations/0002_unified-videofile.py (schema migration)

from south.db import db
from django.db import models
from movies.models import *

class Migration:

    def forwards(self, orm):

        # Adding model 'MovieVideoFile'
        db.create_table('movies_movievideofile', (
            ('videofile_ptr', orm['movies.movievideofile:videofile_ptr']),
            ('movie', orm['movies.movievideofile:movie']),
        ))
        db.send_create_signal('movies', ['MovieVideoFile'])

    def backwards(self, orm):

        # Deleting model 'MovieVideoFile'
        db.delete_table('movies_movievideofile')

movies/migration/0003_vidoefile-to-movievidoefile-data.py (data migration)

from south.db import db
from django.db import models
from movies.models import *

class Migration:

    def forwards(self, orm):
        for movie in orm['movies.videofile'].objects.all():
            new_movie = orm.MovieVideoFile.objects.create(movie = movie.movie,)
            new_movie.videofile_ptr = orm['media.VideoFile'].objects.create()

            # videofile_ptr must be created first before values can be assigned
            new_movie.videofile_ptr.name = movie.name
            new_movie.videofile_ptr.size = movie.size
            new_movie.videofile_ptr.ctime = movie.ctime
            new_movie.videofile_ptr.save()

    def backwards(self, orm):
        print 'No Backwards'

South is awesome!

Ok standard disclaimer: You're dealing with live data. I've given you working code here, but please use the --db-dry-run to test your schema. Always make a backup before trying anything, and generally be careful.

COMPATIBILITY NOTICE

I'm going to keep my original message intact, but South has since changed the command manage.py startmigration into manage.py schemamigration.

T. Stone
Thank you very much for all the work you've put into this. If I could vote more than once I would! I think I'll take your advice and name the models different to avoid name clashes in the future too. This is fantastic.
Andre Miller
I had to modify your example slightly as the new_movie model instance did not get saved (perhaps there is a .save() missing?) - anyway, I got it working, the result here: http://code.google.com/p/medianav/source/detail?r=99
Andre Miller
A couple things: 1) `startmigration` was actually split into `schemamigration` and `datamigration`. The latter doesn't take the `--auto` or `--initial` flags, just the app name and the migration name, and gives you a migration file with empty `forwards` and `backwards` methods. 2) Instead of `print 'No Backwards'`, you should use `raise RuntimeError("Cannot reverse this migration.")` per the South docs.
Mike DeSimone
@Mike good suggestions. Another compatibility item that has since changed is that `South` no longer lets you put dashes `-` in migration names, only underscores.
T. Stone
Why didn't you use `class Meta: abstract = True` ?
muudscope
@muddscope - Based on the project, it seemed like the OP wanted the ability to query all `VideoFile` objects generically.
T. Stone