Data Migrations

Follow the Ana Lopez rows from left to right in the picture. Her name moves to author_name, an empty author_id column appears next to it, the data migration fills that column with 1, and the name column disappears at the end.

Setting Up a Practice Copy

Your blog has already made this change, so practice on a copy. Copy the whole project folder, for example mysite to mysite-practice, and run every command in this lesson inside the copy. In the copy, delete db.sqlite3 and every file in blog/migrations/ whose number is 0004 or higher. Then replace blog/models.py with the Article model from the end of the lesson on models and migrations, where author is a CharField(max_length=100), so the file holds only that class. Run python manage.py migrate to build a new database. Finally, create three articles in python manage.py shell:

from blog.models import Article

Article.objects.create(title="Getting Started with Django", content="Install Django.", author="Ana Lopez")
Article.objects.create(title="Django Models in Practice", content="Fields map to columns.", author="Ana Lopez", view_count=45)
Article.objects.create(title="Understanding Querysets", content="Querysets are lazy.", author="Ben Carter", view_count=120)

Steps 1 and 2: Rename the Old Field and Add the New One

In blog/models.py, rename the field author to author_name and run makemigrations. Both fields are CharFields, so Django asks the rename question from the lesson on migrations. Answer y:

Was article.author renamed to article.author_name (a CharField)? [y/N] y
Migrations for 'blog':
  blog/migrations/0004_rename_author_article_author_name.py
    ~ Rename field author on article to author_name

The SQL from sqlmigrate blog 0004 is one statement, ALTER TABLE "blog_article" RENAME COLUMN "author" TO "author_name";, so every name stays in place. Run migrate, and then add the Author model and the new foreign key:

# blog/models.py
from django.db import models
from django.utils import timezone


class Author(models.Model):
    name = models.CharField(max_length=100)
    email = models.EmailField(unique=True)

    def __str__(self):
        return self.name


class Article(models.Model):
    title = models.CharField(max_length=200)
    content = models.TextField()
    author_name = models.CharField(max_length=100)
    author = models.ForeignKey(Author, on_delete=models.CASCADE, null=True)
    published_date = models.DateTimeField(default=timezone.now)
    view_count = models.PositiveIntegerField(default=0)

    def __str__(self):
        return self.title

The null=True is temporary. The articles exist before any author does, so the new column has to accept an empty value until the data migration fills it. Run makemigrations and migrate again:

Migrations for 'blog':
  blog/migrations/0005_author_article_author.py
    + Create model Author
    + Add field author to article

Each article now has both fields, and only the old one holds data:

from blog.models import Article

for article in Article.objects.all():
    print(article.id, article.author_name, article.author_id)

1 Ana Lopez None
2 Ana Lopez None
3 Ben Carter None

Step 3: Write the Data Migration

A data migration starts as an empty migration file. The --empty option asks makemigrations for one, and --name gives it a readable name:

python manage.py makemigrations blog --empty --name copy_author_names

Migrations for 'blog':
  blog/migrations/0006_copy_author_names.py

Django fills in the dependencies, so this migration runs after 0005, and leaves operations empty. Replace the file's contents with this:

# blog/migrations/0006_copy_author_names.py
from django.db import migrations
from django.utils.text import slugify


def names_to_authors(apps, schema_editor):
    Article = apps.get_model("blog", "Article")
    Author = apps.get_model("blog", "Author")
    authors = {}
    for article in Article.objects.all():
        name = article.author_name
        if name not in authors:
            email = f"{slugify(name)}@example.com"
            authors[name] = Author.objects.create(name=name, email=email)
        article.author = authors[name]
        article.save(update_fields=["author"])


def remove_authors(apps, schema_editor):
    Article = apps.get_model("blog", "Article")
    Author = apps.get_model("blog", "Author")
    Article.objects.update(author=None)
    Author.objects.all().delete()


class Migration(migrations.Migration):

    dependencies = [
        ("blog", "0005_author_article_author"),
    ]

    operations = [
        migrations.RunPython(names_to_authors, remove_authors),
    ]

The RunPython operation takes two functions. Django calls the first one, the forward function, when migrate applies the migration. It calls the second one, the reverse function, when you undo the migration. Each function receives apps and schema_editor; this migration only needs apps.

The models come from apps.get_model(), not from from blog.models import Article. Django builds these historical models from the migration files, so they have exactly the fields the table has at this point in the history. An import would give you the current class instead. It works today, but after step 4 the current Article has no author_name. A teammate whose database is still at 0005 would then run this migration and get AttributeError: 'Article' object has no attribute 'author_name'. Historical models also have none of your custom methods, such as __str__().

The forward function keeps a dictionary from names to authors, so the two articles by Ana Lopez share one Author row. The old table holds no email addresses, but email must be unique, so the migration makes a placeholder from the name. The slugify() helper turns "Ana Lopez" into "ana-lopez", and an editor can replace the placeholders later.

The reverse function undoes the forward one: it clears every link and then deletes the authors. The order matters. Historical models keep on_delete=models.CASCADE, so deleting an author who still has articles would delete the articles too. Without the update(author=None) line, undoing this migration deleted all three articles.

Run migrate and look at both tables:

from blog.models import Article, Author

for author in Author.objects.all():
    print(author.id, author.name, author.email)
for article in Article.objects.all():
    print(article.id, article.author_name, article.author_id)

1 Ana Lopez ana-lopez@example.com
2 Ben Carter ben-carter@example.com
1 Ana Lopez 1
2 Ana Lopez 1
3 Ben Carter 2

Step 3 is a migration of its own, with no schema change in it. Django's documentation advises this for PostgreSQL in particular, where mixing schema changes and RunPython in one migration can fail with an error about "pending trigger events".

Migrating Back

A data migration deserves a test in both directions before you build on it. Migrate back to 0005:

python manage.py migrate blog 0005

Operations to perform:
  Target specific migration: 0005_author_article_author, from blog
Running migrations:
  Rendering model states... DONE
  Unapplying blog.0006_copy_author_names... OK

The same shell code now prints no authors and None in every author_id, which is exactly the state before 0006. This is the usual way to fix a mistake in a data migration: migrate back, edit the forward function, and run migrate again. The second run creates the same two authors, with the ids 3 and 4, because SQLite does not reuse the ids of deleted rows.

A migration without a reverse function cannot be undone. With migrations.RunPython(names_to_authors), the command above stops with IrreversibleError: Operation <RunPython <function names_to_authors at 0x...>> in blog.0006_copy_author_names is not reversible.

When the reverse direction really has nothing to do, pass migrations.RunPython.noop as the reverse function. It does nothing, but it makes the migration reversible. It would be the wrong choice here. With noop, migrating back to 0005 leaves both authors in the table, and the next migrate fails with IntegrityError: UNIQUE constraint failed: blog_author.email. On SQLite and PostgreSQL each migration runs in a transaction, so the failed run changes nothing, and 0006 stays unapplied. Use noop only when undoing the step leaves nothing behind that would make a later run fail.

Step 4: Remove the Old Field

With every article linked, delete the author_name line and remove null=True from author. The two models now match the lesson "Linking Models with ForeignKey". Because the field used to accept empty values, makemigrations asks what to do about existing empty rows:

It is impossible to change a nullable field 'author' on article to non-nullable without providing a default. This is because the database needs something to populate existing rows.
Please select a fix:
 1) Provide a one-off default now (will be set on all existing rows with a null value for this column)
 2) Ignore for now. Existing rows that contain NULL values will have to be handled manually, for example with a RunPython or RunSQL operation.
 3) Quit and manually define a default value in models.py.
Select an option: 2
Migrations for 'blog':
  blog/migrations/0007_remove_article_author_name_alter_article_author.py
    - Remove field author_name from article
    ~ Alter field author on article

Option 2 is right here, because the data migration has already filled every row. Had it missed an article, migrate would stop with IntegrityError: NOT NULL constraint failed: new__blog_article.author_id and leave 0007 unapplied. After migrate, the articles read their authors through the link:

from blog.models import Article

for article in Article.objects.select_related("author"):
    print(article.title, "by", article.author.name, article.author.email)

Getting Started with Django by Ana Lopez ana-lopez@example.com
Django Models in Practice by Ana Lopez ana-lopez@example.com
Understanding Querysets by Ben Carter ben-carter@example.com

Migration 0007 cannot be undone as written. Undoing a removed field adds the column back, and the database needs a value for every row. A required CharField with no default has none, so on SQLite migrating back to 0006 fails with NOT NULL constraint failed: new__blog_article.author_name. If you need that direction too, give author_name a default in a migration before you remove it. The column then comes back filled with that default, not with the old names, as the lesson on migrations showed for view_count.

Key Takeaways

• A data migration changes rows, not tables. Start it with makemigrations <app> --empty --name <name> and put a RunPython operation in it.
• Change a field's type in steps: add the new field as optional, copy the data, then remove the old field and make the new one required.
• Inside a migration, load models with apps.get_model(). These historical models match the table at that point in the history and have no custom methods.
• Write a reverse function that really undoes the forward one, and test it with migrate <app> <previous number>. Without one, migrating back raises IrreversibleError; RunPython.noop fits only when undoing needs no work.