Zero-Downtime Django Schema Evolution: Multi-Step Migrations with SeparateDatabaseAndState

The Danger of Default Django Migrations: Table Locks Under Load

Django's automated migration engine (makemigrations / migrate) is widely celebrated for developer productivity. However, in production systems operating under continuous write traffic (1,000+ writes/second), standard Django migrations are an operational landmine.

When executing seemingly innocuous migration operations, Django wraps statements in explicit database transactions and issues commands that acquire aggressive locks:

• migrations.AddField(..., null=False, default=...): Prior to PostgreSQL 11, this rewrote the entire physical table on disk while holding an ACCESS EXCLUSIVE lock.
• migrations.AddIndex(...): Executes CREATE INDEX, which locks the table against all concurrent INSERT, UPDATE, and DELETE operations for the duration of the index build.
• migrations.RemoveField(...): Drops the column immediately, causing instant 500 errors on running Gunicorn workers that still have the old model schema in memory.
• migrations.RenameField(...): Acquires an exclusive lock, breaks running workers, and causes downtime across rolling deployments.

To achieve true zero-downtime schema evolution, engineering teams must adopt the Expand-and-Contract pattern and master Django's migration abstraction tool: SeparateDatabaseAndState.

1. The Expand-and-Contract Migration Lifecycle

Zero-downtime schema changes are never executed in a single atomic release. They are partitioned across four distinct operational phases:

Phase 1: Expand
- Add new column as NULLABLE in PostgreSQL.
- Model is updated to accept the new field. Old column remains active.

Phase 2: Dual-Write
- Application code is deployed.
- Reads continue from OLD column. Writes write to BOTH old and new columns.

Phase 3: Batched Backfill
- Background script copies historic data from old column to new column.
- Executed in small chunked transactions (e.g. 5,000 rows/batch) to prevent replication lag.

Phase 4: Contract
- Application code switches reads to NEW column. Old column writes are stopped.
- Old column is safely dropped in a subsequent release.

2. Decoupling ORM State with SeparateDatabaseAndState

Django tracks two distinct concepts during migrations: Database Operations (the actual SQL executed against the database) and State Operations (the internal representation of models in Django's virtual project state). By default, Django ties them together. With migrations.SeparateDatabaseAndState, you can decouple them completely:

# Inside your migration file: 0042_expand_phone_number.py
from django.db import migrations, models

class Migration(migrations.Migration):
    dependencies = [
        ('users', '0041_previous_migration'),
    ]

    operations = [
        migrations.SeparateDatabaseAndState(
            # State operations: Tells Django ORM the model now has 'phone_e164'
            state_operations=[
                migrations.AddField(
                    model_name='userprofile',
                    name='phone_e164',
                    field=models.CharField(max_length=20, null=True, blank=True),
                ),
            ],
            # Database operations: Raw, safe SQL executed on PostgreSQL
            database_operations=[
                migrations.RunSQL(
                    sql="ALTER TABLE users_userprofile ADD COLUMN phone_e164 VARCHAR(20) NULL;",
                    reverse_sql="ALTER TABLE users_userprofile DROP COLUMN IF EXISTS phone_e164;",
                ),
            ],
        ),
    ]

3. Non-Blocking Index Creation: CREATE INDEX CONCURRENTLY

By default, Django wraps every migration in an atomic transaction (BEGIN ... COMMIT). However, PostgreSQL prohibits running CREATE INDEX CONCURRENTLY inside a transaction block. To build an index on a billion-row table without locking write traffic, set atomic = False and use AddIndexConcurrently:

# 0043_add_index_concurrently.py
from django.contrib.postgres.operations import AddIndexConcurrently
from django.db import migrations, models

class Migration(migrations.Migration):
    # CRITICAL: Disable Django transaction wrapping
    atomic = False

    dependencies = [
        ('users', '0042_expand_phone_number'),
    ]

    operations = [
        AddIndexConcurrently(
            model_name='userprofile',
            index=models.Index(fields=['phone_e164'], name='idx_users_phone_e164'),
        ),
    ]

4. Safe Batched Backfilling Without Replication Lag

Never run a single monolithic UPDATE table SET new_col = old_col; in production. Doing so locks millions of rows, floods the Write-Ahead Log (WAL), and causes replica lag to skyrocket. Execute backfills using small cursor slices with sleep intervals:

import time
from django.db import connection

def backfill_phone_numbers(batch_size: int = 5000, sleep_sec: float = 0.2):
    total_updated = 0
    with connection.cursor() as cursor:
        while True:
            # Update only rows where new column is NULL, bounded by batch size
            cursor.execute("""
                UPDATE users_userprofile
                SET phone_e164 = phone_legacy
                WHERE id IN (
                    SELECT id 
                    FROM users_userprofile 
                    WHERE phone_e164 IS NULL 
                      AND phone_legacy IS NOT NULL 
                    LIMIT %s
                );
            """, (batch_size,))
            
            rows_affected = cursor.rowcount
            total_updated += rows_affected
            print(f"Updated {rows_affected} records. Total so far: {total_updated}")

            if rows_affected < batch_size:
                break

            # Sleep briefly to allow autovacuum and WAL replication to catch up
            time.sleep(sleep_sec)

    print(f"Backfill complete! Total updated: {total_updated}")

5. Safe Column Deletion Without Gunicorn 500s

When you drop a column from a Django model (RemoveField) and run migrate before all application servers reload, running workers continue issuing SELECT queries listing the dropped column. This triggers fatal database errors:

django.db.utils.ProgrammingError: column users_userprofile.phone_legacy does not exist

The Safe Deletion Recipe:

1. In Release 1: Remove the field from the Django model in Python, but do not delete the column from the database. Use SeparateDatabaseAndState to update state without touching PostgreSQL. Deploy application code and verify all Gunicorn workers have reloaded.
2. In Release 2: In a subsequent deployment, run a raw SQL migration to physically drop the column: ALTER TABLE users_userprofile DROP COLUMN IF EXISTS phone_legacy;.

By enforcing this phased lifecycle, database schemas evolve seamlessly alongside application features with zero downtime, zero lock contention, and zero customer-facing errors.