How do you do zero‑downtime migrations (DB + app coordination)?

Zero downtime migration means changing a live database and application without causing user impact. The concept revolves around the expand and contract pattern. You first expand the schema to support both old and new code, then contract it after the new system is fully stable. The challenge lies in orchestrating both database and application changes safely with rollback options at every stage.

Why It Matters

In high availability systems like Netflix or Amazon, downtime is not an option. These companies run 24/7, serving millions of global requests per second. Every deployment must be seamless. In interviews, knowing how to handle such migrations shows you understand data consistency, backward compatibility, and safe deploy patterns in distributed systems.

How It Works (Step-by-Step)

Step 1. Prepare Safety Nets

• Define migration steps, rollbacks, and success criteria.
• Introduce feature flags for reads and writes separately.
• Set up monitoring for read/write latency, errors, and data parity.
• Validate backups and ensure point-in-time restore is possible.

Step 2. Expand Schema (Backward Compatible)

• Add new columns or tables as nullable to prevent locks.
• Build indexes concurrently or online to avoid blocking writes.
• Avoid renaming columns directly; introduce new ones instead.
• For NoSQL, make code tolerant of both old and new data shapes.

Step 3. Backfill Data Gradually

• Copy historical data in small chunks using background jobs.
• Use throttling or batching to avoid database overload.
• Monitor progress metrics like rows processed and failed counts.
• Use Change Data Capture (CDC) to sync new writes during backfill.

Step 4. Enable Dual Writes

• Update both old and new structures for new transactions.
• Use idempotent operations and message queues to prevent duplication.
• Implement shadow reads for verification between old and new data.

Step 5. Switch Reads Gradually

• Start with a small canary rollout.
• Use shadow queries to verify results before full cutover.
• Monitor metrics to detect latency or consistency issues.
• Keep rollback switches ready in case of errors.

Step 6. Contract Schema

• After full cutover, remove unused tables, columns, and indexes.
• Clean up dual-write logic and feature flags.
• Archive migration logs and metrics for audit and rollback traceability.

Real World Example

Netflix once refactored its catalog database to support richer metadata and faster search. They first created new tables and backfilled data from the old schema. Next, they dual wrote updates to both tables. When confidence was high, they gradually redirected reads to the new structure. Finally, the old tables were dropped once all clients had switched over. The migration completed without user-visible downtime.

Common Pitfalls or Trade-Offs

1. Dropping Too Early

Removing columns or tables before all reads switch can break existing code. Always expand first and contract last.

2. Non-Online Schema Changes

Avoid blocking alters that rewrite entire tables. Use online or concurrent operations.

3. Missing Idempotency

If dual writes are not idempotent, retries may duplicate data. Always design for replays.