Decoupling Data: How to Split a Monolith Database

If the new database performs poorly, the system can instantly revert to reading from the old database, which has been kept up to date via the dual write process.

Dark Launching and Canary Deployments

Even with the code moved and the data synchronized, releasing the new system to 100% of users immediately is risky.

The new service might have performance issues that only appear under heavy load.

To mitigate this, engineers use Dark Launching and Canary Deployments.

Dark Launching

Dark launching involves deploying the new service and having it process requests, but without showing the results to the user.

For example, when a user searches for a product, the system sends the search query to both the old search engine and the new search microservice.

The application returns the results from the old engine to the user.

However, in the background, the engineers capture the results from the new service and compare them to the old ones. This allows them to test the accuracy and performance of the new system with real production traffic without the user knowing.

Canary Deployments

Once the new service is proven to work in the dark, the team begins a Canary Deployment.

This technique involves routing a small percentage of real user traffic to the new service.

The team might configure the proxy to send 1% of all requests to the new microservice. The remaining 99% go to the legacy system.

The team monitors the 1% closely. They look for error rates, slow response times, or crashes.

If the metrics look good, they increase the traffic to 5%, then 10%, and eventually 100%.

If errors occur at the 1% level, only a tiny fraction of users are affected.

The team can quickly route that 1% back to the legacy system and fix the bug.

This controlled rollout significantly reduces the blast radius of any potential errors.

Managing Distributed Transactions

In a monolith, saving data across different tables is easy because of database transactions. A transaction ensures that either all data is saved correctly, or none of it is.

For example, when a user places an order, the system must deduct stock from the inventory table and add an entry to the order history table. If the database fails halfway through, the transaction rolls back, and no data is corrupted.

In a microservices architecture, the inventory lives in one database and the order history lives in another. A single database transaction cannot span across two different physical servers. This introduces the problem of distributed transactions.

If the order service saves the order, but the inventory service fails to deduct the stock, the data becomes inconsistent. The system thinks an item was sold, but the inventory count does not reflect it.

Eventual Consistency

To handle this, distributed systems rely on Eventual Consistency. This concept accepts that the system might not be consistent for a few milliseconds, but it will eventually become consistent.

A common pattern to implement this is the Saga Pattern.

Instead of one big transaction, the process is broken down into a series of local transactions.

1. The Order Service creates the order and publishes an event called "OrderCreated."
2. The Inventory Service listens for this event. When it receives it, it deducts the stock.
3. If the Inventory Service fails to deduct stock, it publishes an "OutOfStock" event.
4. The Order Service listens for this event and executes a compensating transaction, such as cancelling the order.

This approach is more complex than a monolithic transaction but is necessary for the system to scale and remain decoupled. It requires a shift in mindset from expecting immediate consistency to designing for failure and recovery.

The Role of Observability

When a system moves from a single application to many distributed services, debugging becomes much harder.

In a monolith, you can follow a request by looking at a single log file.

In a microservices architecture, a single user request might hop through five different services.

Migration cannot succeed without Observability. This goes beyond simple logging. It involves three pillars:

1. Logs: Detailed records of discrete events (e.g., "Database connection failed").
2. Metrics: Aggregated numerical data over time (e.g., "Average CPU usage is 40%").
3. Tracing: The ability to track a single request as it travels across different services.

Distributed tracing assigns a unique ID to every request that enters the system. This ID is passed along to every service that handles the request.

If a user reports an error, an engineer can search for that ID and see exactly which service failed and why.

Without this visibility, fixing bugs during a migration is like trying to find a needle in a haystack in the dark.

Handling Dependencies and Shared Libraries

A hidden challenge in migration is dealing with shared code.

In a monolith, it is common to have a "Utils" or "Common" folder that contains helper functions used by every part of the application.

When extracting a service, developers often feel the urge to copy this shared code into the new service or create a shared library that all microservices import.

While this seems efficient, it can reintroduce coupling. If the shared library is updated, every single microservice might need to be rebuilt and redeployed to use the new version.

A better approach during migration is to tolerate some code duplication. It is often safer to copy a specific helper function into the new microservice than to create a rigid dependency on a shared library.

This allows the new microservice to evolve independently without fear of breaking other services when a shared library changes.