0% completed
Dynamo: Introduction
On This Page
Goal
What is Dynamo?
Background
Designing a Key-Value store (video)
Design goals
Dynamo's use cases
System APIs
Goal
Add an item to your cart on Amazon. Behind that click, something has to save it. It has to work every time. It has to answer fast, no matter how many other shoppers click at the same moment. In this chapter we design that something: a distributed key-value store that is
(that is, reliable), , and .What is Dynamo?
Dynamo is the highly available key-value store Amazon built for its own internal use. Many Amazon services, like the shopping cart, bestseller lists, sales rank, and product catalog, need only primary-key access to their data. A multi-table relational database would be overkill for services like these. It would also limit how far they could scale and how available they could stay. Dynamo gives applications a flexible design instead, so each one can choose its own trade-off between availability and
.Background
Dynamo should not be confused with DynamoDB, which Dynamo's design later inspired. Dynamo itself is a distributed key-value storage system built for an "always-on" experience, meaning high availability, at a massive scale. In CAP theorem terms, that makes it an AP system: available and
, at the cost of .That trade-off follows from one observation: a system's availability correlates directly with how many customers it serves. So the main goal is to stay available to the customer even when the system is imperfect. That is what drives customer satisfaction. Inconsistencies can wait. They get resolved in the background, and most of the time the customer never notices. Dynamo is built around that principle, and it is optimized aggressively for availability as a result.
The design proved influential. It inspired several NoSQL databases, among them Cassandra, Riak, and Voldemort. It also inspired Amazon's own DynamoDB.
Designing a Key-Value store (video)
Here is a video that walks through how to design a key-value store:
Design goals
As we said above, the main goal of Dynamo is to be highly available. Three more goals sit alongside it.
- Scalable. The system should be highly scalable. Add a machine, and the system should get proportionally better, not just bigger.
- Decentralized. There should be no central or leader process. A single point like that would also become a performance bottleneck.
- Eventually consistent. Data is replicated optimistically, so it becomes rather than instantly consistent. Strong consistency pays a cost on every write, to keep the whole system correct right away. Dynamo instead resolves inconsistencies later, often during a read. That trade-off is what buys the high availability we are after.
Dynamo's use cases
By default, Dynamo is an eventually consistent database. So it fits any application that does not need strong consistency. Dynamo can support strong consistency too, but that comes at a performance cost. If an application requires strong consistency, Dynamo may not be the right choice.
Amazon uses Dynamo to run services with very high reliability needs. These services need tight control over the trade-off between availability, consistency, cost-effectiveness, and performance. Amazon's platform holds a wide range of applications, each with different storage needs. Many of them chose Dynamo because it lets each one pick the right trade-off. That trade-off still reaches high availability and guaranteed performance, in the most cost-effective way.
Many services on Amazon's platform need only primary-key access to a data store. For services like these, a relational database would be inefficient, and would limit scalability and availability. Dynamo gives them a simple, primary-key-only interface instead.
System APIs
Dynamo clients use put() and get() operations to read and write data for a given key. The key uniquely identifies one object.
-
get(key): Thegetoperation finds the nodes that hold the object for the givenkey. It returns either one object, or a list of objects with conflicting versions, along with acontext. Thecontextis encoded metadata that means nothing to the caller by itself. It includes the object's version, which matters later. -
put(key, context, object): Theputoperation finds the nodes responsible for the givenkey, and writes theobjectto disk there. Thecontextis the value agetreturned earlier, now sent back with theput. Dynamo stores it alongside the object and uses it like a cookie, to check that the object being written is still valid.
Dynamo treats both the object and the key as an arbitrary array of bytes, typically under 1 MB. It applies the
to the key, producing a 128-bit identifier. That identifier determines which storage nodes are responsible for the key.Reading Progress
0%
On This Page
Goal
What is Dynamo?
Background
Designing a Key-Value store (video)
Design goals
Dynamo's use cases
System APIs