Dynamo Characteristics and Criticism

Responsibilities of a Dynamo's node

Dynamo has no central server and no leader. A system like GFS has one, but Dynamo does not, so every node here takes on the same three jobs.

1. Managing get() and put() requests. A node can act as the coordinator for a key, running the operation itself, or forward the request to whichever node should handle it.
2. Keeping track of membership and detecting failures. Every node uses gossip protocol, exchanging state with other nodes peer to peer, to learn which nodes exist and which hash ranges they own.
3. Local persistent storage. Each node stores the key-value pairs that hash into its range, as the primary copy or as a replica. Which storage engine holds them depends on the application. A few options:
   • BerkeleyDB Transactional Data Store
   • MySQL (for large objects)
   • An in-memory buffer (for best performance) backed by persistent storage

Characteristics of Dynamo

A few properties explain why Dynamo became so influential:

• Distributed. Dynamo runs across a large number of machines.
• Decentralized. No central coordinator oversees operations. Every node is identical and can perform every function Dynamo needs.
• Scalable. Adding nodes scales the cluster horizontally, with no manual rebalancing. That gives Dynamo linear scalability and proven fault tolerance on commodity hardware.
• Highly available. Dynamo tolerates failures, so data stays available even when one or several nodes, or whole data centers, go down.
• Fault-tolerant and reliable. Fault tolerance stays high because data is replicated across multiple nodes.
• Tunable consistency. Applications adjust the trade-off between availability and consistency, usually by setting the replication factor and the consistency level.
• Durable. Data, once written, stays stored.
• Eventually consistent. Dynamo trades strong consistency for higher availability. Two replicas can disagree right after a write, but eventual consistency means they always converge to the same value once the update finishes spreading.

Criticism on Dynamo

The design also drew real criticism:

• Every node holds a full copy of the routing table. As the cluster grows, that table grows too, which is a real concern for scalability at very large sizes.
• Dynamo claims symmetry: every node is supposed to have the same roles and responsibilities. But seed nodes break that claim a little. They are externally discoverable, used to stop the ring from splitting into logical partitions. That makes them special, in a system that is supposed to have no special nodes.
• Security was never a goal, since Dynamo was built only for internal use. But a distributed hash table (DHT), which is what Dynamo's ring is, can still be open to several kinds of attack. Amazon could assume a trusted environment. Even so, buggy software can behave a lot like a malicious actor.
• Client applications often have to manage inconsistency themselves, and the experience is not 100 percent seamless. That is why Dynamo's design is sometimes called a leaky abstraction: the database does not hide every hard part of being distributed. Shopping cart inconsistency is the clearest example: a user who notices it may just conclude the site is buggy or unreliable.

Datastores developed on the principles of Dynamo

Dynamo itself was never open-sourced. It was built to run services inside Amazon. But two well-known datastores carried its ideas forward: Riak and Cassandra.

Riak is a distributed NoSQL key-value store, highly available, scalable, fault-tolerant, and easy to operate. Cassandra is a distributed, decentralized NoSQL wide-column database, also scalable and highly available.

The table below shows which of Dynamo's techniques each one adopted.

<table class="tg" style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent; border-spacing: 0px; border-color: black; --tw-border-opacity:1; margin: 2em auto; table-layout: fixed; width: 600px; color: rgb(61, 61, 78); font-family: &quot;Droid Serif&quot;, Georgia, serif; font-size: 18px;"><tbody style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent;"><tr style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent;"><td class="tg-1yw4l" style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent; padding: 10px 5px; border-width: 1px; border-style: solid; border-color: black; --tw-border-opacity:1; font-weight: bold; word-break: normal; hyphens: auto; font-family: Arial, sans-serif; font-size: 17px; overflow: hidden; color: black; background-color: rgb(103, 171, 159); vertical-align: top; width: 329.438px;">Technique</td><td class="tg-1yw4l" style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent; padding: 10px 5px; border-width: 1px; border-style: solid; border-color: black; --tw-border-opacity:1; font-weight: bold; word-break: normal; hyphens: auto; font-family: Arial, sans-serif; font-size: 17px; overflow: hidden; color: black; background-color: rgb(103, 171, 159); vertical-align: top;">Apache Cassandra</td><td class="tg-1yw4l" style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent; padding: 10px 5px; border-width: 1px; border-style: solid; border-color: black; --tw-border-opacity:1; font-weight: bold; word-break: normal; hyphens: auto; font-family: Arial, sans-serif; font-size: 17px; overflow: hidden; color: black; background-color: rgb(103, 171, 159); vertical-align: top;">Riak</td></tr><tr style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent;"><td class="tg-1rmb8" style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent; padding: 10px 5px; border-width: 1px; border-style: solid; border-color: black; --tw-border-opacity:1; font-weight: bold; word-break: normal; hyphens: auto; font-family: Arial, sans-serif; font-size: 17px; overflow: hidden; color: black; background-color: rgb(197, 214, 196); vertical-align: top;">Consistent Hashing with virtual nodes</td><td class="tg-rmb8" style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent; padding: 10px 5px; border-width: 1px; border-style: solid; border-color: black; --tw-border-opacity:1; word-break: normal; hyphens: auto; font-family: Arial, sans-serif; font-size: 17px; overflow: hidden; color: black; background-color: rgb(197, 214, 196); vertical-align: top;">✔</td><td class="tg-rmb8" style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent; padding: 10px 5px; border-width: 1px; border-style: solid; border-color: black; --tw-border-opacity:1; word-break: normal; hyphens: auto; font-family: Arial, sans-serif; font-size: 17px; overflow: hidden; color: black; background-color: rgb(197, 214, 196); vertical-align: top;">✔</td></tr><tr style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent;"><td class="tg-1yw4l" style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent; padding: 10px 5px; border-width: 1px; border-style: solid; border-color: black; --tw-border-opacity:1; font-weight: bold; word-break: normal; hyphens: auto; font-family: Arial, sans-serif; font-size: 17px; overflow: hidden; color: black; background-color: rgb(103, 171, 159); vertical-align: top;">Hinted Handoff</td><td class="tg-yw4l" style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent; padding: 10px 5px; border-width: 1px; border-style: solid; border-color: black; --tw-border-opacity:1; word-break: normal; hyphens: auto; font-family: Arial, sans-serif; font-size: 17px; overflow: hidden; color: black; background-color: rgb(103, 171, 159); vertical-align: top;">✔</td><td class="tg-yw4l" style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent; padding: 10px 5px; border-width: 1px; border-style: solid; border-color: black; --tw-border-opacity:1; word-break: normal; hyphens: auto; font-family: Arial, sans-serif; font-size: 17px; overflow: hidden; color: black; background-color: rgb(103, 171, 159); vertical-align: top;">✔</td></tr><tr style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent;"><td class="tg-1rmb8" style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent; padding: 10px 5px; border-width: 1px; border-style: solid; border-color: black; --tw-border-opacity:1; font-weight: bold; word-break: normal; hyphens: auto; font-family: Arial, sans-serif; font-size: 17px; overflow: hidden; color: black; background-color: rgb(197, 214, 196); vertical-align: top;">Anti-entropy with Merkle trees</td><td class="tg-rmb8" style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent; padding: 10px 5px; border-width: 1px; border-style: solid; border-color: black; --tw-border-opacity:1; word-break: normal; hyphens: auto; font-family: Arial, sans-serif; font-size: 17px; overflow: hidden; color: black; background-color: rgb(197, 214, 196); vertical-align: top;">✔<br style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent;">(manual repair)</td><td class="tg-rmb8" style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent; padding: 10px 5px; border-width: 1px; border-style: solid; border-color: black; --tw-border-opacity:1; word-break: normal; hyphens: auto; font-family: Arial, sans-serif; font-size: 17px; overflow: hidden; color: black; background-color: rgb(197, 214, 196); vertical-align: top;">✔</td></tr><tr style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent;"><td class="tg-1yw4l" style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent; padding: 10px 5px; border-width: 1px; border-style: solid; border-color: black; --tw-border-opacity:1; font-weight: bold; word-break: normal; hyphens: auto; font-family: Arial, sans-serif; font-size: 17px; overflow: hidden; color: black; background-color: rgb(103, 171, 159); vertical-align: top;">Vector Clocks</td><td class="tg-yw4l" style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent; padding: 10px 5px; border-width: 1px; border-style: solid; border-color: black; --tw-border-opacity:1; word-break: normal; hyphens: auto; font-family: Arial, sans-serif; font-size: 17px; overflow: hidden; color: black; background-color: rgb(103, 171, 159); vertical-align: top;">✘<br style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent;">(last-write-wins)</td><td class="tg-yw4l" style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent; padding: 10px 5px; border-width: 1px; border-style: solid; border-color: black; --tw-border-opacity:1; word-break: normal; hyphens: auto; font-family: Arial, sans-serif; font-size: 17px; overflow: hidden; color: black; background-color: rgb(103, 171, 159); vertical-align: top;">✔</td></tr><tr style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent;"><td class="tg-1rmb8" style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent; padding: 10px 5px; border-width: 1px; border-style: solid; border-color: black; --tw-border-opacity:1; font-weight: bold; word-break: normal; hyphens: auto; font-family: Arial, sans-serif; font-size: 17px; overflow: hidden; color: black; background-color: rgb(197, 214, 196); vertical-align: top;">Gossip-based protocol</td><td class="tg-rmb8" style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent; padding: 10px 5px; border-width: 1px; border-style: solid; border-color: black; --tw-border-opacity:1; word-break: normal; hyphens: auto; font-family: Arial, sans-serif; font-size: 17px; overflow: hidden; color: black; background-color: rgb(197, 214, 196); vertical-align: top;">✔</td><td class="tg-rmb8" style="--tw-shadow:0 0 transparent; --tw-ring-inset:var(--tw-empty, ); --tw-ring-offset-width:0px; --tw-ring-offset-color:#fff; --tw-ring-color:rgba(59,130,246,0.5); --tw-ring-offset-shadow:0 0 transparent; --tw-ring-shadow:0 0 transparent; padding: 10px 5px; border-width: 1px; border-style: solid; border-color: black; --tw-border-opacity:1; word-break: normal; hyphens: auto; font-family: Arial, sans-serif; font-size: 17px; overflow: hidden; color: black; background-color: rgb(197, 214, 196); vertical-align: top;">✔</td></tr></tbody></table>