How would you implement content‑addressable storage at scale?

Content addressable storage (CAS) stores data based on its content rather than its location. Instead of saving files by path or name, each piece of data is identified by a cryptographic hash of its bytes. If two data chunks are identical, their hashes match, and the system stores the content only once. This approach enables deduplication, immutability, and strong integrity checks, which are crucial concepts in scalable architecture and distributed systems.

Why It Matters

CAS helps systems like Git, Docker, and IPFS manage massive amounts of data efficiently. By using the hash of the content as its identity, it ensures data consistency, enables natural deduplication, and prevents tampering. For interview preparation, understanding CAS shows mastery of data integrity, storage optimization, and distributed system design — all common topics in system design interviews.

How It Works (Step-by-Step)

Step 1: Chunking the Data Split large files into smaller chunks. These can be fixed-size (simple and fast) or content-defined (smarter deduplication but more complex).

Step 2: Hashing Each Chunk Compute a cryptographic hash like SHA-256 for every chunk. The hash acts as a unique fingerprint.

Step 3: Deduplication and Indexing Check if a chunk with the same hash already exists. If it does, skip storing it again. Maintain a deduplication index mapping hashes to locations.

Step 4: Manifest Creation Create a manifest (or Merkle tree) describing how to reassemble chunks into full files. The manifest itself gets a hash — forming a hierarchical structure.

Step 5: Storage Strategy Store chunks on distributed object storage (like S3 or HDFS). Use hash prefixes to shard data evenly across nodes.

Step 6: Retrieval When a user requests data, fetch the manifest, locate chunks by hash, and reconstruct the content. Each chunk’s hash is verified to ensure data integrity.

Step 7: Maintenance Garbage collection removes chunks no longer referenced. Background jobs verify hashes, repack data, and maintain cache for popular objects.

Real-World Example

Git is a textbook implementation of CAS. Every commit, tree, and blob is stored by the SHA hash of its contents. This makes versioning and verification automatic. Similarly, Docker and OCI registries use content-addressed layers to deduplicate shared base images across containers, drastically saving space and bandwidth.

Common Pitfalls or Trade-offs

• Chunk size choice — small chunks increase deduplication but bloat indexes; large chunks reduce overhead but miss dedup opportunities.
• Index scalability — billions of hashes can stress memory; use Bloom filters or tiered LSM trees.
• Encryption and privacy — deduplication across encrypted content is difficult; convergent encryption has privacy trade-offs.
• Read performance — fetching many small chunks can increase latency; pack related chunks together to reduce I/O.