Data Integrity & Caching

Reads and writes only matter if the bytes that come back match the bytes that went in. This lesson looks at how HDFS checks that, and how it speeds up reads for data used often.

Data integrity

Data integrity means the data a client reads back is exactly the data it wrote. A block can arrive corrupted, from a bad disk, a network fault, or a bug in the software. HDFS catches this with a checksum: a short value computed from a block's bytes, which changes if even one bit changes.

When a client writes a file, it computes a checksum for each block