How do you ensure ordering guarantees across partitions?

Compaction in log structured storage is the quiet background worker that keeps your LSM tree fast and lean. Data is first written to memory, flushed into sorted files on disk, then periodically merged so that reads do not scan a pile of overlapping files. Tuning compaction is the art of deciding when to merge, how much to merge, and which files to merge so that you hit your target service level while staying within CPU, memory, and IO budgets.

Why It Matters

Compaction directly controls three user visible outcomes in any distributed system that relies on an LSM tree.

• Read latency and tail behavior because overlapping files create extra lookups and cache misses.

• Write throughput and durability because aggressive merges can steal IO from foreground appends or cause write stalls.

• Cost because unnecessary rewriting increases write amplification which burns SSD lifespan and compute time. For a system design interview, the strongest answers show you can reason about these trade offs and choose a compaction strategy that matches workload shape, retention policy, and hardware.

How It Works step by step

1. Incoming writes go to a memtable and a write ahead log for durability.

2. When the memtable fills, the engine flushes it to disk as an immutable sorted file often called an SSTable. These fresh files usually land in level zero where they overlap in key space.

3. A background scheduler picks compaction jobs. Each job reads a set of sorted files, merges keys, discards tombstoned or expired rows, and writes new files to deeper levels with less overlap.

4. The engine limits read write interference using compaction rate limits, job concurrency caps, and soft or hard stall thresholds when level zero grows too large.

5. The scheduler computes compaction debt which is an estimate of bytes that need merging to keep the tree healthy. Your goal is to keep debt bounded while meeting tail latency goals.

Key tuning knobs you will use:

• Choose a strategy fit to the workload

  • Size tiered compaction merges a handful of similar sized files at the same level. This favors fast writes and low compaction pressure at the cost of higher read amplification and more overlap.

  • Leveled compaction keeps almost non overlapping ranges per level so each key is present once per level. This improves point reads and range scans but increases the number of bytes rewritten.

  • Universal compaction chooses small sets of files regardless of level and is often used for SSD only deployments with smaller datasets or frequent updates.

  • Time window compaction groups files by time windows which is great for time series data and TTL heavy workloads because old windows can be merged and dropped with little overlap with fresh data.

• Shape the level sizes

  • Pick a base level size and a level size multiplier. Larger multipliers reduce the total number of levels which can help writes, while smaller multipliers reduce read amplification.
  • Let the engine adapt base level size to the dataset. Dynamic level sizing helps when data volume grows quickly.

• Control file size and file count

  • Target file size should align with your IO block size and cache. Larger files cut down on metadata and open file overhead but make each compaction job heavier.
  • Set level zero file thresholds. A soft limit triggers more aggressive scheduling and a hard limit stalls writes to protect read latency.
  • Enable subcompactions when available so a large job is split into independent ranges that run in parallel.

• Bound background pressure

  • Use separate thread pools for flush and compaction.
  • Cap the number of concurrent jobs per level so compactions do not thrash the same hot ranges.
  • Rate limit compaction IO to preserve headroom for foreground reads and writes.

• Manage tombstones and TTLs

  • Turn on compaction filters to drop expired data early.
  • If range tombstones are common, schedule priority passes that compact overlapping ranges quickly to reclaim space and reduce read amplification from shadowed keys.

• Compression and caching

  • Keep compression light or disabled on level zero and the first disk level to minimize write latency.
  • Use stronger compression on deeper levels where files change less often.
  • Size your block cache and Bloom filters to make the best of the reduced overlap after compaction.

• Separate hot and cold data

  • Use column families or table level strategies so hot write heavy tables can stay on size tiered or time window while read critical tables use leveled.
  • For cold historical data, widen time windows and increase file size to reduce compaction frequency.

• Monitor the right signals

  • Level zero file count, pending compaction bytes, estimated compaction debt.
  • Foreground write stall time and ratio of bytes read to bytes written by compaction.
  • P99 and P999 read latency during compaction, cache hit rate, and Bloom false positive rate.
  • SSD wear metrics if you manage your own hardware.

• Tune by experiment not guesswork

  • Start from a strategy that matches the workload, then change one knob at a time and measure.
  • Benchmark with production like key distribution, value sizes, and TTL patterns.
  • Track both average and tail behavior since compaction effects often show up as outliers.