How do you tune compaction in log‑structured storage?

Time partitioning is a powerful technique for managing massive datasets efficiently. By dividing data into time-based partitions (for example, by day or hour), systems can query, store, and maintain data more effectively. This approach is widely used in analytics, logging, and observability systems like Netflix, Amazon, and Uber to handle petabytes of time-series data. When done right, it improves query performance, reduces storage cost, and simplifies data lifecycle management.

Why It Matters

In large-scale distributed systems, scanning unnecessary data can be extremely costly. Time partitioning allows systems to skip irrelevant data by reading only partitions that match a query’s time range. It also enables efficient data retention and backfilling while supporting concurrent ingestion at scale. From a system design interview perspective, this concept demonstrates your ability to balance query latency, cost efficiency, and scalability. You are showing not only how to store data but how to structure it for predictable performance as volume grows.

How It Works (Step-by-Step)

1. Profile the workload

Understand the workload pattern — what time range users query, how often new data arrives, and the acceptable latency for analytics and reporting. This helps determine partition granularity and compaction strategy.

2. Choose the correct time attribute

Use event time (when an event occurred) for analytics, since it aligns with user behavior. Ingest time (when data is stored) is useful for debugging or monitoring ingestion pipelines.

3. Decide partition granularity

Daily partitions are standard for most analytical workloads. Hourly partitions are preferred for near-real-time monitoring systems. Use the coarsest granularity that still meets your query filter requirements.

4. Define partition naming and layout

Organize partitions consistently. Example paths: dt=2025-11-05 for daily, or dt=2025-11-05/hr=14 for hourly. Always store and partition in UTC to avoid timezone confusion.

5. Manage file format and size

Use columnar formats like Parquet or ORC. Keep file sizes around 100–500 MB to optimize scan performance. Merge smaller files periodically using a compaction job.

6. Handle late or out-of-order data

Real-world systems receive delayed data. Use a watermark strategy (for example, event time minus one hour/day) to accept late arrivals and reprocess only impacted partitions.

7. Use clustering and indexing

Within each time partition, cluster or sort data by frequently filtered columns (for example, user_id or service_name) to enhance scan efficiency.

8. Plan data retention and tiering

Older partitions can be dropped or moved to cheaper storage (like S3 Glacier or cold tier). This keeps operational datasets light while retaining historical data affordably.

9. Distribute write load

If an hour becomes a hotspot, add secondary bucketing (for example, hashing by user ID within that hour) to distribute load across multiple files or writers.

10. Automate maintenance tasks

Automate partition discovery, compaction, and metadata cleanup. Monitor metrics like small-file count, query pruning efficiency, and partition skew.

Real-World Example

At Netflix, client logs stream into an event lake partitioned by dt and hr. Streaming jobs write data into hourly folders, compact them periodically, and move older data to cheaper tiers. Dashboards scan only recent partitions, reducing scan cost dramatically. Late arrivals are re-merged with watermark-based jobs that rewrite only affected partitions. The combination of time partitioning and compaction keeps storage efficient while ensuring analytics remain accurate and low-latency.