How do you build materialized views from streams (refresh/invalidation)?

Materialized views built from streams make real time analytics possible while keeping reads fast and cheap. They convert continuous event data into query ready state for dashboards, APIs, and reporting. This guide explains how to design, refresh, and invalidate them effectively for scalable systems and system design interviews.

Introduction

A materialized view from a stream is a precomputed dataset derived from an event source like Kafka or Kinesis. Instead of scanning millions of events for every query, the system maintains a continuously updated table with the latest computed values such as totals, averages, or counts. The challenge is keeping it correct and fresh as data flows endlessly.

Why It Matters

Materialized views help achieve low latency reads and offload expensive computation from online queries. They are used in recommendation systems, real time dashboards, and monitoring pipelines. In interviews, this topic reveals your understanding of data freshness, consistency, and throughput trade offs — core to scalable architecture and distributed systems.

How It Works (Step by Step)

1. Define the target view Start by deciding what entity you are aggregating or summarizing. Example: daily active users or per product revenue. Define keys, time windows, and the update granularity.

2. Capture and model the input stream Use a structured stream with clear event IDs and timestamps. Choose partitioning keys carefully to preserve ordering for deterministic updates.

3. Select processing semantics Most stream processors (Kafka Streams, Flink, Spark) offer at least once delivery. Design idempotent updates by using event IDs or deduplication stores to avoid double counting.

4. Maintain intermediate state For aggregations, keep partial states like running sums or counts in local stores such as RocksDB. For joins, maintain dimension tables in memory or with CDC (Change Data Capture) updates.

5. Write to the sink Output results to a key value store, warehouse table, or cache. Use upsert semantics to replace previous rows atomically. For analytical systems, write partitioned files periodically.

6. Set refresh policies

   • Continuous refresh applies each incoming event instantly.
   • Micro batch refresh commits updates every minute or hour.
   • Bulk refresh replays historical data for backfills or schema changes.

7. Set invalidation rules

   • Time-based: Expire old rows using TTL or partition pruning.
   • Event-based: Invalidate when an upstream change or delete event arrives.
   • Dependency-based: Rebuild when related datasets are updated.

8. Handle out-of-order events Use watermarks based on event time to close windows gracefully while allowing late events to adjust previous aggregates.

9. Serve and monitor Expose the view through APIs or BI tools. Track lag, watermark delay, and correctness. Publish freshness metrics to show users how recent the data is.

Real World Example

Netflix maintains a “viewership summary” table updated from continuous play events. A Flink job aggregates watch duration per user and title, writing incremental updates to Cassandra. The refresh interval is continuous, while invalidation happens when content is removed. Late events within 24 hours are merged using watermarking to keep the aggregates accurate.