Designing a Real-Time Event Pipeline with Kafka, Flink, and Elasticsearch

What This Blog Covers

• Why real-time pipelines need three components, not one
• What each technology does in the pipeline
• How data flows from ingestion to searchable index
• Handling failures, backpressure, and exactly-once semantics
• How to discuss this in system design interviews

Most systems start by writing events directly to a database.

A user clicks a button, the application inserts a row, and a dashboard queries it five minutes later in a batch job.

This works until it does not.

The business wants real-time dashboards.

The fraud team needs sub-second detection. The search results need to reflect changes within seconds, not minutes.

And the batch job that runs every 5 minutes is suddenly 10 minutes behind because the data volume doubled overnight.

This is the point where engineering teams start building a real-time event pipeline: a system that continuously ingests events as they happen, processes and enriches them on the fly, and delivers the results to a serving layer where they can be queried instantly.

The most common architecture for this pipeline in 2026 uses three technologies: Apache Kafka for ingestion and transport, Apache Flink for stateful stream processing, and Elasticsearch for indexing and search.

Each one solves a different part of the problem, and none of them alone is sufficient.

This guide explains the role of each component, how they connect into a working pipeline, what breaks in production, and how to discuss this architecture in system design interviews.

Why Three Components?

A natural question is:

why not just use Kafka?

Or just Flink?

Or just write directly to Elasticsearch?

Kafka alone handles ingestion and transport brilliantly, but it is not a processing engine. You cannot run windowed aggregations, join streams with reference data, or enrich events with complex logic in Kafka. Kafka stores events and moves them between systems. It does not transform them.

Flink alone can process streams, but it needs a durable source to read from and a sink to write to. Without Kafka in front of it, Flink has no replay capability. If Flink crashes, it needs to reprocess events from a certain point. Kafka's persistent, replayable log provides that capability.

Elasticsearch alone can ingest and index data, but it is not designed for stream processing. Writing raw events directly to Elasticsearch skips enrichment, filtering, deduplication, and aggregation. The index fills up with noise, queries slow down, and the cluster struggles under write pressure that could have been absorbed by Kafka.

The three-component architecture separates concerns. Kafka handles durability and decoupling. Flink handles computation and state. Elasticsearch handles serving and search. Each component does what it is best at, and failures in one component do not bring down the others.

Component 1: Kafka as the Ingestion Layer

Kafka sits at the front of the pipeline.

Every event produced by your application, whether a user action, a system metric, a transaction, or a log entry, is written to a Kafka topic.

Why Kafka for Ingestion

• Durability: Events written to Kafka are replicated across multiple brokers. If a broker crashes, no data is lost. This is critical because the ingestion layer is the first thing your events touch. If it loses data, everything downstream is wrong.
• Decoupling: Producers (your application services) and consumers (Flink, analytics, other services) do not know about each other. A producer writes to a topic. Any number of consumers can read from that topic independently. If Flink goes down for maintenance, Kafka holds the events until Flink comes back and catches up. No events are lost.
• Replay: Kafka retains events for a configurable period (days, weeks, or indefinitely with tiered storage). If you deploy a bug in Flink that corrupts processed data, you can fix the bug and replay events from Kafka to reprocess them correctly. This replay capability is one of Kafka's most valuable properties for pipeline reliability.
• Throughput: Kafka handles millions of events per second on commodity hardware. It is designed for exactly this workload: high-throughput, append-only event ingestion with sequential disk writes. A single Kafka cluster with a dozen brokers can sustain over 2 million events per second, making it the standard ingestion layer for every major real-time pipeline in production today.

Design Decisions

• Topic design: Use one topic per event type (user-clicks, orders, payments) rather than a single "all-events" topic. This allows different consumers to subscribe to only the events they care about, and different topics to have different retention policies and partition counts.
• Partitioning: Choose a partition key that distributes events evenly and groups related events together. For user events, partition by user ID. This ensures that all events for a given user land in the same partition, preserving ordering for that user while distributing the total load across partitions.
• Schema management: Use a schema registry (Confluent Schema Registry or Apicurio) to define and enforce event schemas. Producers register their event schema. Consumers validate against it. This prevents the "producer changed the field name and broke three consumers" problem.

Component 2: Flink as the Processing Layer

Flink sits in the middle. It reads events from Kafka topics, processes them, and writes the results to Elasticsearch (and potentially other sinks like another Kafka topic, a database, or S3).

What Flink Does That Kafka Cannot

Stateful processing. Flink can maintain state across events. A fraud detection job might track each user's transaction history over the last 24 hours and flag a transaction if it deviates from the pattern. This requires state: remembering what the user did previously. Kafka does not maintain application state.

Windowed aggregations. Flink can group events into time windows and compute aggregations. "How many orders were placed in the last 5 minutes?" is a windowed count. "What is the average order value per region in the last hour?" is a windowed aggregation with grouping. These computations happen continuously as events arrive, not in batch after the fact.

Stream joins. Flink can join two event streams in real time. A click stream from the website can be joined with an order stream to compute click-to-purchase conversion rates as they happen. Or an event stream can be enriched by joining it with a slowly changing reference table (product names, user profiles) to add context before indexing.

Filtering and deduplication. Not every event should reach Elasticsearch. Flink can filter out noise (health check events, internal system events) and deduplicate events (the same click recorded twice because of a client retry) before they reach the index. This keeps Elasticsearch lean and fast.

How Flink Processes Events

Flink reads from Kafka using a Kafka source connector. Each Flink task consumes one or more Kafka partitions. Flink's checkpoint mechanism periodically saves the consumer offsets and application state to a durable store (like S3 or HDFS). If Flink crashes, it restarts from the last checkpoint, re-reads events from Kafka starting at the checkpointed offset, and resumes processing without data loss or duplication.

This checkpoint-based recovery is what gives Flink its exactly-once processing guarantee within the pipeline. Combined with Kafka's exactly-once producer semantics, the end-to-end pipeline can provide exactly-once processing from Kafka through Flink to the output sink.

For understanding the fundamental difference between stream and batch processing and when each is appropriate, Batch vs. Stream Processing: How to Balance Latency and Accuracy covers the architectural trade-offs.

Component 3: Elasticsearch as the Serving Layer

Elasticsearch sits at the end of the pipeline. It receives processed, enriched events from Flink and indexes them for fast full-text search and analytics queries.

Why Elasticsearch for Serving

• Near real-time search: Elasticsearch indexes documents and makes them searchable within 1 second by default (the refresh interval). For a real-time pipeline, this means an event that happened 2 seconds ago is already searchable in your dashboard or search interface.
• Full-text search: If your pipeline feeds a search feature (searching through logs, transactions, or user activity), Elasticsearch's inverted index provides millisecond-latency full-text search that no relational database or time-series database can match.
• Aggregation queries: Elasticsearch can compute aggregations (counts, averages, histograms, percentiles) across millions of documents in milliseconds. This powers real-time dashboards that show "orders per minute by region" or "P99 latency by endpoint" without a separate analytics database.
• Schema flexibility: Elasticsearch handles semi-structured data natively. Events with slightly different fields can coexist in the same index without schema migration headaches.

Design Decisions

• Index strategy: Use time-based indices: events-2026-04-20, events-2026-04-21. This makes retention management simple (delete old indices) and keeps each index at a manageable size. Use index aliases so queries can span multiple indices transparently.
• Mapping design: Define explicit mappings for your event fields rather than relying on dynamic mapping. Specify which fields are keyword (exact match, aggregations) vs text (full-text search). Incorrect mappings are the most common source of poor query performance in Elasticsearch.
• Write throughput tuning: Flink writes to Elasticsearch in bulk batches (typically 1,000 to 5,000 documents per batch). Tune the refresh interval to 5 to 30 seconds if near-real-time search is sufficient for your use case. The default 1-second refresh creates many small segments and increases merge overhead.

For understanding how time-series data is stored and queried in systems like this, System Design Deep Dive: Time Series Databases (TSDBs) Explained covers the underlying storage concepts.

The End-to-End Flow

Here is what a complete request flow looks like in a real-time analytics pipeline:

A user places an order on the e-commerce platform.

The Order Service publishes an OrderCreated event to the orders Kafka topic.

The event includes the order ID, user ID, items, total amount, and timestamp.

Flink's order processing job reads the event from Kafka.

It enriches the event by joining it with a user profile table (adding the user's region and account tier) and a product catalog (adding category names).

It computes a running total of orders per region in a 5-minute tumbling window. It writes the enriched event to Elasticsearch and the windowed aggregation to a separate Kafka topic for downstream dashboards.

Elasticsearch indexes the enriched event.

Within 1 second, the event appears in the operations dashboard. A search for "orders from region=US-WEST in the last hour" returns the result instantly.

The entire flow, from the user clicking "place order" to the event being searchable, takes 2 to 5 seconds under normal load.

What Breaks in Production

Backpressure

If Flink cannot keep up with the event rate from Kafka, events queue up in Kafka. This is by design: Kafka is the buffer.

But if the backlog grows too large, end-to-end latency increases from seconds to minutes.

Monitor Kafka consumer lag (the difference between the latest offset and the consumer's current offset) and alert when it exceeds your latency budget.

The fix is usually scaling Flink parallelism (adding more task slots to process more partitions concurrently) or optimizing the processing logic (reducing the cost of enrichment joins or aggregations).

Elasticsearch Write Rejections

Elasticsearch rejects writes when its write queue is full (bulk thread pool exhaustion). This causes Flink to retry, which slows down processing and increases Kafka consumer lag.

The fix is tuning Elasticsearch's bulk thread pool size, reducing the write rate from Flink (larger batches, less frequent flushes), or adding more Elasticsearch data nodes.

Schema Evolution

When the event schema changes (a new field is added, a field type changes), every component in the pipeline must handle the change.

Kafka's schema registry enforces backward compatibility.

Flink's deserialization must handle events in both old and new formats.

Elasticsearch's mapping must be updated to include the new field.

The golden rule: always make schema changes backward compatible.

Add new optional fields.

Never remove or rename existing fields.

Use a schema registry with compatibility checks enabled.

Flink Checkpoint Failures

If Flink's checkpoint takes too long (because the state is too large or the checkpoint storage is slow), the checkpoint times out and Flink cannot guarantee exactly-once semantics.

Monitor checkpoint duration and tune the checkpoint interval and state backend (use RocksDB for large state, keep the state small by expiring old entries).

For understanding the broader principles of event-driven systems that this pipeline is built on, Event-Driven Architecture 101: Decoupling Systems for Scale covers the foundational patterns.

How This Shows Up in System Design Interviews

Real-time pipeline design comes up in questions like "design a real-time analytics dashboard," "design a fraud detection system," or "design a log search platform." The Kafka-Flink-Elasticsearch stack is a strong default answer for any question that requires continuous event processing with low-latency querying.

Here is how to present it:

"For the real-time analytics pipeline, I would use Kafka as the ingestion layer. Every service publishes events to Kafka topics partitioned by user ID. Flink consumes these events, enriches them by joining with reference data, computes 5-minute windowed aggregations, and writes the results to Elasticsearch for the dashboard. Kafka provides durability and replay. Flink provides stateful processing with exactly-once checkpointing. Elasticsearch provides sub-second search and aggregation queries. If Flink goes down, Kafka holds events until Flink recovers and catches up from the last checkpoint. If Elasticsearch is slow, Flink buffers writes and Kafka absorbs the backpressure."

That answer names each component's role, explains the data flow, addresses fault tolerance, and demonstrates understanding of backpressure. It is a complete senior-level answer.

Common Mistakes

1. Writing directly to Elasticsearch without Kafka. If your application writes events directly to Elasticsearch and Elasticsearch goes down, events are lost. Kafka absorbs writes during downstream outages and replays them when the sink recovers.

2. Using Kafka Streams when you need Flink. Kafka Streams is great for lightweight, embedded processing within a Kafka-centric application. But for complex windowed aggregations, stream joins, and large stateful processing, Flink is significantly more capable. Kafka Streams runs as a library inside your application. Flink runs as a dedicated cluster with its own resource management, state backends, and checkpoint coordination.

3. Not monitoring consumer lag. Kafka consumer lag is the single most important metric for pipeline health. If lag is growing, your pipeline is falling behind real-time. If lag is stable near zero, your pipeline is healthy. Alert on lag exceeding your latency SLO.

4. Over-indexing in Elasticsearch. Indexing every field of every event wastes storage and slows writes. Use Flink to filter, deduplicate, and select only the fields that need to be searchable before writing to Elasticsearch. A lean index is a fast index.

5. Ignoring exactly-once semantics. Without exactly-once guarantees, duplicate events in Elasticsearch corrupt aggregation results (double-counted orders, inflated revenue). Enable idempotent producers in Kafka, use Flink's checkpoint-based exactly-once, and deduplicate at the Elasticsearch sink using document IDs derived from the event's unique key.

6. Not planning for schema evolution. Your event schema will change. If you do not use a schema registry with backward compatibility enforcement, a producer can deploy a breaking change and silently corrupt downstream processing.

When This Architecture Is Overkill

Not every system needs a Kafka-Flink-Elasticsearch pipeline. Here is when simpler alternatives work better.

• If your latency requirement is minutes, not seconds, a batch pipeline (Kafka to S3, then Spark or dbt to a data warehouse) is dramatically simpler to build and operate. Real-time is a feature, not a default. Only pay the complexity cost when the business genuinely needs sub-second results.

• If your event volume is under 1,000 events per second, you might not need Kafka or Flink at all. A simple architecture where your application writes to a PostgreSQL table and Elasticsearch reads from it via Change Data Capture (Debezium) gives you near-real-time search with far less operational overhead.

• If you only need search, not aggregations, Elasticsearch with direct writes (buffered through a simple queue like SQS or RabbitMQ) might be sufficient. Flink adds value when you need stateful computation: joins, windows, enrichment, deduplication. If your events just need to be indexed as-is, skip the processing layer.

• If you only need aggregations, not search, a time-series database like InfluxDB or Prometheus with Grafana might be a better fit. Elasticsearch is powerful but operationally complex. If your use case is "dashboard with metrics over time" and you do not need full-text search, a purpose-built TSDB is simpler.

The best architecture is the simplest one that meets your requirements.

Start simple and add complexity (Kafka, Flink, Elasticsearch) only when you hit the limits of the simpler approach.

Many teams adopt the full three-component stack prematurely and spend months managing operational complexity that a PostgreSQL instance with Debezium could have handled.

Conclusion: Key Takeaways

• The Kafka-Flink-Elasticsearch stack separates ingestion, processing, and serving. Each component does what it is best at. Kafka handles durability. Flink handles computation. Elasticsearch handles search and analytics.

• Kafka provides the durable, replayable buffer. If any downstream component fails, Kafka holds events until it recovers. Replay capability makes bug fixes and reprocessing possible.

• Flink provides stateful stream processing. Windowed aggregations, stream joins, enrichment, filtering, and deduplication happen in Flink before data reaches the serving layer. Checkpoint-based recovery provides exactly-once guarantees.

• Elasticsearch provides near-real-time search and aggregations. Events are searchable within seconds of occurring. Time-based indices and explicit mappings keep the cluster fast and manageable.

• Monitor Kafka consumer lag as the primary health metric. Growing lag means your pipeline is falling behind. Stable lag near zero means you are keeping up with the event rate.

• In interviews, name each component's role and explain the fault tolerance story. The pipeline's value is not just speed. It is that each component can fail independently without losing data.