Batch Processing vs Stream Processing

A payroll system runs once a month. A fraud detector must decide before the card transaction completes. Both process data. The difference in when the answer is needed changes everything about how they are built.

Batch processing collects data over a period and processes it all at once, on a schedule. Stream processing handles each record as it arrives, and never stops.

The decision is not about which is more modern. It is about one question. How stale may the answer be? A stale answer is one built from old data. Batch answers are hours behind. Stream answers are seconds behind, and you pay for the difference.

Batch Processing

Data accumulates in a store, usually a data lake or a warehouse. A job then runs against the whole collection on a schedule: hourly, nightly, or monthly.

The input is bounded, which means it has a clear start and end. The job sees a complete set of records, and that gives it real advantages.

• It can see everything at once. It can sort the full dataset, join two very large tables, and make several passes over the same records.
• It can fail safely. If a nightly job crashes halfway, you fix the bug and run it again on the same input. The result is identical. That re-run property makes batch systems easy to trust.
• It is cheap and simple. The work runs in bursts, often on cheap spare capacity, and nothing runs between jobs.

Typical tools: Apache Spark, Hadoop MapReduce, and a scheduler such as Airflow.

Good fits: payroll, billing runs, nightly reports, warehouse loads, retraining a machine learning model, and any backfill, which means reprocessing history.

What it costs: freshness. The answer is only as fresh as the last run. If the job runs at 2 am, a question asked at 3 pm is answered with data thirteen hours old.

Stream Processing

Records are processed as they arrive, one at a time or in very small groups. The input is unbounded, which means it has no end. So the job runs forever, and the answer is always seconds behind.

The hard part is that you never hold the complete dataset. You work with an endless sequence, and you must decide how much of it to keep in memory. That decision creates three ideas that do not exist in batch.

Windows. The stream never ends, so a count needs a boundary. A window is that boundary, such as "purchases in the last five minutes". Windows come in three shapes.

• Fixed: each five-minute block, counted separately.
• Sliding: the last five minutes, recomputed every minute.
• Session: one user's activity, grouped until a gap appears.

Late events. A phone loses signal in a tunnel and uploads its events ten minutes later. Those events belong to a window you already closed and reported. You have three options, and each has a cost. Drop the events. Hold windows open longer, using a watermark, a rule that says how late a record may arrive before its window closes. Or close on time and send a correction later.

Delivery guarantees. A processing node dies in the middle of a record. Was that record handled? Most systems promise at-least-once delivery, so the record may be processed twice. Your logic must then be idempotent, meaning that doing the work twice has the same effect as once. Exactly-once processing exists in Flink and Kafka Streams, and it costs more and must be configured on purpose.

Typical tools: Apache Flink, Kafka Streams, and Spark Structured Streaming.

Good fits: fraud detection, live dashboards, alerting, and anything where a late answer is the same as no answer.

What it costs: more parts to operate, harder debugging, and a system that must stay up at all times. A failed batch job is re-run tomorrow. A failed stream job is an incident now.

The Trade-off

Two rows deserve a closer look. The first is cost per record. Batch is cheaper for the same volume, because streaming infrastructure runs whether data is flowing or not. If nobody needs the answer sooner, paying for streaming is paying for nothing.

The second is reprocessing. Business logic changes, and someone will ask you to recompute the last two years under the new rules. Batch does this naturally. A pure streaming system needs either a stored event log to replay, or a separate batch path added later.

Using Both

Many companies run both paths on the same data. The streaming path produces fast, approximate numbers for dashboards and alerts. A nightly batch path recomputes the same numbers exactly and replaces them. This two-path design is called a Lambda architecture, and its cost is clear: you maintain the same logic twice. The alternative, called a Kappa architecture, keeps only the stream. To reprocess, it replays a durably stored event log through a new version of the job. That works when the log, often Kafka with long retention, keeps history for as long as you need.