Stream Processing

Stream processing is a computation model that ingests, transforms, and reacts to data continuously as it arrives, rather than collecting it into batches and processing it later. A distributed commit log (like Apache Kafka) is the backbone: it durably stores an ordered, append-only stream of events that multiple consumers can read independently at their own pace.

The distributed commit log

A distributed commit log (Kafka, Amazon Kinesis, Apache Pulsar) is an append-only, durable, ordered sequence of records partitioned across brokers. Producers append events; consumers read from a position (offset) at their own speed. Because the log is persistent and replayable, consumers can rewind to re-process historical data - a property batch systems take for granted but traditional message queues lack.

Partitions and ordering

A topic is split into partitions, each an independent ordered log. A producer assigns each event to a partition by a partition key (e.g., user_id). Events with the same key always land in the same partition, guaranteeing per-key ordering. Global ordering across partitions is not guaranteed - and at scale, you do not want it, because it would serialize all writes through one node.

Consumer groups

A consumer group is a set of consumers that collectively read a topic. Each partition is assigned to exactly one consumer in the group, so work is shared and no event is processed twice within the group. Different consumer groups each get their own full copy of the stream - this is how one event can feed alerting, analytics, and indexing simultaneously.

Delivery guarantees

At-least-once with idempotent consumers is the most common practical choice.

Windowing

Stream processors often need to aggregate over time (e.g., "clicks in the last 5 minutes"). Windowing divides the stream into bounded chunks:

• Tumbling window: fixed-size, non-overlapping (every 5 min).
• Sliding window: fixed-size, overlapping (5 min window sliding every 1 min).
• Session window: dynamic, grouped by activity with a gap timeout.

Stream processing gives low-latency results but adds stateful complexity: windowing, checkpointing, out-of-order event handling, and exactly-once guarantees all need careful engineering. Batch processing is simpler and naturally idempotent (re-run the whole job), but its results are always stale by the batch interval. Many production systems use a Lambda architecture (batch + stream in parallel) or a Kappa architecture (stream only, replay for reprocessing) to balance the two.

• When you draw a message queue in a case study, the interviewer may ask "tell me more about how this works" - that is your cue for partitions, consumer groups, and ordering guarantees.
• Say "at-least-once delivery with idempotent consumers" as your default; explain exactly-once only if pressed.
• Mention Kafka's partition key as the tool for per-entity ordering (e.g., user_id).
• If the design needs both real-time reactions and historical reprocessing, mention the log's replayability.