Event Streaming with Apache Kafka: A Beginner's Guide to Real-Time Data Processing

Updated on Sep 15, 2026

13 min read

Event streaming is the practice of capturing facts as they happen, storing them in an ordered log, and allowing multiple applications to process that log at their own pace. Apache Kafka is one of the most widely used platforms for this model. It can connect services, feed analytics systems, collect telemetry, and support event-driven applications without forcing a producer to call every consumer directly.

This guide explains Kafka for developers and data engineers who need a durable mental model rather than a list of commands. You will learn what Kafka stores, how partitions and consumer groups scale processing, how KRaft manages a cluster, and how to run a small local topic safely.

What Is Event Streaming with Apache Kafka?

An event is a record of something that happened: an order was placed, a payment was authorized, or a device reported a temperature. Event streaming treats those records as a continuous flow. Producers append events, and consumers read them when they are ready.

Apache Kafka is a distributed event-streaming platform built around a partitioned, replicated log. A Kafka cluster:

Accepts records from producer applications.
Appends records to named topics, divided into partitions.
Retains records according to configured policies instead of deleting them immediately after one consumer reads them.
Lets independent consumer groups read the same topic at different positions.
Replicates partitions across brokers so a broker failure does not automatically destroy the data.

Kafka is therefore more than a transient message queue. It is a shared, replayable event log. That distinction matters when a new service needs historical events, an analytics job must reprocess a time range, or a consumer has to recover from a failure.

Why Event Streaming Needs a Durable Log

Traditional request-response integration works well when one service needs an immediate answer. It becomes harder to operate when one action must notify many systems, when consumers have different processing speeds, or when a downstream system is temporarily unavailable.

A durable stream addresses these problems by separating the time at which an event is produced from the time at which it is processed:

A producer publishes a record without knowing every downstream consumer.
Kafka appends the record to a partition and assigns it an offset.
Each consumer group tracks its own progress.
Consumers can retry, pause, or replay records without asking the producer to send them again.

This decoupling does not remove distributed-systems problems. Teams still need event schemas, idempotent handlers, access controls, retention policies, and monitoring for consumer lag. Kafka provides the storage and coordination primitives; it does not decide whether a business operation is safe to repeat.

Kafka Compared with Queues and Pub/Sub

Characteristic	Kafka event log	Traditional work queue	Basic pub/sub topic
Primary model	Append and read a retained log	Deliver work to a worker	Fan out a notification
Consumer progress	Offset per partition and consumer group	Usually an acknowledgement or deletion	Subscription-specific cursor or delivery state
Replay	Native when records are retained	Often limited or unavailable	Depends on the service
Parallelism	Partitions and consumer-group members	Queue workers	Subscription and service scaling
Ordering	Guaranteed within a partition	Depends on queue and configuration	Depends on provider
Best fit	Integration, analytics, CDC, and streams	Background jobs and task processing	Notifications and loosely coupled reactions

These categories overlap. A Kafka topic can carry task-like records, and a managed pub/sub service can provide retention. The design decision should follow the required replay, ordering, throughput, and operational model rather than the product name.

How Kafka Works: Architecture and Data Flow

The core Kafka flow is:

Producer -> topic -> partition leader -> replicated partition -> consumer group -> application

Records, keys, and offsets

A Kafka record can contain a key, value, timestamp, headers, and other metadata. The key is important because Kafka normally hashes it to choose a partition. Records with the same key are routed to the same partition, which gives a useful ordering boundary for entities such as one customer or one order.

An offset is the monotonically increasing position of a record within one partition. It is not globally unique: offset 42 in partition 0 is a different position from offset 42 in partition 1. A consumer commits offsets to record progress, but the offset does not prove that business processing completed successfully. Applications should commit only according to their processing and retry strategy.

Topics and partitions

A topic is a logical name such as orders.created. Partitions are the topic’s append-only logs. Kafka can distribute partitions across brokers, allowing a topic to scale beyond one server.

Partition count is an architectural decision. More partitions can increase producer and consumer parallelism, but they also add files, metadata, open connections, and coordination work. A topic cannot use more active consumer workers in one group than it has partitions, because each partition is assigned to at most one member of that group at a time.

Kafka preserves record order inside one partition, not across an entire topic. If a workflow requires order per order ID, use the order ID as the record key and ensure the topic’s partitioning remains stable.

Consumer groups and rebalancing

A consumer group is a set of consumers cooperating to process a topic. Kafka assigns each partition to one group member. If a member stops, Kafka rebalances its partitions to other members. If a new member joins, work can be redistributed.

Different groups receive independent views of the same topic. For example, fraud-detection can process every payment event while warehouse-loader processes the same events for analytics. Within one group, adding consumers improves parallelism only while unassigned partitions remain available.

Brokers, replication, and availability

A broker stores partition replicas and serves client requests. One replica is the leader for a partition; producers and consumers normally interact with that leader. Followers copy the leader’s log. The replication factor determines how many copies exist, while the in-sync replica set identifies replicas that are caught up enough to participate in safe leadership changes.

Replication is not a backup by itself. Accidental deletes, corrupt payloads, bad retention settings, or an application publishing invalid events can be copied to every replica. Use backups, access controls, retention reviews, and recovery exercises in addition to replication.

KRaft and cluster metadata

Modern Kafka deployments use KRaft, Kafka’s built-in metadata quorum, rather than requiring ZooKeeper. KRaft controllers manage cluster metadata such as topics, partitions, and broker membership. A deployment can run nodes in combined broker/controller mode for development or separate those roles for larger production clusters.

ZooKeeper-based instructions still appear in older tutorials, but they should not be the default for a new cluster. Always match the commands and configuration properties to the Kafka version you are deploying, and follow the current Apache Kafka quickstart.

Kafka Components and Variants

Kafka’s platform includes several APIs and deployment choices:

Producer API: Publishes records and controls batching, acknowledgements, compression, retries, and delivery timeouts.
Consumer API: Polls records, manages group membership, and commits offsets.
Kafka Streams: A Java library for stateful transformations, joins, windows, and aggregations without building a separate processing engine.
Kafka Connect: A framework for moving data between Kafka and external systems through source and sink connectors. See the Kafka Connect documentation.
Schema management: JSON, Avro, Protobuf, or other formats can be paired with a compatibility policy and schema registry. Kafka itself stores bytes; the application contract is a separate concern.
Managed Kafka: A cloud provider or vendor operates brokers, upgrades, storage, and some scaling tasks. Teams still own topic design, client configuration, security, and data governance.
Self-managed Kafka: The team operates the cluster, networking, disks, upgrades, capacity, and incident response. It offers control but requires substantial platform expertise.

Delivery semantics

Kafka clients commonly describe delivery as:

At-most-once: Commit or move the offset before processing. A crash can lose a record, but duplicates are less likely.
At-least-once: Process before committing the offset. A crash between processing and committing can cause a record to be processed again.
Exactly-once processing: Use Kafka transactions and compatible processing boundaries to reduce duplicates within a defined scope. This is not a blanket guarantee that every external database, email provider, or side effect will happen exactly once.

At-least-once processing with an idempotent consumer is often the clearest default. Use a stable event ID, enforce a uniqueness constraint or deduplication record where appropriate, and make retries observable.

Real-World Use Cases

Kafka is a strong fit when events need to be retained, processed by multiple independent consumers, or replayed:

Service integration

An order service can publish OrderPlaced once. Inventory, billing, notifications, and fulfillment services can subscribe through separate consumer groups. The producer does not need synchronous knowledge of every downstream system.

Activity and audit pipelines

Web, mobile, and service activity can flow into Kafka before being loaded into a warehouse or search system. Retention gives data teams a window in which to repair a sink or rebuild a derived dataset.

Change data capture

A connector can publish database changes as events. Downstream systems can update search indexes, caches, or analytical stores without repeatedly polling the source database. CDC requires careful handling of transaction boundaries, deletes, schema changes, and replay order.

Fraud detection and operational decisions

Payment and account events can be processed by a low-latency service that maintains state across a stream. A decision service should still define timeouts and fallback behavior because a stream processor cannot make an unavailable dependency reliable.

For the downstream risk components that consume these events, see the fraud detection system architecture guide.

Telemetry and observability

Applications and devices can publish high-volume measurements to partitioned topics. Stream processors can aggregate windows, detect thresholds, and route alerts while a separate sink stores longer-term data.

Practical Considerations: Run Kafka Locally

The following workflow uses a recent Kafka distribution in KRaft standalone mode. It is intended for learning, not production. Download the current binary using the version and package shown in the official downloads page.

Start a local KRaft broker on Linux or macOS

From the extracted Kafka directory, generate a cluster ID, format the local storage, and start the server:

KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
bin/kafka-storage.sh format --standalone -t "$KAFKA_CLUSTER_ID" -c config/server.properties
bin/kafka-server-start.sh config/server.properties

On Windows, use the .bat files from the distribution and set the ID in PowerShell:

$KAFKA_CLUSTER_ID = (bin\kafka-storage.bat random-uuid)
bin\kafka-storage.bat format --standalone -t $KAFKA_CLUSTER_ID -c config\server.properties
bin\kafka-server-start.bat config\server.properties

Do not run the format command against an existing production data directory unless you have verified the recovery procedure. Storage formatting initializes the local metadata required by the KRaft server.

Create, produce, and consume a topic

In a second terminal, create a topic with one partition for a simple demonstration:

bin/kafka-topics.sh --create \
  --topic orders.created \
  --bootstrap-server localhost:9092 \
  --partitions 1 \
  --replication-factor 1

Publish a few records:

bin/kafka-console-producer.sh \
  --topic orders.created \
  --bootstrap-server localhost:9092
>{"orderId":"order-1001","status":"created"}
>{"orderId":"order-1002","status":"created"}

Read them from the beginning with a named consumer group:

bin/kafka-console-consumer.sh \
  --topic orders.created \
  --bootstrap-server localhost:9092 \
  --from-beginning \
  --group order-audit

Run a second consumer with the same group and add more records. Kafka will divide partitions between the members. Because this example has one partition, only one member can actively consume at a time. Run a different group to read the complete stream independently.

Production configuration checklist

Before using Kafka for important data, define:

Partitioning: Choose a key and partition count based on ordering and throughput requirements. Document what order is guaranteed.
Replication and acknowledgements: Use a replication factor appropriate for the failure domain, require suitable producer acknowledgements, and monitor under-replicated partitions.
Retention: Set time- or size-based retention for replay needs and storage capacity. Compaction is useful for a latest-value topic but is not a general replacement for backups.
Consumer lag: Alert on lag and processing latency, not only broker CPU. A consumer can be connected and healthy while falling behind.
Retries and dead letters: Bound retries, distinguish transient from permanent failures, and preserve failed records with enough context to diagnose them.
Security: Use TLS for transport, authentication for clients, and ACLs or equivalent authorization for topics and consumer groups.
Schema evolution: Make compatibility rules part of the deployment process. Prefer additive, optional changes until all consumers have migrated.

The Kafka design documentation explains the storage, replication, and delivery model in more detail. Client behavior also depends on settings such as fetch limits, poll intervals, and offset reset policy; review the consumer configuration reference instead of copying defaults blindly.

Common Misconceptions About Kafka

“Kafka guarantees global ordering.”

Kafka orders records within a partition. A topic with multiple partitions has no single total order. Use a meaningful key for per-entity ordering, and do not infer cross-partition order from timestamps.

“A message disappears when one consumer reads it.”

A consumer advances its own offset. The record remains available until the retention or compaction policy removes it, so another consumer group can read it independently.

“Replication means Kafka is a backup system.”

Replication protects against some broker failures. It does not provide point-in-time recovery from operator mistakes, malicious writes, invalid producers, or a site-wide outage. Backups and recovery testing remain necessary.

“Exactly once applies to every side effect.”

Kafka transactions can coordinate certain Kafka reads and writes. They cannot automatically make an email, HTTP request, or unrelated database transaction exactly once. External effects need idempotency keys, transactional outbox patterns, or other explicit coordination.

“More partitions always improve performance.”

Partitions enable parallelism, but they also increase resource usage and coordination. Start with measured throughput and expected growth, then validate partition counts with realistic producers and consumers.

“Kafka replaces every queue and database.”

Kafka is excellent for retained streams and high-throughput integration. A task queue may be simpler for one-off work with per-task visibility, and a database remains the system of record for many transactional entities. Choose the primitive that matches the data and failure model.

Learn the broader patterns in event-driven architecture in the cloud.
Document event contracts with AsyncAPI for event-driven architectures.
Compare asynchronous options in microservices communication patterns.
Separate retained event transport from state reconstruction with Event Sourcing and CQRS.