Kafka Connect Architecture: Workers, Connectors, and Tasks

Kafka Connect is the integration framework that moves data between Apache Kafka and external systems without requiring every team to write and operate a custom producer or consumer. This explainer is for developers and data engineers who need to understand how Connect divides work among connector plugins, connector instances, tasks, and workers, and what makes a deployment reliable.

Why Kafka Connect Is Being Discussed

Data pipelines often need to copy database changes into Kafka, or deliver Kafka events to a search index, warehouse, or object store. A custom integration can handle one source and destination, but operating dozens of separate programs means repeatedly solving configuration, retries, offsets, monitoring, and deployment.

Kafka Connect provides a shared runtime and standard integration APIs around those jobs. Teams can install a connector plugin, configure a named connector instance, and let Connect run its tasks. The framework is especially useful when a company has many independent pipelines and wants consistent operations. It does not remove the need to understand each source system’s consistency, schema, rate limits, or failure behavior.

What Is Kafka Connect?

Kafka Connect is a framework for continuously moving data into or out of Kafka. A source connector reads from an external system and writes records to Kafka topics. A sink connector reads records from Kafka and writes them to an external system. A connector plugin implements the system-specific behavior, while Kafka Connect supplies the runtime, worker coordination, configuration, and monitoring interfaces.

The Apache Kafka Connect user guide describes standalone and distributed workers, connector configuration, and their internal topics. The Apache Kafka project includes Connect among Kafka’s data-integration capabilities. Kafka Connect is not itself a database, message broker, or general-purpose stream processor: Kafka stores the shared event log, and Connect moves records across the system boundary.

The Problem Kafka Connect Solves

Without a common runtime, each integration often needs bespoke logic for reading source data, publishing to Kafka, committing progress, handling transient errors, and exposing health. A second integration may repeat the same machinery with different assumptions. Failure recovery then depends on the individual script, and operators have to learn a different deployment model for every pipeline.

Connect standardizes several of those operational tasks. It separates a connector’s description of an integration from the tasks that perform its work, exposes a management API, and can distribute tasks across a group of workers. It also gives connectors a common data model and configurable converters to translate records to and from Kafka’s serialized bytes.

That standardization has limits. The connector still determines which source records are visible, how a table snapshot is taken, how destination writes behave, and what counts as a successful delivery. Connect cannot make a database transaction and a Kafka write atomic merely because one connector coordinates them. For example, PostgreSQL logical replication and CDC explains the database-side change stream that a source connector may consume; slot retention, schema changes, and replay still require operational planning.

How Kafka Connect Works

A useful mental model is:

External system <-> connector task <-> Connect worker <-> Kafka

A connector plugin is installed on workers. An operator then creates a connector instance, a configured job such as “read this database” or “write these topics to this object store.” The connector coordinates the work and can produce task configurations. A task is the unit that actually reads or writes records. The worker process hosts connector instances and runs their tasks.

For a source pipeline, a task polls or receives changes from its source, converts them to Kafka Connect records, and sends them to Kafka. Connect records carry a key, value, optional schema, and metadata such as the source partition and source offset. Connect saves source offsets so a task can resume after a restart. A source connector defines how source partitions and positions are represented; this behavior is therefore connector-specific.

For a sink pipeline, tasks subscribe to one or more Kafka topics, convert serialized Kafka data into Connect records, and invoke the destination system’s write operation. Sink tasks use Kafka consumer-group progress to track records they have processed. The destination may still receive a duplicate if a task writes successfully and fails before its Kafka progress is committed. A connector’s idempotent writes or destination-side deduplication may be needed when that matters.

Connectors, tasks, and workers compared

tasks.max places an upper bound on the number of tasks a connector may create. It does not force that amount of parallelism: the connector may be unable to divide the source into that many independent units. For example, a single file reader or a source with one unpartitioned cursor cannot necessarily use several tasks productively.

Standalone and distributed workers

In standalone mode, one worker process runs the connector jobs. This is straightforward for local development, a small one-off import, or a pipeline where the operator accepts that the process is a single point of failure. Source offsets are commonly kept in a local file, so replacing the host or losing that file can affect recovery.

In distributed mode, multiple worker processes join a Connect cluster using the same group.id. The workers coordinate connector and task assignments, and Connect can rebalance work when workers join or leave. The cluster stores connector configurations, source offsets, and status in Kafka topics. A worker failure can therefore move supported tasks to another worker, but a failed task may still require an operator or automation to restart it.

The distributed workers must share the same internal topic names and compatible settings. The configuration topic should have one partition and compaction enabled; the offsets topic needs multiple partitions and compaction; the status topic is also compacted. Those topics hold framework state, not the business events moved by the connectors. Configure replication based on the cluster’s failure requirements, and monitor the state topics as carefully as other Kafka infrastructure. The Kafka Connect concepts and architecture documentation from Confluent provides a second explanation of workers, tasks, and rebalancing.

Key Components and Concepts

• Plugin path: Workers load connector, converter, and transformation plugins from configured locations. Install compatible plugin versions on every worker and restart workers when needed; otherwise a task may be assigned to a process that cannot load its class.
• Converters: A converter handles serialization between Connect’s internal record representation and Kafka bytes. JSON, Avro, and other formats are choices for data encoding; they do not automatically enforce a schema contract across producers and consumers.
• Source offsets: These identify how far a source task has progressed, such as a log position or database log sequence number. They are distinct from Kafka topic offsets, which identify records in Kafka partitions.
• Sink consumer offsets: A sink task tracks its progress through Kafka as a consumer group. Retrying records can still repeat a destination write, so “Connect processed the message” is not synonymous with “every external side effect happened exactly once.”
• Internal topics: In distributed mode, Connect persists its configuration, source offsets, and status in Kafka topics. They must be durable and available for the lifetime of the Connect cluster.
• REST API: Operators use HTTP endpoints to create, inspect, pause, resume, and restart connectors and tasks. A successful request means the API accepted an operation; check connector and task status to confirm the pipeline is healthy. HTTP method and response semantics are defined more broadly by IETF RFC 9110.
• Single Message Transforms (SMTs): These are small, per-record changes such as extracting a field or adding metadata. Use a stream-processing application or a destination-side transformation for joins, aggregation, complex state, or logic that needs a broader view of the data.

Kafka Connect’s internal coordination is separate from the Kafka cluster’s metadata control plane. Apache Kafka KRaft metadata quorums explains how Kafka controllers coordinate cluster metadata; Connect’s internal topics and worker group handle a different layer of state.

Real-World Use Cases

Database change capture: A source connector can read committed changes and publish them to Kafka, where multiple consumer groups can update analytics, search, or caches. The PostgreSQL CDC explainer covers one source-side mechanism, while the Kafka event streaming guide explains how downstream groups consume and replay records.

Operational telemetry: A source connector can collect metrics or logs from external systems and place them in topics for alerting and analysis. Search and warehouse delivery: A sink connector can consume selected topics and write data to a search index or analytical store. In either direction, monitor both Connect task status and the source or destination’s own lag, throttling, and error signals.

Migration and integration: During a system migration, connectors can copy data and continue streaming changes while an application gradually moves between systems. The cutover still needs an explicit plan for schema compatibility, write ownership, lag, and rollback; a running connector alone does not guarantee that the two systems are fully synchronized.

Getting Started with a Local Distributed Worker

This example uses a local Kafka broker and the file source connector bundled with some Kafka distributions. It demonstrates Connect’s configuration and REST workflow, not a production pipeline. Start Kafka locally using the distribution’s quickstart, then create config/connect-distributed.properties with settings like these:

bootstrap.servers=localhost:9092
group.id=connect-local
config.storage.topic=connect-configs
offset.storage.topic=connect-offsets
status.storage.topic=connect-status
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
plugin.path=/absolute/path/to/kafka/libs
config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1

The replication-factor values are for a single-broker local demonstration only. A production cluster should use replication appropriate to its availability requirements and create or verify the internal topics with the intended partitioning and compaction settings. Replace the example plugin path with the absolute path to the directory containing the connector JAR, and install the file connector if it is not already included in the distribution.

From the Kafka installation directory, start the distributed worker:

bin/connect-distributed.sh config/connect-distributed.properties

In another terminal, create a small file before registering the source connector:

printf '%s\n' 'order-1001' 'order-1002' > /tmp/connect-input.txt

Register the connector with the Connect REST API on its default local port:

curl -sS -X POST http://localhost:8083/connectors \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "local-file-source",
    "config": {
      "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
      "tasks.max": "1",
      "file": "/tmp/connect-input.txt",
      "topic": "connect-demo"
    }
  }'

Then inspect the connector and task states, and read the resulting topic:

curl -sS http://localhost:8083/connectors/local-file-source/status

bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic connect-demo \
  --from-beginning \
  --max-messages 2

Look for RUNNING connector and task states, then confirm the two lines arrive in the topic. A FAILED task needs investigation in the Connect worker logs and the connector’s source-system settings; an HTTP 201 response from connector creation is not a health check. On Windows, use the distribution’s .bat scripts and a Windows path for the input file; the JSON connector configuration must use that same path. For production, also configure authentication, authorization, TLS, plugin compatibility, durable internal topics, and alerting for task failures and source lag.

Common Misconceptions

“A connector is the worker process.”

A connector instance describes and coordinates a job. Workers are the processes that host connector instances and execute their tasks. A distributed Connect cluster can contain several workers running tasks for many connector instances.

“Setting tasks.max guarantees that many parallel tasks.”

It is a limit, not a promise. The connector’s source or destination and its own partitioning strategy determine how much of the job can actually be divided.

“Distributed mode makes every pipeline exactly once.”

Distributed mode helps coordinate and recover task execution; it does not make an arbitrary external write atomic with Kafka offset progress. Use idempotent destination operations, deduplication, or a connector-specific guarantee where duplicate effects would be harmful. For Kafka-to-Kafka processing, Kafka transactions and exactly-once processing describes a narrower guarantee with explicit boundaries.

Related Articles

• Event streaming with Apache Kafka
• PostgreSQL logical replication and CDC
• Apache Kafka KRaft metadata quorums
• Kafka transactions and exactly-once processing

Changelog

• Initial publication.