Apache Kafka for Developers #10: Setting Up Kafka Locally with Docker

on
To set up Apache Kafka for development in KRaft mode using Docker

This configuration sets up a single Kafka broker running in KRaft mode

version: '3'
services:  
  kafka-broker-1:
    image: 'bitnami/kafka:3.3.1'
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker,controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=MY_CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,MY_CONTROLLER://:9094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=MY_CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_BROKER_ID=1
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka-broker-1:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka-broker-1:9092
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_KRAFT_CLUSTER_ID=d8zv92ecQk6ZrAAA35SDbw
    ports:
      - 9092:9092

As per above docker-compose.yml file

version: '3'
Specifies the version of the Docker Compose file format

services:

Defines the services that will be part of the Docker Compose app. Each service represents a container.

kafka-broker-1:
Indicates name of service and also name of the Kafka broker container name

image: 'bitnami/kafka:3.3.1'
Specifies the name of the Kafka docker image used for this service

environment:
Defines environment variables that will be passed to the Kafka broker node or container

KAFKA_ENABLE_KRAFT=yes
It Enables KRaft mode and allows Kafka to run without Zookeeper.

KAFKA_CFG_PROCESS_ROLES=broker,controller
It specifies the roles that a Kafka node can perform. Here, it is configured to act as both a broker and a controller

Broker Role:
A broker handles client requests like producing and consuming messages, stores data on disk, manages topics, partitions, and logs, handles client connections, and replicates data across other brokers.

Controller Role:
A controller manages the metadata for the Kafka cluster, including brokers, topics, and partitions. It elects partition leaders, manages topic creation and deletion, and ensures metadata consistency across the cluster.

When a Kafka process is configured to act as both a broker and a controller, it handles both client requests and metadata management. This setup can simplify the deployment by reducing the number of separate processes needed

KAFKA_CFG_CONTROLLER_LISTENER_NAMES=MY_CONTROLLER
It specifies the names of the listeners that the Kafka controller will use to communicate with each other and with brokers.
The property is a comma-separated list of listener names(you can use any names). These names must match the listener names defined in the KAFKA_CFG_LISTENERS property

KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,MY_CONTROLLER://:9094
It defines the listeners for the Kafka brokers and specifies how it will accept connections from clients and other brokers.
It supports various protocols like PLAINTEXT, SSL, SASL and etc.

It supports various protocols like PLAINTEXT,SSL, SASL and etc

Format :
{Protocol}:{Host}://{Port}

Example:
PLAINTEXT:10.0.0.1//:9092   ---> listen for 10.0.0.1
PLAINTEXT://:9092                ---> listen for all available network interfaces

PLAINTEXT Listener:
This listener will accept plaintext (unencrypted) connections on port 9092. It is typically used for client connections (producers and consumers).

MY_CONTROLLER Listener:
This listener will accept connections on port 9094 specifically for controller communication. It is used for internal communication between Kafka controllers and brokers

KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=MY_CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
It specifies which security protocol each listener will use. Here, both MY_CONTROLLER and PLAINTEXT listeners will use the plaintext protocol.

This setting ensures that each listener uses the appropriate security protocol.

MY_CONTROLLER:PLAINTEXT
This listener will use the plaintext protocol, meaning the communication is unencrypted. This listener is typically used for internal communication between Kafka controllers and brokers.

PLAINTEXT:PLAINTEXT 
This listener will also use the plaintext protocol. This listener is typically used for client connections (producers and consumers) and broker-to-broker communication.

KAFKA_CFG_BROKER_ID=1
Sets the broker ID to 1. Each broker in a Kafka cluster must have a unique ID

KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka-broker-1:9094
This is used in Apache Kafka's KRaft mode to define the set of controller nodes that participate in the quorum for voting to elect the Controller or Leader. Here, it specifies that the controller quorum consists of the broker itself, listening on port 9094.

Format: node_id@host:port

Example:
if you have three Kafka nodes acting as controllers, the configuration might look like this

KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka-broker-1:9094,2@kafka-broker-2:9094,3@kafka-broker-3:9094 
The above specified nodes form the quorum. A majority of these nodes must agree on any changes to the cluster metadata

KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka-broker-1:9092
It specifies the addresses that clients and other brokers used to connect to the given node.

This listener will advertise the address kafka-broker-1:9092 to clients and other brokers. Clients and brokers will use this address to connect to the Kafka broker

ALLOW_PLAINTEXT_LISTENER=yes
This configuration allows Kafka to start with plaintext listeners defined in the KAFKA_CFG_LISTENERS property, which means that the communication between clients and the broker, as well as between brokers, is not encrypted. 

This setting is typically used for development and testing environments

KAFKA_KRAFT_CLUSTER_ID=d8zv92ecQk6ZrAAA35SDbw
It is typically a base64 format of UUID used to identify the Kafka cluster.This ID is crucial for the KRaft mode, where Kafka operates without Zookeeper and uses a quorum-based consensus algorithm for metadata management.

This is important for ensuring that all nodes in the cluster recognize each other and can participate in the quorum.

It can be generated using kafka-storage.sh tool provided with Kafka
> bin/kafka-storage.sh random-uuid

You can also use below Online tool for generating random uuid
https://www.fileformat.info/tool/guid-base64.htm

    ports:
      - 9092:9092
Maps port 9092 on the host to port 9092 on the container, allowing external access to the Kafka broker 

Format:  host_port:container_port

docker compose -f "docker-compose.yml" up -d --build // run to build and start the container


Kafka UI Docker image

The Kafka UI Docker image from Provectus Labs makes it easy to set up a web-based interface for managing and monitoring your Kafka clusters.

Use the below configurations to start the Kafka UI docker container
  kafka-ui:
    container_name: kafka-ui
    image: 'provectuslabs/kafka-ui:latest'
    ports:
      - "8080:8080"
    environment:
      - KAFKA_CLUSTERS_0_BOOTSTRAP_SERVERS=kafka-broker-1:9092
      - KAFKA_CLUSTERS_0_NAME=d8zv92ecQk6ZrAAA35SDbw    



Here's the complete Docker Compose file for a single Kafka broker running in KRaft mode

version: '3'
services:  
  kafka-broker-1:
    image: 'bitnami/kafka:3.3.1'
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker,controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=MY_CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,MY_CONTROLLER://:9094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=MY_CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_BROKER_ID=1
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka-broker-1:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka-broker-1:9092
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_KRAFT_CLUSTER_ID=d8zv92ecQk6ZrAAA35SDbw
    ports:
      - 9092:9092
  kafka-ui:
    container_name: kafka-ui
    image: 'provectuslabs/kafka-ui:latest'
    ports:
      - "8080:8080"
    environment:
      - KAFKA_CLUSTERS_0_BOOTSTRAP_SERVERS=kafka-broker-1:9092
      - KAFKA_CLUSTERS_0_NAME=d8zv92ecQk6ZrAAA35SDbw      




Here's the complete Docker Compose file for multiple Kafka brokers running in KRaft mode

version: '3'
services:  
  kafka-broker-1:
    image: 'bitnami/kafka:3.3.1'
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker,controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=MY_CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,MY_CONTROLLER://:9094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=MY_CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_BROKER_ID=1
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka-broker-1:9094,2@kafka-broker-2:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka-broker-1:9092
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_KRAFT_CLUSTER_ID=d8zv92ecQk6ZrAAA35SDbw
    ports:
      - 9092:9092
  kafka-broker-2:
    image: 'bitnami/kafka:3.3.1'
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker,controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=MY_CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,MY_CONTROLLER://:9094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=MY_CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_BROKER_ID=2
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka-broker-1:9094,2@kafka-broker-2:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka-broker-2:9092
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_KRAFT_CLUSTER_ID=d8zv92ecQk6ZrAAA35SDbw
    ports:
      - 9093:9092      
  kafka-ui:
    container_name: kafka-ui
    image: 'provectuslabs/kafka-ui:latest'
    ports:
      - "8080:8080"
    environment:
      - KAFKA_CLUSTERS_0_BOOTSTRAP_SERVERS=kafka-broker-1:9092
      - KAFKA_CLUSTERS_0_NAME=d8zv92ecQk6ZrAAA35SDbw


docker compose -f "docker-compose.yml" up -d --build // run to build and start the container

Apache Kafka for Developers Journey:

Happy Coding :)

Comments

Post a Comment