Introduction to Apache Kafka Using Docker | iCert Global

Docker is a utility that allows you to execute applications within small, isolated environments known as containers. The containers are miniature computers where you can code, execute, and debug your applications without affecting your primary system.

Apache Kafka is software most big companies utilize to handle vast amounts of data in a timely manner. It is used in collecting logs, streaming real-time data, and passing messages between programs. It is used because it is efficient, scalable to handle more data, and is extremely reliable.

What is Docker Kafka and how do I use it?

In this article, we shall be describing what Kafka Docker is and how to uninstall Docker when working with Kafka. We shall also discuss the steps on how to run Apache Kafka through Docker.

If you are running Kafka using Docker, you will need to know some things such as how to use Docker, what is Advertised Port and Advertised Hostname, etc.

What Are Kafka Dockers?

Pre-requisites

Before applying Kafka using Docker, you have a few things to do beforehand. Let's note the fundamental steps:

1. Install Docker Compose:

• Docker Compose is available for Windows, MacOS, and 64-bit Linux.

2.assistant

Docker Engine:

• Docker Compose requires the Docker Engine in order to carry out most tasks. You have to install Docker Engine on your system, either locally or globally, depending on your environment.

3. Desktop Systems:

If you are on MacOS or Windows, Docker Compose typically comes pre-installed with Docker Desktop or Docker Toolbox. So, Mac users do not need to install Docker separately.

Installation of Kafka Docker

Once Docker Compose has been installed, you must set the KAFKA_ADVERTISED_HOST_NAME in your docker-compose.yml to the IP address of your Docker host.

If you wish to modify other Kafka settings, you can include them as environment variables within the same docker-compose.yml.

For instance, to modify how Kafka logs things (log4j), include environment variables that begin with LOG4J_.

You may switch ports and broker IDs through modification of the Docker Compose configuration file.

What are Broker IDs?

A broker ID assists you in locating all the Kafka brokers (servers) within the cluster. You can configure it in many ways:

Set the broker ID through the KAFKA_BROKER_ID variable.

• Utilize a command such as that to make the broker ID only:

bash

CopyEdit

BROKER_ID_COMMAND: "hostname | awk -F'-' '{print $2}'"

If you don't specify a broker ID, Docker Compose will generate one for you.

Broker IDs enable scaling up or down the number of brokers with ease.

Automatic Topic Generation in Kafka Docker

To have Kafka create topics programmatically when it starts, you must include a special configuration named KAFKA_CREATE_TOPICS in your docker-compose.yml.

Here's an example:

yml

CopyEdit

environment:

KAFKA_CREATE_TOPICS: "Topic1:1:3,Topic2:1:1:compact"

This means:

• Topic1 has 1 partition and 3 replicas.

• Topic2 has 1 partition and 1 replica, and it employs a cleanup mechanism of compact.

You can control how Kafka consumes this list of topics by specifying a separator between topic definitions with KAFKA_CREATE_TOPIC_SEPARATOR. You can use a new line for instance instead of a comma if you prefer your list to be more readable.

Make sure to apply the correct rules for putting these lines in docker-compose.yml because YAML files come with certain formatting requirements.

Promoted Hostname

The hostname in the advertisement informs Kafka where other applications can find it on the network.

You can establish it in numerous ways:

• Set KAFKA_ADVERTISED_HOST_NAME and print the hostname straight out.

• Or use a command to obtain the hostname automatically.

For instance:

yaml

CopyEdit

HOSTNAME_COMMAND: "route -n | awk '/UG[ \t]/{print $2}'"

If KAFKA_ADVERTISED_HOST_NAME is defined, it will be used in place of the command.

If you are running Kafka on AWS, you can automatically determine the host IP with this command:

bash

CopyEdit

wget --timeout=3 --tries=2 --quiet --output-document=- http://169.254.169.254/latest/meta-data/local-ipv4

You can also install the command in other Kafka configurations by typing _ {HOSTNAME_COMMAND} wherever necessary.

For instance:

yaml

CopyEdit

KAFKA_ADVERTISED_LISTENERS=SSL://_{HOSTNAME_COMMAND}:9093,PLAINTEXT://9092

This tells Kafka to utilize the IP address the command discovered.

Port Promoted

Kafka needs to be instructed which port (a number that helps computers to communicate) it should use.

You can select if the port is dynamic or static. You can have the port determined by a command using the PORT_COMMAND variable.

For example:

yaml

CopyEdit

KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://1.2.3.4:${PORT_COMMAND}

You can utilize the {PORT_COMMAND} string elsewhere in Kafka config to set it as the correct port number.

Examples

Some of the test commands and setups are:

1. Get the hostname automatically:

bash

CopyEdit

curl http://169.254.169.254/latest/meta-data/public-hostname

2. Set advertised listeners:

yaml

CopyEdit

INSIDE//9092,OUTSIDE://HOSTNAME_COMMAND://9094

3. Set listeners and security:

yaml

CopyEdit

KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094

KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT

4. Configure broker listeners:

yaml

CopyEdit

advertised.listeners = OUTSIDE://ec2-xx-xx-xxx-xx.us-west-2.compute.amazonaws.com:9094,INSIDE://:9092

listeners = OUTSIDE://9094,INSIDE://9092

inter.broker.listener.name = INSID

End

How to obtain Apache Kafka Certification? 

We are an Education Technology company providing certification training courses to accelerate careers of working professionals worldwide. We impart training through instructor-led classroom workshops, instructor-led live virtual training sessions, and self-paced e-learning courses.

We have successfully conducted training sessions in 108 countries across the globe and enabled thousands of working professionals to enhance the scope of their careers.

Our enterprise training portfolio includes in-demand and globally recognized certification training courses in Project Management, Quality Management, Business Analysis, IT Service Management, Agile and Scrum, Cyber Security, Data Science, and Emerging Technologies. Download our Enterprise Training Catalog from https://www.icertglobal.com/corporate-training-for-enterprises.php and https://www.icertglobal.com/index.php

Popular Courses include:

• Project Management: PMP, CAPM ,PMI RMP

• Quality Management: Six Sigma Black Belt ,Lean Six Sigma Green Belt, Lean Management, Minitab,CMMI

• Business Analysis: CBAP, CCBA, ECBA

• Agile Training: PMI-ACP , CSM , CSPO

• Scrum Training: CSM

• DevOps

• Program Management: PgMP

• Cloud Technology: Exin Cloud Computing

• Citrix Client Adminisration: Citrix Cloud Administration

The 10 top-paying certifications to target in 2025 are:

• Certified Information Systems Security Professional® (CISSP)

• AWS Certified Solutions Architect

• Google Certified Professional Cloud Architect 

• Big Data Certification

• Data Science Certification

• Certified In Risk And Information Systems Control (CRISC)

• Certified Information Security Manager(CISM)

• Project Management Professional (PMP)® Certification

• Certified Ethical Hacker (CEH)

• Certified Scrum Master (CSM) 

Conclusion

We have understood Kafka Docker and how useful it is through this post. With Docker, we can execute an Apache Kafka server inside a container, and our application can easily communicate with it through Docker's network.

Kafka Docker enables us to handle data in real time. It is appropriate for collecting a lot of data and delivering it immediately. It is also appropriate for quick microservices to store data securely and persistently. Additionally, Kafka Docker can deliver events to more advanced event streaming systems.

Contact Us For More Information:

Visit :www.icertglobal.com Email :