Apache Kafka provides an unified, high-throughput, low-latency platform for handling real-time data feeds. Installing Apache Kafka, especially the right configuration of Kafka Security including authentication and encryption is kind of a challenge. This should give a brief summary about our experience and lessons learned when trying to install and configure Apache Kafka, the right way.

In my last post I wrote about first steps and lessions learned when setting up Apache Kafka with encryption, SASL SCRAM/Digest authentication and ACL authorization using Confluent Platform. This secures Kafka using SASL SCRAM between clients and Kafka Brokers and SASL MD5 digest between Kafka Brokers and ZooKeeper. This approach has some drawbacks i.e. the passwords must be stored on the clients and ZooKeeper is using MD5 hashes for passwords on the wire. So we try another approach by using Mutual TLS (mTLS) only, which seems a bit easier and which also seems to be suitable for a corporate environment.