In earlier posts (1, 2), we've described the different use cases for Apache Kafka and the supported method for Kafka authentication based on the well known Ansible playbooks for the Confluent Platform. Since this deployment method is hardly flavored by Confluent and it deploys the Confluent Server instead of Apache Kafka by default, it is maybe a better choice, to deploy Apache Kafka from scratch.
Keycloak is a versatile Opensource Identity Provider. Amongst all its features, there is the possibility of building a High-Availability Cluster. Keycloak itself has two different storages: one for persistent data, the database, mostly MariaDB or Postgres, and one for more frequently accessed data such as sessions and used action tokens (InfiniSpan).
We build an universal deployment tool for Kubernetes that supports rendering and deployment of lightweight Jinja templated k8s manifests as well as complex Helm charts. We've added support for easy secret management based on Gopass, running tests in CI/CD pipelines, extending upstream Helm Charts with custom Jinja-templates manifests as well as patching upstream Helm Charts before deploying.
Apache Kafka provides an unified, high-throughput, low-latency platform for handling real-time data feeds. Installing Apache Kafka, especially the right configuration of Kafka Security including authentication and encryption is kind of a challenge. This should give a brief summary about our experience and lessons learned when trying to install and configure Apache Kafka, the right way.
In my last post I wrote about first steps and lessions learned when setting up Apache Kafka with encryption, SASL SCRAM/Digest authentication and ACL authorization using Confluent Platform. This secures Kafka using SASL SCRAM between clients and Kafka Brokers and SASL MD5 digest between Kafka Brokers and ZooKeeper. This approach has some drawbacks i.e. the passwords must be stored on the clients and ZooKeeper is using MD5 hashes for passwords on the wire. So we try another approach by using Mutual TLS (mTLS) only, which seems a bit easier and which also seems to be suitable for a corporate environment.