My organization requires a 99.99% uptime for our Kafka infrastructure. How should we set up MirrorMaker 2 or Confluent Cluster Linking to ensure that if an entire AWS or Azure region goes down, our producers and consumers can failover to a secondary region with minimal data loss (RPO/RTO)?
3 answers
For multi-region DR, MirrorMaker 2 (MM2) is the standard open-source tool. It uses the Connect framework to replicate topics, configurations, and ACLs. However, for a 99.99% SLA, you should look at "Active-Passive" or "Active-Active" setups. Cluster Linking is often preferred in Confluent environments because it replicates the exact offsets, which simplifies consumer failover significantly. Without offset preservation, your consumers might reprocess a lot of data upon failover. Always ensure your "acks=all" setting is enabled on producers to guarantee that data is durable before the replication occurs.
Have you calculated the potential cost of cross-region data transfer fees, as these can become quite expensive when replicating high-throughput topics?
We use MirrorMaker 2 for our cross-cloud replication and it works well, though the initial setup of the heartbeat and checkpoint topics can be tricky.
I agree, Nancy. Once the checkpointing is tuned, the RPO is usually just a few seconds, which fits most enterprise DR requirements perfectly.
Joseph makes a great point. To save on costs, we only replicate "critical" topics for DR. Using compression like Zstd can also help reduce the bandwidth needed for cross-region replication in the cloud.