I am currently working on a development cluster and need to reset our environment. What is the most reliable way to delete all existing messages from a specific Kafka topic without actually deleting and recreating the topic itself? I’ve heard about retention policies and offset resets, but I want to ensure I don’t disrupt the consumer groups or the schema registry integrations.
3 answers
To clear messages without deleting the topic, the most common "SEO-friendly" hack is to temporarily adjust the retention time. You can use the kafka-configs.sh tool to set the retention.ms to something very low, like 1000 (1 second). Once the background cleaner thread runs, it will purge all segments that are older than that threshold. After the messages are gone, remember to revert the retention setting back to your original production value. This method is preferred over deleting the topic because it preserves all the metadata, partition settings, and security ACLs you already have in place.
Are you specifically trying to clear the data because of a schema mismatch or just to save disk space? Also, are you aware that if you change the retention settings, you might need to check your log.retention.check.interval.ms to see how fast the cleanup will actually trigger?
You should use the kafka-configs command to set retention.ms to 1. It is the cleanest way to purge data while keeping the topic structure and partition count intact for your team.
I agree with Michael. We used this exact approach during our last sprint when our staging brokers ran out of disk space due to a massive ingestion test. It worked perfectly without needing a restart.
Robert, that is a great point regarding the check interval. Usually, it defaults to 5 minutes, so the deletion isn't always "instant" after running the config command. If the user is in a hurry, they can also use the kafka-delete-records.sh tool with a JSON file specifying the offsets to move the "low watermark" to the end of the log, which effectively hides all old messages immediately.