I am currently testing a real-time data pipeline and I need to clear out all the existing messages in a specific Kafka topic to start fresh with a new test dataset. Since Kafka doesn't have a simple "truncate" command like a SQL database, what is the standard procedure to delete all records without deleting the entire topic configuration? I've seen some methods involving retention time, but I'm worried about the delay in execution.
3 answers
The most common way to "empty" a topic without deleting the topic itself is to temporarily lower its retention period. You can use the kafka-configs.sh tool to set the retention.ms to a very small value, such as 1000 (1 second). The command is: kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name your_topic --add-config retention.ms=1000. Once applied, Kafka’s log cleaner will eventually flag the segments as expired and delete them. After the messages are gone, you must remember to set the retention back to its original value, or you'll lose new data instantly. This method is safe but not instantaneous as it depends on the cleaner's schedule.
Is there a reason you aren't using the kafka-delete-records command? It allows you to specify a JSON file with a target offset (setting it to -1 usually clears the partition) and it tends to trigger the deletion logic much faster than waiting for a retention timeout.
If your cluster allows it, just delete the topic and recreate it. It’s the only way to be 100% sure everything is wiped including internal indexes, though you lose all custom metadata.
I agree with David, but only for development! If you are in production, deleting a topic can cause a "Leader Not Available" error for active producers and consumers. If the environment is live, stick to the retention policy shift or the delete-records script to keep the connection stable.
You're absolutely right, Steven. For those who want more control, creating a JSON file like {"partitions": [{"topic": "test", "partition": 0, "offset": -1}], "version":1} and running kafka-delete-records.sh is the "surgical" way to do it. It updates the low watermark of the log, making the old data inaccessible to consumers immediately. It’s definitely the preferred method in high-throughput environments where you can't afford to wait for the background retention thread to wake up.