In our Big Data pipeline, we ingest millions of events per second via Kafka. We're seeing a lot of "dirty data"—missing fields, incorrect types, and late-arriving records. How do you implement data quality checks at scale without killing the throughput? Are you using Great Expectations, or is it better to build custom validation inside the Flink/Spark jobs?
3 answers
The most efficient way to handle this at the "Velocity" you're describing is to use a Schema Registry (like Confluent's). By enforcing Avro or Protobuf schemas at the producer level, you prevent "malformed" data from ever entering the Kafka topic. For the logic-level "dirty data" (like negative prices), we use a "Dead Letter Office" (DLO) pattern. Inside our Spark Streaming job, we wrap our transformations in a try-catch block. Valid records go to the main sink, and invalid ones are routed to a separate "Error" topic with the original metadata. This keeps the pipeline moving while allowing us to audit the bad data later.
That handles the structure, but what about the "late-arriving" data? If a sensor goes offline and dumps 5 hours of data at once, it can mess up your windowed aggregates. How do you handle "Watermarking" to ensure your 5-minute averages remain accurate?
We use Great Expectations, but only on the "Batch" side of our Lambda architecture. Trying to run it on every single stream event was way too slow for our SLA
Same here, Lisa. We use the "Streaming" job for immediate action and then run the deep "Great Expectations" suite on the Data Lake every hour to catch the subtle drift issues.
Mark, in Flink, we set a "Max Out-of-Orderness" watermark. If data is later than that threshold, we simply drop it or send it to a "late-data" side output for manual reconciliation. For the "incorrect types" Susan mentioned, we actually use a light-weight Python validation script running in a Lambda function via a Kafka Connect transform. It adds about 5ms of latency but catches 90% of the logical errors before they even hit our processing cluster, which saves us a lot of expensive "re-processing" compute time.