We are integrating sensors from three different manufacturers, and they all use different payload formats. One uses JSON, another uses Protobuf, and the third is a proprietary binary format. What is the most efficient way to normalize this data before it hits our analytics engine?
3 answers
The most scalable solution is to implement an "Abstraction Layer" using a message broker like EMQX or HiveMQ. You can write simple "Hooks" or use a serverless function (like AWS Lambda) to intercept the messages as they arrive. These functions act as a translator, converting the Protobuf or binary payloads into a standardized JSON schema that your data lake expects. This ensures that your downstream analytics team doesn't have to worry about which sensor the data came from. Standardizing early in the pipeline prevents "Data Swamp" issues and makes adding a fourth vendor in the future much easier.
Are you finding that the overhead of converting everything to JSON is significantly increasing your storage costs compared to keeping the binary data as-is in your S3 buckets?
I highly recommend looking into the Sparkplug B specification. It’s designed specifically to bring some order to the chaos of different MQTT payload formats.
I agree, Steven. Sparkplug B is gaining a lot of traction in the industrial space for exactly this reason—it provides a clear structure for state management.
We actually kept the raw binary data in a 'Bronze' layer for auditing purposes, Robert. However, the JSON conversion is only for the 'Silver' layer where our dashboards live. While it does increase storage slightly, the time saved by our data scientists not having to manually decode hex strings for every new project is worth every penny. We also started using Parquet for the final storage, which compresses the JSON very efficiently while keeping it easy for SQL engines to query.