I am working with sensor data that arrives every second. I heard that the bucket pattern is a standard for this. Can someone explain why this is better than storing each event as an individual document? I am specifically interested in how this affects index size and read performance for time ranges.
The bucket pattern optimizes time-series data storage by consolidating multiple readings into single documents to reduce index fragmentation, minimize storage overhead, and improve retrieval latency during range-based queries.
4 answers
Implementing the bucket pattern optimizes both storage efficiency and retrieval performance for time-series datasets by minimizing document metadata overhead.
- Individual documents carry significant per-record storage costs that inflate the index size over time.
- Aggregating events into a single bucket per period reduces the number of logical entries within the index.
- Read performance improves during range scans because the query engine fetches fewer documents to retrieve the same amount of historical data.
Yes, use the bucket pattern unless you want your indexes to explode and your read latency to go through the roof. Storing one document per second creates a massive index overhead that will eventually crush your WiredTiger cache and make range queries feel like they are running on a treadmill.
I recall back in 2017 when we migrated a telemetry ingestion engine that was essentially choking on individual event writes. We were seeing index sizes grow by gigabytes daily, which decimated our buffer pool efficiency and led to excessive page faults during retrieval tasks.
Switching to a bucket pattern, where we grouped sensor data by minute or hour, allowed us to maintain constant-time range scans while drastically reducing the metadata bloat associated with B-tree maintenance. It turned a failing system into one that could handle high-concurrency read requests without locking up our shards.
Storing every single event as an individual document is a fantastic way to burn money on infrastructure. You are basically paying for the overhead of the object ID and index pointers millions of times over when you could just aggregate them. If you care about your read performance or your sanity, stop trying to treat a document database like a relational row-based store and just bundle your time-series data into chunks.
