I am working with sensor data that arrives every second. I heard that the bucket pattern is a standard for this. Can someone explain why this is better than storing each event as an individual document? I am specifically interested in how this affects index size and read performance for time ranges.
The bucket pattern optimizes time-series data storage by consolidating multiple readings into single documents to reduce index fragmentation, minimize storage overhead, and improve retrieval latency during range-based queries.
4 answers
Implementing the bucket pattern optimizes both storage efficiency and retrieval performance for time-series datasets by minimizing document metadata overhead.
- Individual documents carry significant per-record storage costs that inflate the index size over time.
- Aggregating events into a single bucket per period reduces the number of logical entries within the index.
- Read performance improves during range scans because the query engine fetches fewer documents to retrieve the same amount of historical data.
Yes, use the bucket pattern unless you want your indexes to explode and your read latency to go through the roof. Storing one document per second creates a massive index overhead that will eventually crush your WiredTiger cache and make range queries feel like they are running on a treadmill.
I recall back in 2017 when we migrated a telemetry ingestion engine that was essentially choking on individual event writes. We were seeing index sizes grow by gigabytes daily, which decimated our buffer pool efficiency and led to excessive page faults during retrieval tasks.
Switching to a bucket pattern, where we grouped sensor data by minute or hour, allowed us to maintain constant-time range scans while drastically reducing the metadata bloat associated with B-tree maintenance. It turned a failing system into one that could handle high-concurrency read requests without locking up our shards.
I was just frantically researching this exact telemetry problem, Parth Shenoy. Your point about metadata bloat is spot on. I noticed those same page fault spikes and feel a bit better knowing I'm on the right track.
Storing every single event as an individual document is a fantastic way to burn money on infrastructure. You are basically paying for the overhead of the object ID and index pointers millions of times over when you could just aggregate them. If you care about your read performance or your sanity, stop trying to treat a document database like a relational row-based store and just bundle your time-series data into chunks.
Ronith Prabhakaran, you’re totally right about the costs. I’m drowning in document overhead right now, so hearing that we need to stop treating this like a relational store is exactly the wake-up call I needed.
I appreciate the blunt advice, Ronith Prabhakaran. I have been worried about my aggregation strategy, but seeing your perspective on index pointer overhead confirms that grouping into chunks is the safer, more logical path forward.


Parth Shenoy, your 2017 experience sounds so impressive. I’m honestly just terrified I'll mess up our own migration, but reading about your bucket pattern makes me feel slightly less hopeless about our B-tree issues.