Our social media app’s "likes" table is growing by 5 million rows a day. We’ve already upgraded our RDS instance to the largest size available. Should we start implementing application-level sharding now, or is there a better way to handle this write-heavy load without making the code too complex?
3 answers
Sharding is the "nuclear option." It gives you infinite scalability but at a huge cost in application complexity—joins across shards become nearly impossible and re-sharding later is a nightmare. Before going there, I'd suggest looking at "Read Replicas" for the read load and "Write-Through Caching" using Redis to batch your updates. In 2023, we faced this and instead of sharding, we moved to a "Distributed SQL" database like CockroachDB. It handles the sharding automatically at the storage layer while giving you a standard SQL interface. It saved us from rewriting our entire backend logic while still giving us the horizontal growth we needed.
Is your data naturally partitionable by something like UserID or Region, which would make a sharding strategy much simpler to implement at the architectural level?
Vertical scaling is always cheaper in terms of "human hours." If you can buy a bigger server, do it. Only shard when you literally cannot buy a bigger box.
Mary is right. Sharding introduces "technical debt" that you'll be paying off for years. Max out your hardware and optimize your queries first.
Daniel, yes, we can definitely partition by UserID. Most of our queries are user-specific anyway. My concern is the "Global Feed" or analytics. If I shard by UserID, how do I run a query to find the "Top 10 most liked posts globally" without querying every single shard and aggregating the results in the application? That sounds like it would be incredibly slow and resource-intensive for the web server to handle.