My CTO has mandated a Disaster Recovery strategy where we can recover our primary production environment in under 60 minutes. We are currently on Google Cloud. Should we look at an Active-Active multi-region setup, or is a "Pilot Light" approach sufficient for these strict RTO/RPO requirements?
3 answers
Achieving a sub-hour RTO/RPO on GCP is definitely doable but requires careful data replication. A "Pilot Light" approach—where your database is live-replicating to another region but compute resources are off—can usually meet a 60-minute RTO if your infrastructure is fully scripted with Terraform. However, for the RPO to stay under an hour, you need asynchronous or synchronous database mirroring. We implemented this using Cloud Spanner for its global consistency. For a true "zero downtime" feel, Active-Active is better, but it doubles your infrastructure costs and complicates traffic routing via Global Load Balancing.
Are you planning to automate the failover process using health checks, or will there be a "human-in-the-loop" decision before shifting traffic to the secondary region during a failure?
The most important part is regular testing. A DR plan that hasn't been tested in a simulated outage is just a piece of paper. We run "Game Day" drills every quarter.
Exactly, Susan. Testing also helps identify bottlenecks in the recovery scripts that might otherwise go unnoticed until a real disaster strikes.
Jason, that is the million-dollar question. We are leaning toward a human-in-the-loop for the final cutover to avoid "split-brain" scenarios where both regions think they are primary. However, we are worried that the human delay might push us past our one-hour RTO goal if an outage happens at 3 AM. We need a very clear runbook and automated alerts to make sure the team reacts instantly.