Our engineers are spending 60% of their time on repetitive manual tasks like ticket approvals and manual patching. As we scale our <DevOps & SRE> practices, how can we identify and eliminate this 'toil' effectively without burning out the current engineering staff?
3 answers
Toil reduction is the heartbeat of SRE. Start by having your team log every manual task they perform over a two-week period. Once you have the data, categorize them by frequency and complexity. Target the high-frequency, low-complexity tasks first for automation. This could be as simple as writing a Python script to handle user permissions or using Ansible for patch management. The goal is to cap toil at 50% of an engineer's time, ensuring they have the remaining 50% to focus on high-value project work that actually improves the system's long-term scalability and resilience.
Does anyone have experience with AI-driven AIOps tools helping with toil, or do they usually just create more 'noise' and false positives that the team then has to investigate?
Self-service portals are a game changer. If developers can provision their own environments, it removes the SRE team as a bottleneck for basic requests.
Exactly, Martha. Shifting left by empowering developers not only reduces toil but also fosters a culture of shared responsibility for the entire infrastructure.
Raymond, they can be a double-edged sword. If you don't tune your thresholds correctly, you'll end up with alert fatigue. I suggest starting with rule-based automation before moving to ML-based anomaly detection so that the team understands the underlying logic of the "automated" decisions.