I am auditing our current continuous delivery process and I want to ask the right questions to uncover hidden technical debt. What are the top 10 questions I should be asking the engineering managers? I want to focus on areas like:
- Deployment frequency
- Mean time to recovery (MTTR)
- Change failure rates
What are the 'gotcha' questions that reveal if the pipeline is fragile?
Effective delivery pipeline auditing requires assessing end-to-end traceability, artifact immutability, secret management security, environment reproducibility, and the reduction of manual dependencies to eliminate configuration drift and minimize recovery times.
7 answers
When examining CI/CD infrastructure, I look for the hidden coupling that creates fragility. Ask these questions to determine if your pipeline is actually scalable:
- Can you trace a production bug back to a specific commit hash and the associated build environment configuration?
- How does the pipeline handle infrastructure state changes, and what is the recovery time from a corrupted state file?
- Are your CI/CD runner agents ephemeral and hardened, or are they persistent nodes that accumulate drift?
- What happens to your deployment if your artifact registry goes down?
- How do we validate that our pipeline security policies are enforced at every branch?
- Are environment variables for production secrets injected at runtime via a vault, or are they stored in the pipeline configuration?
- What is the ratio of automated tests to manual regression tests in your current release cycle?
- How long does it take for a security patch to reach production from the time it is released by the upstream vendor?
- Are there any hard dependencies between parallel jobs that cause bottlenecks in the pipeline graph?
- If we had to migrate to a new CI/CD provider tomorrow, how much of our logic is locked into proprietary plugins?
Fragility is usually found in the manual workarounds used to bridge these gaps.
Look, stop chasing vanity metrics like 'deployment frequency' if your team is just pushing garbage faster. The pipeline isn't a race; it's a filtration system. If you want to find the real rot, ask these:
- Can a new engineer push to production on their first day without a senior holding their hand? If no, your pipeline is a secret club, not a tool.
- How many manual 'click-ops' steps exist between commit and deploy? Every mouse click is a point of failure.
- What is the process for a rollback? If the answer is 'fix-forward,' you are lying to yourselves.
- Does the pipeline have a 'kill switch' that actually works, or is that just a slide in a deck?
- Who is paged when the build fails? If it's a designated 'build sheriff' rather than the dev who broke it, you have a culture problem.
Most managers will tell you what you want to hear. Go check the logs yourself. The logs never lie, even when the managers do.
Auditing a pipeline requires looking past the UI to the underlying architecture. When I perform these assessments, I focus on the friction points that impede throughput. Here are the questions that expose systemic fragility:
- What is the current percentage of flaky tests in the suite, and what is the policy for managing them?
- How long does a developer wait from commit to pipeline completion?
- Is the environment configuration immutable, or are we relying on drift-prone manual patching?
- What is the mean time to detect a failure versus the time to recover?
- How are dependencies managed, and do we have a cached mirror for external artifacts?
- Can we recreate the entire environment from scratch in under an hour?
- Is there a documented post-mortem for the last three incidents, and were the remediation items implemented?
- How do we ensure that secret management isn't hardcoded or insecurely stored in the VCS?
- Is there a clear separation between build time, test time, and deployment time?
- How do we verify that the code deployed to staging is identical to the production artifact?
If they cannot answer these with data-backed responses, the debt is likely deeper than the code itself.
Mandy Harris, these points are incredibly helpful for my upcoming audit. I've been struggling to document our secret management practices, and your list gives me a much better framework to follow.
Efficiency is meaningless without observability. If you cannot measure the cost and speed of every stage, you are just guessing. These ten questions will strip away the veneer of your delivery process:
- What is the cost per deployment in terms of compute and engineer time?
- How many times has a hotfix been deployed because the primary pipeline failed?
- Are we using dynamic analysis tools or just relying on linting?
- What is the percentage of successful builds without manual intervention?
- Do we have a rollback artifact that is guaranteed to be stable?
- How many different configurations are currently being managed manually?
- What is the average time taken to provision a new testing environment?
- How are we alerting on pipeline performance degradation?
- Is the pipeline source code version-controlled and peer-reviewed?
- Are the delivery logs centralized and searchable for audit purposes?
If you aren't tracking these, you're flying blind.
Sorry to bother, but this is exactly what I needed. Theresa, I feel like we’re definitely flying blind right now, and these metrics might finally help me prove we need better tooling.
Theresa Rivera, I really appreciate you highlighting observability. Dealing with constant production fires, I definitely need to centralize our delivery logs to see why we keep falling back on hotfixes.
In my experience, teams often hide debt behind 'process.' If you want to see how fragile the system is, watch them during an outage. Ask these:
- What is the specific MTTR for a production database migration failure?
- Do we have automated smoke tests that run post-deployment, or do we rely on user reports?
- When was the last time the entire pipeline failed, and what did we learn?
- How many environments do we have, and are they parity-tested?
- Who holds the 'keys to the kingdom' for environment configuration?
- What is the fail-safe if the automated deployment stalls mid-way?
- How do we audit the changes being made by the pipeline?
- Are our pipeline tools updated regularly, or are we lagging by three major versions?
- What is the impact of a 30-minute pipeline outage on our business revenue?
- Can we deploy a fix without merging to main?
If the answers are vague, you've found your debt.
Audit the pipeline by testing the assumptions. If the pipeline is fragile, it will crack under these questions:
- Does the pipeline have a single point of failure (e.g., one specific Jenkins controller)?
- Are build artifacts signed and verified before deployment?
- Is there a mechanism to prevent code from being deployed if security scans fail?
- How do we handle rollbacks when database schemas have already changed?
- Are we using infrastructure as code (IaC) consistently for the pipeline itself?
- What is the total duration of the build-test-deploy loop?
- How often do we perform 'Game Days' to practice recovery?
- Are we running tests in parallel, or are we limited by sequential execution?
- Does the pipeline support feature flags, or is every deployment a big-bang release?
- How do we ensure the pipeline runner has the least privilege required?
Keep it simple. Measure, iterate, and remove the manual steps.
I don't care about the frequency; I care about the predictability. Here are the questions that expose the technical debt in your delivery lifecycle:
- What is the success rate of the first build attempt after a commit?
- Can you identify every change currently running in production via the Git commit log?
- What is the blast radius of a failed deployment?
- Are we using ephemeral runners, or is there a 'persistent' build server that needs manual cleaning?
- What is the MTTR specifically for infrastructure-as-code failures?
- Are our tests deterministic, or do we have 'flaky' tests that are ignored?
- How do we handle stateful application deployments during rolling updates?
- Is the pipeline configuration stored in the same repository as the application code?
- What is the policy for managing pipeline secret rotation?
- Can we deploy a previous version of the application with a single command?
If they can't answer these, start by standardizing the build environment and moving to GitOps. Everything else is secondary.
Kenneth Williamson, that bit about 'click-ops' really hit home for me. I've been googling how to automate our manual steps for weeks, and your point about the 'build sheriff' makes total sense.