8 Key Upgrades: How Cloudflare's 'Fail Small' Overhaul Boosts Network Resilience

Over the past two and a bit quarters, Cloudflare undertook an intensive engineering effort internally code-named "Code Orange: Fail Small." The goal: make its infrastructure more resilient, secure, and reliable for every customer. Earlier this month, the team completed this work. While resilience is never truly done, this project specifically addressed the root causes of the November 18, 2025, and December 5, 2025, global outages. The result is a stronger network that can better withstand failures. Here are eight key things you need to know about this transformative overhaul.

1. Health-Mediated Configuration Rollouts

Cloudflare internal configuration changes no longer reach the network instantly. Instead, they are rolled out progressively with real-time health monitoring. This allows observability tools to catch problems and revert issues before they affect customer traffic. To catch dangerous deployments before production, the team identified high-risk configuration pipelines and built new tools to manage changes better. For products processing customer traffic, configuration changes now follow a "health-mediated deployment" methodology—the same used when releasing software. Central to this is a new internal component called Snapstone, which bundles configuration changes into packages and gradually releases them with health mediation. This closes the gap where per-team effort was previously inconsistent.

2. Reducing the Blast Radius of Failures

The project focused on limiting the impact of any single failure. By designing configurations to affect smaller segments of the network, a problem with one change won't cascade globally. This approach includes compartmentalizing data and control flags so that even if a mistake slips through, only a subset of traffic is affected. This reduces the blast radius from a total outage to a localized issue that can be quickly resolved. For customers, this means higher overall uptime and less risk of widespread disruptions.

3. Revamped Break Glass Procedures

Emergency access procedures—"break glass"—were revised to ensure that when human intervention is needed during critical incidents, it doesn't introduce additional risks. The new procedures include stricter controls, better logging, and mandatory peer review before emergency changes go live. This prevents the very rare but dangerous scenario where a rushed fix inadvertently causes more harm. The balance between speed and safety is now tightly managed.

4. Strengthened Incident Management

Incident response processes were overhauled to handle large-scale problems more effectively. This includes clearer role definitions, faster escalation paths, and automated runbooks that reduce human error. Training exercises now simulate past outage scenarios to ensure teams are prepared. The result is a more disciplined, predictable response that minimizes downtime and gets services back online faster.

5. Proactive Drift and Regression Prevention

To prevent the network from slowly degrading over time, Cloudflare introduced measures to detect and block configuration drift. Automated systems now regularly compare current configurations against known good baselines and flag any unauthorized or accidental changes. Regression testing is also integrated into the deployment pipeline, so new updates must pass strict compatibility checks. This ensures that fixes introduced by Code Orange remain effective long-term.

6. Real-Time Health Monitoring Integration

Health monitoring is now deeply woven into every configuration change. Custom dashboards and alerting systems provide instant feedback on traffic patterns, error rates, and latency. If any metric deviates from expected thresholds, automated rollbacks trigger within minutes—long before customers notice issues. This real-time feedback loop transforms safety from a manual check into an automated safeguard.

7. Unified Configuration Management with Snapstone

Snapstone is not just for high-risk pipelines; it provides a unified way to apply health-mediated deployment to any configuration unit. Teams can dynamically define what needs health mediation—whether it's a data file like the one that caused the November outage or a control flag from December. By standardizing the process across all teams, Snapstone eliminates inconsistency and ensures that every configuration change benefits from progressive rollout and automatic rollback.

8. Improved Customer Communication During Outages

Finally, Cloudflare strengthened how it communicates with customers during incidents. New systems provide more frequent, transparent updates via status pages and direct notifications. The language is clearer, with less jargon, and includes expected resolution times and root cause explanations as they are known. This rebuilds trust by keeping customers informed even when things go wrong.

The completion of Code Orange: Fail Small marks a significant milestone in Cloudflare's journey toward unparalleled reliability. While no network can be perfect, these eight upgrades dramatically reduce the likelihood of repeat outages and ensure that when failures do occur, they are small, contained, and quickly resolved. For customers, this means a stronger, more trustworthy Cloudflare network—one that consistently delivers on its promise of performance and security.