The Incident Readiness Gap: How Engineering Organizations Fail When It Matters Most

When a large online service breaks, the instinct is to look for a broken machine. Nearly 40% of organizations have suffered a major outage caused by human error in the past 3 years, and 85% of those incidents trace back to people not following procedures or to the procedures themselves being inadequate. The failure is rarely the hardware. It is the gap between what a system needs in its worst moment and what the people responding to it have actually been prepared to do. That gap has a name worth taking seriously: incident readiness.

Semyon Slepov is a Site Reliability Engineer with 15 years spent building reliability into systems across online banking, e-learning, and large public-facing internet services. He has also served as a reviewer for International Conference on Computer Science, Technology and Engineering (ICETICS 2026), where he evaluates work on dependable systems. The pattern he keeps returning to is not about technology at all. It is about who in an organization actually knows how to respond when something goes wrong, and how few of them there usually are.

When a Few People Carry the System

The uncomfortable truth inside many engineering organisations is that incident response is a skill held by a handful of people. Around 65% of engineers reported burnout in the past year, and on-call duty is one of the largest contributors, because the same senior responders get paged again and again. When those few are on vacation, asleep, or simply gone, the organization is not as resilient as its architecture diagrams suggest. It is one unavailable person away from a slow, fumbling response.

Slepov saw this clearly at a company running a modern e-learning platform, where most incidents fell to a small group of tenured engineers who carried the institutional knowledge in their heads. The architecture was sound. The fragility was organizational. He took on the work of rebuilding the incident response process so that handling a failure did not depend on whether one specific person happened to be reachable.

"A system is only as reliable as the median engineer's ability to respond to it, not the best engineer's," Semyon Slepov says. "If your incident response lives in 3 people's heads, you do not have a reliable system. You have 3 people who are not allowed to take a real vacation."

Designing the Response Instead of Hoping for It

Most teams treat the response to failure as something that will simply happen when it is needed. The numbers say otherwise. When a deployment fails, 15% of teams need more than a week to recover, and a large share of organizations still operate without the documented runbooks, clear ownership, and rehearsed procedures that turn a crisis into a routine. Readiness is not a property teams have by default. It is something a few of them deliberately build.

Slepov approached the problem as a design task rather than a documentation chore. He studied how the strongest engineering organizations handled the same problem, then rewrote the incident documentation so it could be followed under stress, brought the runbooks back in line with how the systems actually behaved, and wired up the tooling integrations that incident management depends on. The aim was a process any on-call engineer could execute, not a binder that only made sense to the people who wrote it.

"Documentation that only the author understands is not documentation. It is a diary," Slepov notes. "The test of a runbook is whether someone who has never seen the system can follow it at 3 in the morning and reach the right outcome."

Practice Before the Pressure

The hardest part of incident response is that it is a high-stakes skill people are expected to perform without ever having practiced it. Pilots train in simulators precisely because the cockpit is the wrong place to learn. Engineering has been slower to adopt the same logic, even though the cost of a fumbled response compounds with every minute it runs.

The center of Slepov's revamp was a scenario-based training program. Small groups of engineers were handed a deliberately broken service, a "shadow" copy running in a staging environment rather than the systems serving live user traffic, and had to work the whole problem end to end: detect the issue through observability tools, declare the incident, drive it to mitigation, and write the postmortem. The point was not to test them but to build the reflex of responding before a real outage demanded it. Slepov also wrote a technical breakdown of how serious failures can hide inside ordinary-looking error logs, the kind of subtle signal these drills are built to teach responders to catch.

"Confidence under pressure is not a personality trait. It is a rehearsed skill," Slepov explains. "The first time an engineer declares an incident should not be during an actual incident."

Taking Blame Out of the Postmortem

There is a quieter problem that slows recovery long before the technical fix arrives: fear. When the engineer who wrote the offending change is the one expected to account for it, the postmortem becomes a defense rather than an investigation. Information gets withheld, root causes get softened, and the same failure returns later wearing a different mask.

Slepov pushed the organization to adopt blameless reviews and, in a more pointed change, to move ownership of the post-incident report away from the author of the breaking change and onto the incident responders. That single shift reframed the postmortem from a search for who was at fault into a study of how the system allowed the failure. A member of the IEEE Computer Society, Slepov has spent enough time around the research side of dependable systems to know that the organizations that learn fastest are the ones that make it safe to tell the truth about what broke.

"The moment a postmortem feels like a trial, you stop learning from it," Slepov observes. "You want people running toward the incident with the facts, not away from it with a lawyer."

Readiness as a System, Not a Hero

As more services lean on AI-driven traffic and less predictable load, the surface where things can quietly go wrong keeps growing, and the old model of a few people who know everything scales worse every year. The organizations that hold up will be the ones that treated readiness as infrastructure, built on purpose, rather than as a trait they hoped their best people would supply for free.

The results of Slepov's program were practical rather than dramatic: incidents were contained faster, and the responsibility for handling them spread across many engineers instead of resting on a few. He is now focused on extending the approach to environments where automated agents generate uneven, hard-to-predict load, and where the early signals of a building failure are even easier to miss than they used to be. The goal stays the same, which is to make the ordinary on-call engineer genuinely capable, not just the veterans.

"Reliability is not the absence of failure. It is an organization that does not fall apart when failure arrives," Slepov reflects. "The work is making sure that when something breaks at the worst possible moment, the response does not depend on luck, on one person's pager, or on anyone being a hero. It just depends on the system you built before you needed it."