Notice: Microsoft has not published a full writeup on what happened. Only a basic timeline of what they have found so far.
“When 90% of your infrastructure code is written by an algorithm and you’ve laid off the humans who built the original plumbing, who the hell is left to perform a black start when the machines go dark?” — Dispatches from the War Room | #Infrastructure
I think we have officially reached a crossroads in how we build software and deploy critical infrastructure. If you’ve been paying attention over the last few days, it feels like the digital duct tape holding modern civilization together is starting to lose its stickiness all at once.
In the span of a single week, we’ve watched a worldwide email outage bring global business to a screeching halt, a complete ground stop of all aircraft, and a major bank outage that rippled outward to infect other financial institutions like a bad oyster at a buffet. While the forensic details of the aviation ground stop and the banking domino effect are still buried under crisis PR teams, the postmortem details of Microsoft’s monster email outage are finally dripping out via The Register.
Spoiler alert: it wasn’t North Korean hackers, solar flares, or a backhoe through a fiber line. It was a latent software defect introduced inside a July “fix” that sat quietly in production for four months like an unexploded landmine. When a traffic surge from a Teams failover collided with reduced network gateway capacity, that dormant July bug triggered severe lock contention, starved the servers of resources, and set off a runaway cascading failure.
Once the fleet flatlined, Microsoft engineers were forced into one of the most grueling operations in systems engineering: bringing a massive, dead infrastructure back online from a black start.
The Anatomy of a Black Start (Why You Can’t Just Flip the Switch)
If you want to understand why massive cloud outages take hours or days to recover from even after engineers find the bug, you have to borrow a concept from the electrical utility industry.
In the power grid world, a black start is a bad scenario where operators have to bring a collapsed electrical grid back online from absolute zero. In practice, total black starts are rare, but they happen after a runaway cascading failure i.e: something causes a few generating stations to trip offline, which instantly shunts their electrical load onto neighboring stations. Those stations can’t handle the sudden surge, so their breakers trip in turn. Once that domino effect starts, an entire regional power grid can collapse into total darkness in about sixty seconds.
Once the grid is dead, you cannot simply walk over to a power plant and start a generator straight into an offline grid. If you try, that solitary generator immediately gets slammed with 100% of the region’s demand, the AC frequency craters out of tolerance, and multi-million-dollar turbines physically tear themselves apart. Instead, grid operators have to physically disconnect all customer loads, spin up isolated generation, and painstakingly bring “load blocks” back online in sequence, constantly balancing available megawatts against the size of the next neighborhood they flip on.
Hyperscale cloud outages work the exact same way:
The Trip: A handful of gateway servers get into trouble. Maybe they hit a latent lock-contention bug, enter a crash loop, or lose their warm caches.
The Cascade: The load balancers see unhealthy nodes and dutifully shift their traffic onto the remaining servers, which are already redlining. Those servers immediately choke and fall over too. Within seconds, the entire fleet is dead.
The Thundering Herd: You can’t just reboot the servers and open the floodgates. If you bring cold servers online straight into the teeth of a hundred million furious Outlook clients aggressively spamming retry requests, the fleet gets crushed before its caches and JIT compilers even have time to warm up.
To survive a digital black start, engineers have to sever all incoming customer traffic, bring the backend servers up into a stable idle state, and trickle load blocks back online in small chunks. And you can’t do that naively. If you just tell your load balancer to randomly drop 90% of incoming HTTP requests, nobody gets a working service because loading a single inbox takes dozens of sequential requests, meaning 100% of users just get broken error screens. Your load balancers have to be smart enough to shed traffic atomically, like dropping traffic by the modulus of a session cookie so that the 10% of users you let through get a fully working session while you slowly dial the grid back up.
The Circular Dependency Trap
As veteran SREs from hyperscalers like Google have pointed out in postmortem discussions, big tech teams routinely spin up new clusters and shift live traffic between regions. Everything looks resilient on a slide deck. But actually starting an isolated service instance completely from scratch with zero warm caches and zero existing state is something almost no engineering org practices once a quarter on a test cluster.
And when you are forced to do it for real during a worldwide outage, you inevitably discover the horrifying corpses buried in your architecture like circular dependencies.
You try to boot Service X, only to find out that six months ago a developer (or AI) added a check requiring Service X to already exist. As long as the grid is up and running, nobody ever notices the loop. The moment the whole system goes dark at the same time? Neither service can boot without the other, and your multi-billion-dollar cloud is locked in a Mexican standoff with itself while the whole world watches your status page burn.
The Time Bomb of AI-Generated Code
Naturally, in the aftermath of the email meltdown, Microsoft’s communications team was very quick to assure everyone that the July software defect was definitely not caused by AI. (Which feels a lot like when companies lay off 10,000 engineers because of “Not AI.”)
Sure. Let’s take them at their word on this specific bug. Now let’s look at what is happening to the rest of the codebase across the entire tech industry.
We are currently witnessing the opening act of an era defined by latent technical debt, brought on by executives aggressively firing human developers with institutional knowledge and replacing them with AI code generation and “vibe coding.” Even when AI isn’t left unsupervised in the driver’s seat or even when it is well-prompted and monitored by senior coders and architects, it introduces a fundamentally different, far more dangerous strain of technical debt.
When an overworked human developer writes hacky code at 2:00 AM, it usually looks like hacky code. It’s ugly, it smells bad in code review, and it usually comes with an apologetic comment like // TODO: refactor this garbage before peak season.
AI technical debt doesn’t look like that. AI-generated code is polite. It is immaculately formatted. It passes the linter, generates its own green unit tests, and sails through pull requests while quietly introducing diffuse state assumptions, subtle lock-contention bottlenecks, and invisible circular dependencies across fifty different microservices. It is technical debt that looks like clean code right up until a failover happens four months later and detonates the entire production fleet.
Only the Machines Can Read
Not long ago, Microsoft CEO Satya Nadella bragged that up to 30% of Microsoft’s code is already written by AI, while Microsoft’s CTO predicted that within a few years, 95% of all code will be AI-generated.
Stop and think about what that actually means when the pager goes off (They Won’t).
Any senior engineer will tell you that reading and debugging someone else’s code is twice as hard as writing it yourself. Now imagine trying to debug a cascading failure across ten million lines of synthetic glue code that no human being ever wrote or mentally modeled in the first place, especially after management laid off the greybeards who designed the underlying hardware and network topology.
If 90% of the code is written by AI, how do you actually know how your software and hardware work together under catastrophic stress? You don’t.
Right now, we are handing the keys of our global email systems, our aviation routing networks, and our banking ledgers over to synthetic code factories, all to save a few points on quarterly engineering payroll. And the next time the grid collapses into a cascading failure and requires a manual black start at 3:00 AM, the surviving engineers in the war room are going to open the repository and realize the ultimate horror of the modern tech stack:
Only the machines can read the code and the machines just crashed.





