On July 24, five days after a CrowdStrike update crashed Windows machines around the world, the company published its Preliminary Post Incident Review. A full root cause analysis is promised later. Here's what we know so far, and what we still don't.
What CrowdStrike says happened
The Falcon sensor on Windows gets two kinds of updates:
- Sensor Content, which ships with new versions of the sensor software. Customers can control when they take those, and they go through CrowdStrike's full release testing.
- Rapid Response Content, detection data pushed to every sensor frequently, sometimes several times a day, to respond to new threats quickly. Customers had no control over when this arrived.
In February, sensor version 7.11 added a new capability to detect malicious use of Windows named pipes, an inter-process communication mechanism attackers use for command and control. The detection rules for it are delivered as Rapid Response Content, in a file called Channel File 291.
On July 19 at 04:09 UTC, CrowdStrike pushed two new detection rules in that file. One of them contained problematic data. It should have been caught by CrowdStrike's Content Validator, an automated check that runs before content is published. It wasn't, because the validator itself had a bug that let the bad data through. Earlier rules of the same type, deployed in the spring, had worked, which increased confidence in the new ones.
When the sensor loaded the new rule, it read memory it shouldn't have. Because the sensor runs as a driver in the Windows kernel, that out-of-bounds read crashed the whole system, and because the file loaded again at every boot, the machines kept crashing. CrowdStrike reverted the content at 05:27 UTC, but any Windows machine that had been online during those 78 minutes was already down.
What CrowdStrike has promised
The review lists commitments, and they're the right ones:
- Staggered deployment of Rapid Response Content: canary machines first, then gradually wider rings, with monitoring between stages.
- Customer control over when Rapid Response Content is delivered, and where.
- More testing: local developer testing, update and rollback testing, stress testing, fuzzing and fault injection.
- Better error handling in the sensor's content interpreter, so bad content fails safely instead of crashing the kernel.
- Fixes to the Content Validator itself.
- Release notes for Rapid Response Content updates, and independent third-party reviews of its code and processes.
What it cost
Microsoft estimated that 8.5 million Windows machines were affected, less than one percent of all Windows devices, but concentrated in exactly the organizations that matter most: airlines, hospitals, banks, broadcasters and governments.
The insurer Parametrix estimated that the outage cost Fortune 500 companies, excluding Microsoft, about $5.4 billion in direct losses, of which insurance would cover only a small fraction. Airlines were among the worst hit. Delta canceled thousands of flights over several days as its crew-scheduling system struggled to recover, long after most others had.
The questions still open
The preliminary review explains the mechanism. It leaves bigger questions for the full analysis:
- Why did content go to every machine at once in the first place? Staggered rollout is standard practice for software. Treating detection data as exempt was a choice.
- Why does content parsing happen in the kernel at all, where a parsing bug becomes a machine crash rather than a failed update?
- How was the validator bug introduced, and why didn't testing of the validator catch it?
- What changes on the Windows side? Microsoft has pointed to a 2009 agreement with the European Commission that it says requires it to give third-party security vendors the same kernel access its own products have. Expect a lot of discussion about whether security software should run in the kernel at all.
Lessons for everyone else
You don't need to be CrowdStrike to learn from this:
- Data is code. A configuration file, a rule set or a feature flag can take down production as surely as a binary. Test it and roll it out the same way.
- Test your tests. A validator that passes everything is worse than none, because people trust it.
- Stage everything, with automatic halts when error rates rise.
- Design for bad input. Software that parses data, especially in privileged contexts, should treat malformed input as expected, not impossible.
- Practice recovery at scale. The fix was simple. Applying it by hand to millions of machines, many with encrypted disks whose recovery keys were on servers that were also down, was not.
Since then
Added October 2026. CrowdStrike's full root cause analysis came out on August 6. I covered it in the postmortem of the July 2024 outage. Delta later put its own losses at about $500 million and sued CrowdStrike, and CrowdStrike sued back.