History and stories
The 2024 outage that had to be fixed one computer at a time
Airports, hospitals, banks and shops all saw the same blue screen on July 19, 2024. The fix was simple. Applying it was not.
1 min readStory
The short version
A faulty security software update crashed about 8.5 million Windows computers. Many had to be fixed by hand, and some needed recovery keys that were hard to find.
The story
On July 19, 2024, at 04:09 UTC, the security company CrowdStrike sent out a faulty update to its software for Windows computers. Computers that received it crashed and could not start up properly. About 8.5 million devices were affected.
Airlines grounded flights, hospitals postponed procedures, and banks, broadcasters and stores lost their systems. It has been called the largest IT outage in history.
The faulty update was withdrawn within hours, but many computers that had already crashed could not fix themselves. Someone had to start each one in a recovery mode and delete a file by hand.
On computers with BitLocker disk encryption, that recovery mode first asked for a 48-digit recovery key unique to each machine. Getting those keys to staff working remotely was its own problem. Some organizations stored their keys on servers that had crashed in the same outage.
Why it mattered
It showed how much depends on a single vendor’s update reaching millions of computers at once, and that a security product can cause an outage as large as any attack. Recovery depended on having each machine’s recovery key stored somewhere reachable, and people able to fix computers one at a time by hand.
The lesson today
Every business needs a way to fix computers when the usual tools are down. That means recovery keys stored somewhere that does not depend on the systems they unlock, and a plan for reaching remote staff when their laptops will not start.
Talk to us about your technology
Tell us what you run today and what is getting in the way. We will get back to you about how we can help.