The typo that took down part of the internet

The story

Amazon S3 is a storage service that a huge number of websites and apps use to hold files and data. On February 28, 2017, an Amazon engineer was working on a problem with the billing system in one of its biggest data center regions, in northern Virginia.

Following an established procedure, the engineer ran a command to take a small number of servers out of service. One of the inputs was entered incorrectly, and a much larger set of servers was removed. Those servers supported key parts of S3, which had to be fully restarted.

For several hours, websites and apps that depended on S3 in that region broke or slowed down. Amazon’s own status page could not show the problem at first, because it also relied on S3. The company posted updates on social media instead.

Afterward, Amazon changed the tool so it removed capacity more slowly and could not take a system below the minimum it needed.

Why it mattered

It showed how many businesses depend on a small number of cloud providers, often without realizing it: one mistyped command in one region broke websites and apps across the internet. Even the status page meant to explain the problem depended on the system that had failed.

The lesson today

The cloud is very reliable, but it is still run by people and still has bad days. If your business depends on one provider in one place, an outage there is your outage too. It is worth knowing which services you rely on and what you would do if one went down.