History and stories
GitLab deleted its database and found its backups did not work
It was late at night, the database was under strain, and an engineer was trying to fix it. One command went to the wrong machine.
1 min readStory
The short version
A tired engineer ran a delete command on the wrong server. Then the team discovered how many of its backup methods had been failing.
The story
GitLab is a service that software teams use to store and manage their code. On January 31, 2017, its main database was under heavy load, and the copy that should have stayed in sync with it had fallen behind.
Late at night, an engineer working to fix the copy ran a command to clear out its data. It ran on the main database by mistake. They stopped it a second or two later, but about 300 GB of data was already gone.
The team went looking for backups. In its own public write-up, GitLab explained that several of the backup methods it relied on were not working or were not set up to cover the database. The best copy available was a snapshot made six hours earlier for a test system.
GitLab restored from that snapshot, livestreamed parts of the recovery, and lost about six hours of data, including comments, issues and new user accounts. It then published a detailed explanation and a list of changes.
Why it mattered
Several backup methods were in place, and none of them was working as expected when it mattered. By sharing the recovery publicly, mistakes and all, GitLab turned a bad night into one of the best-known lessons in testing backups.
The lesson today
GitLab believed it had several layers of backup. Nobody had checked recently that they worked. The real test of a backup is a restore, and the best time to find out it fails is during a scheduled test.
Talk to us about your technology
Tell us what you run today and what is getting in the way. We will get back to you about how we can help.