GitLab restored from a snapshot taken six hours before a 2017 database outage

On 31 January 2017, while rebuilding a lagging database replica, a GitLab.com engineer accidentally removed the production PostgreSQL data directory. The process was stopped within seconds, but about 300 GB had already been removed. The event followed database-load and replication problems, not a single careless moment in isolation.

GitLab’s routine pg_dump backups were unavailable because of a PostgreSQL version mismatch, and replication could not help because data had been wiped from both database hosts. An LVM snapshot made manually about six hours earlier did work. It let GitLab restore the database to 17:20 UTC, but database changes made until 23:30 UTC were lost. The lesson for any website owner: a backup file is not yet a recovery plan. You need proof that it restores cleanly, with the people, access, and time needed when a real incident happens.

Each fact is checked against the primary sources listed. Spotted an error? Email the Status Is UP team. Status Is UP helps owners improve their sites; it is not an uptime monitor. For live outage status, see Status Is Down.