Facebook claims Monday’s MASSIVE global outage was an ‘error of our own making’

The release iterates, "this outage was triggered by the system that manages our global backbone network capacity. The backbone is the network Facebook has built to connect all our computing facilities, which consists of tens of thousands of miles of fibre-optic cables crossing the globe and linking all our data centers."

"The data traffic between all these computing facilities is managed by routers, which figure out where to send all the incoming and outgoing data," it reads. "And in the extensive day-to-day work of maintaining this infrastructure, our engineers often need to take part of the backbone offline for maintenance — perhaps repairing a fibre line, adding more capacity, or updating the software on the router itself."

The statement then details that the company issued a command to assess the availability of its global backbone capacity during routine maintenance. Unintentionally, the decision took down all its connections in our backbone network, effectively disconnecting Facebook data centers globally.

"Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool prevented it from adequately stopping the command," continues the release. Facebook's DNS servers, which route internet traffic to the appropriate place internally, disconnect from the rest of the internet.

"All of this happened very fast. And as our engineers worked to figure out what was happening and why."

They faced two large obstacles. First, they could not access its data centers because their networks were down. Second, the total loss of its DNS servers broke the internal tools they'd typically use to investigate and resolve outages like this.

The statement noted, "every failure like this is an opportunity to learn and get better, and there's plenty for us to learn from this one." Facebook intends to conduct an extensive review process to understand how we can make our systems more resilient. "That process is already underway," they said.