Inside Microsoft’s rush to get back online after a global outage
Late into the night Thursday, hundreds of Microsoft engineers and top executives sat in their darkened offices or home basements, logged into a video conference and working through the tech company’s nightmare scenario.
The world still runs on email, and much of it is handled through Microsoft Outlook. The company’s outage affected hospitals, schools, businesses and congressional offices in Washington for several hours. Even Microsoft’s own employees couldn’t use email.
In a video conference call, Scott Guthrie, the top executive overseeing the company’s computing infrastructure, suggested possible fixes, according to a person familiar with the call who was not authorized to speak publicly. People were biting their nails, rubbing their foreheads, and debating how to shut down other services to prevent a “herd of stamps” so that the systems coming back online wouldn’t get overwhelmed with new requests and shut down.
By Friday morning, the company had apologized at least twice. It said it discovered the problem stemmed from a recent update to its network that shut down some servers, and that the systems are constantly being improved. “Due to the scale of demand, we restored services gradually and verified stability at each stage,” the company said in a statement. “Even though it took longer, it helped ensure a steady recovery.”
In the end, they kept the herd at bay.
The long and widespread outage was just the latest reminder that the technological foundations of modern life and business are controlled by a handful of large companies. And while cyberattacks and malicious actors can target their systems, sometimes companies learn the hard way that they can introduce devastating errors when updating or maintaining their own global systems.
Hospitals, law enforcement and government agencies and other vital services are often vulnerable – unable to function for hours or days at a time.
Last year, Amazon’s cloud services were down for 15 hours, knocking many apps and websites, including Zoom and parts of The New York Times, offline. And in the summer of 2024, a botched software update from cybersecurity firm CrowdStrike caused airlines to be grounded, businesses to close early, and doctor’s offices to be rescheduled.
For Microsoft, the problems from this week’s outage aren’t over yet. The short-term financial impact should be “manageable,” analysts at investment bank William Blair said in a research note on Friday. But they said the long-term outage is a reputational hit that could prompt companies and other customers to reconsider relying so much on one technology provider.
“We believe the outage could cause businesses to reevaluate their reliance on Microsoft infrastructure,” they wrote.
Outlook and other affected services, including Teams and its AI assistant Copilot, are part of the Microsoft 365 cloud subscription service, which is used by more than 450 million people and brought in nearly $28 billion in the most recent quarter that ended Sept. 30. Online memes joked that the offer should be renamed Microsoft 364.
These subscriptions are central to the company’s business and the basis on which it promotes AI tools. Microsoft tells customers that if they keep more data in Microsoft systems, they will be able to more easily implement and maximize the benefits of AI. Microsoft acknowledged that about 20 different services were disrupted by the outage, a reminder that vendor consolidation can also create a larger single point of failure.
Microsoft first discovered problems with Outlook Thursday morning around 9 a.m. on the West Coast, where the company is headquartered. It soon spread from America to the whole world.
The company initially said it thought the cause was an unexpected increase in traffic. Then he discovered that a large number of the company’s servers in its data centers were down, putting a strain on the rest of the system. Even when some servers were back online, they shut down again, like a game of Whac-A-Mole.
Luca Arnaboldi, an assistant professor of cyber security at the University of Birmingham in England, said the outage was a reminder that when people use the “cloud,” they are actually relying on servers that have limited resources.
During an overnight video conference, Mr Guthrie said that while other systems were also down, teams should prioritize restoring Outlook. Charles Lamanna, who leads many of Microsoft’s productivity services, was also there. He has already publicly apologized and said that Microsoft has ruled out a cyberattack as the culprit.
The company’s own systems were to blame.
Isabella Kwai contributed reports from London. Michael Levenson contributed reporting from New York.