- Microsoft’s massive outage stemmed from a latent software defect.
- Increased network traffic and reduced capacity worsened impact.
- Cascading failure affected 80% servers, bypassing automated recovery.
- Engineers identified defect, implemented fix, gradually restored services.
Microsoft has revealed the technical failures behind the massive outage that disrupted Outlook, Microsoft 365, Exchange Online and other services for nearly 20 hours last week. According to the company’s post-incident report, a combination of a previously undetected software defect, increased network traffic and reduced server capacity triggered a cascading failure across its global infrastructure.
The disruption affected several Microsoft 365 services, although applications such as Teams, OneDrive, SharePoint and Office recovered sooner than Outlook and Exchange Online.
Microsoft said the incident was not caused by a single malfunction. Instead, three separate problems occurred simultaneously, overwhelming the network gateway responsible for handling traffic across several of its cloud services.
“The incident was caused by three main factors that together caused a global outage for the networking gateway that fronts many Microsoft 365 services,” the company said.
At the height of the disruption, more than 80% of the affected gateway servers became unresponsive, leaving Microsoft’s automated recovery mechanisms struggling to restore normal operations.
Microsoft Says Software Bug Had Been Hidden Since July
The first major contributor was a software defect introduced through a change specifically designed for Microsoft 365 networking gateway servers.
The change began rolling out in July 2026 and was gradually deployed across Microsoft’s infrastructure over several weeks. Despite passing the company’s standard testing and deployment procedures, the defect remained undetected for months.
“For more than three months, the change was deployed worldwide without impact,” Microsoft stated.
The problem emerged only when servers encountered particular combinations of traffic and processing demands. Under those conditions, the affected Windows servers could become overwhelmed and stop responding.
Microsoft acknowledged that its testing environment failed to reproduce the conditions necessary to expose the defect.
“Our normal load testing did not detect the latent issue because of the traffic mix and SKUs used in our labs.”
The company explained that the issue was more likely to occur on newer machines equipped with higher core counts. However, its testing procedures had largely focused on less powerful hardware.
Microsoft clarified that both the defect and the associated software change were specific to its Microsoft 365 gateway infrastructure and did not affect general Windows deployments.
Teams Traffic Shift Increased Pressure On Microsoft’s Servers
The second contributing factor involved an increase in traffic passing through the affected network gateway.
Microsoft operates two separate networking gateway systems to maintain service availability during technical problems or maintenance operations.
“Within Microsoft 365, we maintain two independent and heterogeneous network gateway systems designed for isolation and high availability,” the company said.
“This allows many Microsoft 365 services to fail to/from them during outages or disruptions.”
Approximately 15 hours before the widespread disruption began, Microsoft Teams traffic was transferred from the alternative gateway to the primary gateway, which also handles Exchange Online traffic.
Although such transfers are part of Microsoft’s routine reliability procedures, the additional workload changed traffic patterns and increased pressure on the primary infrastructure.
This higher demand made it more likely that servers would encounter the previously undetected software defect.
Nearly 14% Of Server Capacity Was Already Unavailable
A separate infrastructure problem further complicated the situation. Microsoft disclosed that approximately 14% of its networking gateway server capacity was already unavailable because of an unrelated incident that had occurred the previous day.
The company had planned to restore those servers through its normal maintenance procedures.
“This server capacity was on track to be returned to active service through routine maintenance and health processes,” Microsoft said.
“The service had sufficient capacity headroom, so the reduction in server capacity had no impact on its own.”
However, the combination of reduced capacity and additional Teams traffic pushed the remaining infrastructure into conditions that triggered the software defect.
The situation deteriorated during peak traffic hours in the United States, when requests began accumulating and servers stopped responding.
Microsoft’s traffic management systems attempted to redirect requests away from affected machines towards operational servers.
Under normal circumstances, this process helps prevent widespread interruptions. During this incident, however, the redirected traffic placed additional pressure on the remaining infrastructure, causing more servers to encounter the same defect.
The resulting chain reaction spread rapidly.
Within two hours, the problem had affected Microsoft’s gateway server fleet, with more than 80% of servers becoming unresponsive at the lowest point of service availability.
Microsoft also identified a separate failure in the infrastructure mechanisms designed to prevent problems from spreading between regions.
“Normally, the network gateway is composed of thousands of ‘capacity units’ which represent shards of capacity, hardware, and networking infrastructure,” Microsoft explained.
“These ‘capacity units’ are then organised into regions. In this case, due to a separate issue, the normal bulkheads between these different partitions did not stop the cascading failure.”
As a result, a failure that would ordinarily have been contained spread across the company’s global networking infrastructure.
Why Teams And OneDrive Recovered Before Outlook
Not every Microsoft service experienced the disruption for the same duration.
Teams, OneDrive, SharePoint and Office applications were able to shift traffic away from the malfunctioning gateway infrastructure, allowing them to recover relatively quickly.
“That is why those services were partially restored earlier, with Teams chats and meetings functioning without issue for most of the outage,” Microsoft said.
“But not all services, like Exchange Online, can fail out of this network gateway for scale reasons.”
Exchange Online remained dependent on the affected gateway, making recovery more complicated.
Microsoft’s engineers also faced difficulties restarting servers because accumulated traffic immediately overwhelmed machines as soon as they returned online.
“Our normally automated recovery processes were not designed to restore service from this broad unresponsive state,” Microsoft said.
“Additionally, since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change.”
How Microsoft Finally Restored Services
Once engineers identified the underlying defect, Microsoft developed a software update to restore the affected Windows code path to its earlier working configuration. Rather than restarting the entire infrastructure simultaneously, engineers took groups of servers offline, applied the correction and gradually returned them to service.
This controlled approach was necessary to prevent waiting traffic from overwhelming the recovering systems.
“To effectively apply this fix, and to slowly bring the service online again, we took groups of servers out of service, applied the change, and restarted them before returning them to service,” it said.
Microsoft explained that the recovery operation involved updating substantial portions of its infrastructure in stages, sometimes covering more than half of a region’s capacity at once.
The company said its final Post-Incident Review (PIR) will provide additional technical details about the disruption.
The incident highlights how multiple independent infrastructure problems can combine to defeat systems designed to keep cloud services running. For Microsoft, the outage also exposed limitations in its testing procedures, regional isolation mechanisms and automated recovery processes, particularly when a previously dormant software defect emerges under unusually demanding conditions.
Disclaimer : This story is auto aggregated by a computer programme and has not been created or edited by DOWNTHENEWS. Publisher: abplive.com






