Wide Awake Developers

« July 2009 | Main | December 2009 »

GMail Outage Was a Chain Reaction

Google has published an explanation of the widespread GMail outage from September 1st. In this explanation, they trace the root cause to a layer of "request routers":

...a few of the request routers became overloaded and in effect told the rest of the system "stop sending us traffic, we're too slow!". This transferred the load onto the remaining request routers, causing a few more of them to also become overloaded, and within minutes nearly all of the request routers were overloaded.

This perfectly describes the "Chain Reaction" stability antipattern from Release It!