How Uber Protects Against Retry Storms
summary
Uber implemented a context-aware 'error ownership' mechanism to prevent retry storms, where multiple services in a deep dependency chain each retry a failed request, leading to exponential traffic growth. By distinguishing whether a service is the actual cause of an error or merely passing it along, the system ensures that only the service owning the error attempts a retry. This approach prevents localized outages from escalating into stack-wide incidents without sacrificing overall system availability.
key points
- Retry storms occur because traditional retry budgets lack visibility into deep call chains, causing request volumes to multiply exponentially as they move up the stack.
- The error ownership system uses service dependency analysis to mark errors as 'claimed' by the originating service or 'unclaimed' when propagated, signaling to upstream callers that they should not retry.
- Production data shows this mechanism significantly reduces the 'retry storm radius' from an average depth of 20 to 2, preventing millions of spurious requests during major outages.
who should read it
Software architects and backend engineers working with large-scale microservices and complex dependency chains. It is particularly useful for those designing reliability patterns to prevent cascading failures.
discussion positive
consensus
Commenters generally agree that naive retry strategies, especially multiplicative retries across multiple service layers, lead to catastrophic 'retry storms' that amplify system failure. The prevailing sentiment is that retries should be bounded by a local retry budget (e.g., limiting retries to a small percentage of total traffic) and guided by specific error signals from the failing node rather than generic application-level logic. There is a strong preference for combining these with circuit breakers, exponential backoff with jitter, and load shedding to ensure system stability.
top insight
The most valuable perspective is the implementation of a 'retry budget' using an Exponential Moving Average of requests per second. By allowing only a small overhead (e.g., 20%) of additional requests as retries, a service can bound its total load locally without requiring complex distributed coordination. This approach prevents the multiplicative effect of retries while still allowing transient errors to be resolved.
Generated by an LLM from the article and thread. Check the sources before relying on it.