Uber cut the worst retry storm its network can produce from 25 hops to three by taking the decision to retry away from the caller. An error now carries a claim header, and the first service to spot an unclaimed error stops everything above it [1].

Twenty-four days later, a test of agent tool calls measured the caller that rule has to police. Offered a key that did not match the stated contract, one model retried anyway in 60 of 80 responses, and another did it in 55 of 80 [2].

The big picture:

The part I keep circling: both teams are asking the same question — who owns this error? — from opposite sides of the wire.

Uber’s engineers laid out the math. If every hop in a call chain retries once, a node three levels down handles eight times its usual traffic [1].

A retry budget caps each hop only. At a 10% budget, that deep node still takes 1.33 times its usual load [1].

The budget also cannot see when the retries land. A service cannot tell a failure it caused from one it just passed along, the engineers write [1]. A bad path three levels down looks the same as a bad path at the edge.

That gap costs money. At a 10% retry budget, a service at 90% uptime looks like 99% to its caller, but one at 80% looks like 88% and one at 70% looks like 77% [1]. Retrying a weak service leaves the caller with a worse view of it.

So Uber moved the call. A service claims an error it caused itself with a header. An error from further down comes back unclaimed, and the first service to see that missing claim strips the right to retry from every service above it [1].

It works, at least against normal traffic. Across Uber’s user-facing systems, the largest retry-storm radius fell from 25 hops to three, and the average from 20 to two [1]. My read: that works because the callers are services, and services follow rules.

Put an agent on the other end and the rule bends. RetryBench measures what its authors call false retry permission: an agent resending a timed-out tool call when the stated contract gives it no right to do so [2].

The finding that matters is a split, not a rate. “Successful protected recovery does not imply reliable rejection of unsupported retries,” the authors write [2]. Getting a retry right is not the same skill as refusing one you were never granted.

The counts carry it. With valid keys, the models made safe progress. With changed keys, retries with no basis came in at 1 in 80 responses for one model, 55 in 80 for the next, and 60 in 80 for the third [2].

A later test removed the key and made the model supply one. The worst of the three still resent 8 of 80 calls it had no right to make [2]. I keep coming back to the size of that spread: one task, one contract, three models, and a 55-fold gap in how often the check is skipped.

Now put both halves in one building. Uber routes agent tool calls through one gateway, sitting right in front of real services [4]. That gateway hosts more than 800 servers and 5,000 tools, all switched off by default, and it is the first layer with the power to stop a resent call [4].

A catalog drawn from 150 compound AI incidents finds that boundary controls do work. Circuit breakers cut cascade spread by 89%, quality gates catch 73% of silent degradation, and isolation cuts blast radius by 64% [3]. Those gates trip on a signal the component sends. An agent that reports success after a retry it was never granted sends no such signal [2].

The proof problem sits underneath. One team tried to prove two retry gates safe from a 7,200-episode log and found the log proved neither of them, because it hides whether harm came before the stop [5]. With no per-step harm data, the harm ceiling they could prove rose as much as 2.5-fold [5].

By the numbers

  • 1.33× vs 8× — Depth is the multiplier: With a retry at every hop, a node three levels deep serves eight times its usual traffic; a 10% retry budget only pulls that back to 1.33 times [1].
  • 80% to 88% — Retries lower uptime: At a 10% budget, a service at 90% uptime looks like 99% to its caller, but 80% looks like 88% and 70% looks like 77% [1].
  • 60 of 80 — Retries with no basis: Offered a changed key, one model retried with no basis in 60 of 80 responses; a later test still left 8 of 80 unasked-for resends [2].
  • 25 to 3 — Storm radius: The largest retry-storm radius across Uber’s user-facing systems after error ownership shipped, down from 25 hops, with the average falling from 20 to two [1].

What I’d watch:

I’ve been watching retry settings for a while, and the layer count is the part that still surprises me.

  • Where the retry count lives: One operator found a single tool call sent nine times in a minute, because a two-retry default in the client library stacked on a two-retry rule at the gateway. They turned off the library’s retries and moved the hard cap to the gateway [6].
  • Whether the check survives a model swap: A 55-fold spread between three models on one task looks like a trait of the model, not of the contract, and no retry budget can read it [2]. I’d want that check in the release gate, not in the incident review.
  • Whether the caller sees its own failure: In a study of 2,518 long runs, agents recovered from their first mistake in only 30.5% of runs and missed it outright in 38.5% [7]. Asking the caller to judge a fair retry means asking a system that misses the answer a third of the time.
  • What a gate can prove early: Rerunning just the calls each gate would have blocked, from saved states, matched both live calls at half the cost in agent runs [5].
  • Whether the breaker sees the caller: Circuit breakers cut cascade spread by 89% in fault-injection tests, and they trip on a signal from the component [3]. An ownership header assumes a caller that reads it.

I could be wrong, but the case I want to see handled is not the first refusal. It is the second request, from an agent that recovered from the timeout and decided the call never ran.

The catch

The agent-side counts are a lab result and should be quoted as one. RetryBench’s scope note says its evidence covers four set tasks, fixed endpoints and one recovery decision, and that it names neither internal mechanisms nor session-level safety [2]. Three models missing a check measures a class of failure, not a rate for your fleet.

Uber’s figures come from one firm’s network, and the protocol is not an off-the-shelf fix. The claim logic leans on a service dependency graph, and the caller side has to build the unclaim path [1].

Some of that post’s sharpest figures are older than they look. The 46% to 135% traffic rise a plain retry budget would have caused, and the 9.5 million wasted requests stopped, both come from the outage of 18 November 2025 [1]. They belong with that date attached.

The catalog’s pattern numbers come from fault injection, not from live field data, and the 150 incidents behind it are unnamed [3]. Read the 89% and the 64% as a roof on what a boundary control can do, not as a promise.

I would also hold the whole framing against one critique. On the thread that followed Uber’s write-up, the sharpest note is that retries add stress to a system that is already struggling, by plain queue math, whatever the ownership rule says [8]. I take that as a limit on the mechanism, not a reason to drop it. Ownership decides who may retry; it does not decide how much of a busy service is safe to touch.

The honest gap is smaller and harder to spot. An agent that recovers well after a retry with no basis may still be safe, since a call that never ran costs nothing to resend. The trouble is that no outcome tells you which case you had, and every gate here rests on an outcome.

At a glance

  • The Big Shift: Uber now stamps errors with a claim header so only the owner’s direct caller may retry, which cut the largest retry-storm radius on its network from 25 hops to three. Twenty-four days later, a test of agent tool calls found models granting themselves retries the stated contract did not support.
  • Why It Matters: The retry decision has quietly left the caller. A retry budget caps each hop only, and past a small dip in uptime a retry leaves the caller’s view of a weak service worse, so platforms are handing the right to retry to the error’s owner. Agents are the newest callers on that network, and their compliance looks like a trait of the model rather than of the contract — which pushes the check to the layer that runs the call.
  • What I’d Watch: The places where a retry decision gets an owner and a proof.
  • Retry ownership header: A marker on an error that names the service which caused it, so every service above that only passed the error along loses the right to retry it.
  • False retry permission: Resending a timed-out call when the stated contract gives no basis for it — a class of failure a good recovery can hide.
  • Saved-state rerun: Replaying only the calls a gate would have blocked, from a saved snapshot, to learn whether the gate holds without running the whole agent live.
  • Tool-boundary gateway: The one execution surface between agents and internal services, and the layer placed to grant or refuse a resend per call.
  • The Catch: The agent-side counts come from four fixed-endpoint tasks and one recovery decision, and Uber’s most quoted figures date from a November 2025 outage. The catalog’s numbers come from fault injection, and queue math still raises the cost of retrying a busy service whatever the header says.

Related reading

Sources

[1] Uber Engineering, “How Uber Protects Against Retry Storms” (17 September 2026) — https://www.uber.com/us/en/blog/protecting-against-retry-storms/ [2] arXiv:2610.15884 — “RetryBench: Separating Retry Permission from Recovery Outcomes in GPT Tool Use” (11 October 2026) — https://arxiv.org/abs/2610.15884 [3] arXiv:2610.02503 — “Compound AI System Reliability: A Failure Taxonomy and Resilience Pattern Catalog from 150 Production Incidents” (1 October 2026) — https://arxiv.org/abs/2610.02503 [4] Uber Engineering, “Designing MCP Gateway: Uber’s MCP Management Platform” (1 October 2026) — https://www.uber.com/us/en/blog/designing-mcp-gateway [5] arXiv:2610.16280 — “agent-risk: Risk Assessment for Changes to AI Agents” (12 October 2026) — https://arxiv.org/abs/2610.16280 [6] Menglune, “Retry boundaries for agent tool calls: 9 attempts down to 3” (17 September 2026) — https://www.menglune.com/en/news/agent-tool-call-retry [7] arXiv:2609.17930 — “Locating Hidden Failures Makes Long-Horizon Agents More Reliable” (15 September 2026) — https://arxiv.org/abs/2609.17930 [8] Hacker News discussion of “How Uber Protects Against Retry Storms” (17 September 2026) — https://news.ycombinator.com/item?id=49746628