Seu agente causa retry storms (e você não sabe)

Seu agente causa retry storms (e você não sabe).

Você é founder de SaaS.

Seu agente de IA:

• Chama APIs externas (Google, Stripe, sua DB)
• Se API falhar, tenta de novo (retry logic)
• Your assumption: "Retries são seguras. Está tudo ok."
• Reality: "Seu agente está causando retry storms (cascata de failures)."
• Your blind spot: ├─ API call fails (timeout, 5xx error) ├─ Your agent retries (1st attempt: fail, 2nd: fail, 3rd: fail) ├─ Each retry hits API novamente (load increases) ├─ API already struggling, retries make it worse (overload) ├─ Downstream systems overwhelmed (cascata começa) └─ Result: "You broke the API. Everyone loses. Nobody knows why (you)."

Uber just discovered this problem:

"Retry storms: When clients retry aggressively, they overwhelm backend systems. Cascading failures. Everything breaks. Customers see timeouts. You see nothing wrong (on your side)."

Translation to your SaaS:

• Old assumption: "Retries are good (resilient)."
• New reality: "Naive retries are bad (cause storms)."
• Implication: "Your agent is destroying APIs (silently)."
• Your choice: Implement retry protection or lose reliability.

---

O Problema: Agentes com retry logic ingênua causam cascatas invisíveis

Por que retry storms são silenciosas e destrutivas

=== THE RETRY STORM MECHANICS ===

Scenario: Seu agente chama Google Sheets API

Normal operation: ├─ Agent: "Give me cell A1" ├─ API: "A1 = 'Hello'" (response: 200ms) ├─ Agent: Gets answer, happy └─ Cost: 1 API call

When API is slow (overloaded): ├─ Agent: "Give me cell A1" (call 1, wait 5s) ├─ Timeout! (your timeout = 5s, API is slow) ├─ Agent: "Retry! Give me cell A1" (call 2, wait 5s) ├─ Timeout again! (API still overloaded) ├─ Agent: "Retry! Give me cell A1" (call 3, wait 5s) ├─ Timeout again! (API DYING) ├─ Agent: "Retry! Give me cell A1" (call 4, wait 5s) ├─ Timeout again! (API is DEAD) └─ Result: 4 calls instead of 1 (4x load on struggling API)

=== THE CASCADE EFFECT ===

Now imagine 1000 agents all doing this: ├─ 1 agent: 4 API calls (API struggling but ok) ├─ 10 agents: 40 API calls (API slowing down) ├─ 100 agents: 400 API calls (API timeout now common) ├─ 1000 agents: 4000 API calls (API COMPLETELY DEAD) ├─ Result: "Thundering herd. API collapses. Everyone times out." └─ Root cause: "Innocent retries created perfect storm."

=== THE INVISIBLE FAILURE ===

What you see: ├─ Your agents: "API is timing out. Not our problem." ├─ Your logs: "Timeout errors. Why so many?" ├─ Your dashboard: "Everything red. What happened?" ├─ Your customers: "Service broken. Fix it!" └─ Your reality: "API is dead. But I didn't cause it. Or did I?"

What's actually happening: ├─ API provider sees: "1000 agents slamming us (retry storm)" ├─ API provider analysis: "One customer (you) is DDoS'ing us" ├─ API provider action: "Rate limit that customer (you)" ├─ Your experience: "Suddenly getting rate limited. Why?" └─ Actual reason: "Your naive retries looked like attack."

=== THE FINANCIAL IMPACT ===

Direct costs: ├─ API overages (charged per call, you made 4x calls) ├─ Stripe charges: R$ 100/month → R$ 400/month (4x) ├─ Google Sheets charges: R$ 50/month → R$ 200/month (4x) ├─ Total: +R$ 150/month from nothing └─ Annualized: +R$ 1,800/year

Indirect costs: ├─ Customer complaints ("service is slow") ├─ Support tickets ("why is this timing out?") ├─ Reputation damage ("service is unreliable") ├─ Churn (customers leave for more reliable SaaS) ├─ Lost revenue: R$ 10K-100K+ (depending on scale) └─ Root cause unknown (you blame API, API blames you)

=== WHY IT'S INVISIBLE ===

You don't see retry storms because: ├─ Your logs show: "Timeout to Google API" (you're failing) ├─ You don't see: "I made 4 calls when I should make 1" (your fault) ├─ Google API logs show: "DDoS from customer X" (they see it) ├─ You never see Google's logs (they're not yours) ├─ Result: "You think API is bad. Google thinks you're attacking them." └─ Truth: "You're both right. Your retries are the problem."

---

A Verdade Incômoda: Retry logic é mais perigoso que você pensa

Como retries "bem-intencionadas" destroem sistemas

=== THE RETRY PYRAMID ===

Retry strategy (from safest to most dangerous):

1. No retries (naive, not resilient) ├─ Code: if (API fails) { return error } ├─ Problem: API hiccup = your failure ├─ Resilience: 0% (fails at first problem) ├─ Risk: Safe (no storms), but fragile └─ Use case: None (always add retries)

2. Immediate retries (dangerous, causes storms) ├─ Code: if (API fails) { retry immediately } ├─ Problem: 1000 agents retry → 4000 calls/s ├─ Resilience: Looks good (retries help) ├─ Risk: VERY HIGH (creates retry storms) ├─ Example: Your agent (probably doing this) └─ Uber finding: "This is what breaks systems"

3. Exponential backoff (better, but still risky) ├─ Code: if (API fails) { wait 1s, retry; fail again, wait 2s, retry; etc } ├─ Problem: Still can create storms (delayed, but still storms) ├─ Resilience: 70% (helps with transient failures) ├─ Risk: MEDIUM (slower, less aggressive) └─ Use case: Most systems use this

4. Circuit breaker (safe, prevents storms) ├─ Code: if (API fails 3x) { stop retrying, fail fast } ├─ Problem: None (stops storms before they start) ├─ Resilience: 95% (transient failures + graceful degradation) ├─ Risk: LOW (prevents storms by design) └─ Use case: Uber's recommendation

5. Adaptive retry (best, learns from system state) ├─ Code: Monitor API health, adjust retry strategy dynamically ├─ Problem: None (prevents storms, adapts to conditions) ├─ Resilience: 99% (optimal for any condition) ├─ Risk: NONE (intelligent by design) └─ Use case: Large-scale systems (Netflix, Uber)

=== YOUR CURRENT RETRY STRATEGY ===

If you're not sure, you're probably doing #2 (immediate retries): ├─ Symptom 1: "API timeouts are common" ├─ Symptom 2: "API costs are higher than expected" ├─ Symptom 3: "Getting rate limited by API provider" ├─ Symptom 4: "No idea why" (invisible problem) └─ Diagnosis: "You have retry storms."

=== THE UBER LESSON ===

What Uber discovered: ├─ Immediate retries: Look good locally (agent gets answer) ├─ At scale: Cause cascading failures (system dies) ├─ Solution: Circuit breakers + exponential backoff ├─ Implementation: Seems complex (but actually simple) └─ Impact: Prevents 90% of cascade failures

=== THE STRATEGIC IMPLICATION ===

For your SaaS: ├─ Option A: Naive retries (current, probably) │ ├─ Cost: Free (already coded) │ ├─ Reliability: Looks good, breaks silently │ ├─ Scalability: Fails at 100+ concurrent agents │ └─ Outcome: "Works until it doesn't. Then disaster." ├─ Option B: Exponential backoff + circuit breaker │ ├─ Cost: 4 hours engineering (simple to implement) │ ├─ Reliability: 95%+ (handles most failures) │ ├─ Scalability: Works to 10,000+ concurrent agents │ └─ Outcome: "Resilient, scalable, predictable." └─ Verdict: "Option B is worth 4 hours. Option A will cost you 100 hours of debugging."

---

Como implementar proteção contra retry storms

Step-by-step: Adicione circuit breaker ao seu agente

=== WHAT IS A CIRCUIT BREAKER? ===

Think of it like a electrical circuit breaker: ├─ Normal: Current flows (API works, requests succeed) ├─ Problem: Too much current (API failing, requests timeout) ├─ Breaker trips: Stops current (stops retrying, fail fast) ├─ After cooldown: Tries again (automatic recovery) └─ Result: Prevents downstream damage (no storms)

=== SIMPLE IMPLEMENTATION ===

Pseudocode (works in any language):

class CircuitBreaker: state = "CLOSED" (accepting requests) failure_count = 0 failure_threshold = 5 (fail 5 times, then open circuit)

def call_api(request): if state == "OPEN": return fail_fast("Circuit open, API is broken")

try:
  response = api.call(request)
  failure_count = 0 (reset on success)
  return response

except APIError:
  failure_count += 1
  if failure_count >= failure_threshold:
    state = "OPEN" (stop retrying)
    schedule_retry_in(60_seconds) (try again after 60s)
  raise

=== HOW IT PREVENTS STORMS ===

Without circuit breaker: ├─ Request 1: fail, retry immediately ├─ Request 2: fail, retry immediately ├─ Request 3: fail, retry immediately (API dying) ├─ Request 4: fail, retry immediately (API dead) └─ Total: 1000 requests, each retried 5x = 5000 API calls

With circuit breaker: ├─ Request 1: fail, failure_count = 1 ├─ Request 2: fail, failure_count = 2 ├─ Request 3: fail, failure_count = 3 ├─ Request 4: fail, failure_count = 4 ├─ Request 5: fail, failure_count = 5 (CIRCUIT OPENS) ├─ Request 6-1000: fail fast (no retry, no API call) ├─ After 60s: Try again (if API recovered) └─ Total: 5 API calls (vs 5000 without circuit breaker) └─ Improvement: 1000x fewer calls

=== EXPONENTIAL BACKOFF (COMPLEMENT TO CIRCUIT BREAKER) ===

Add delays between retries (don't retry immediately):

Retry strategy: Attempt 1: immediate (0s wait) Attempt 2: wait 1 second, then retry Attempt 3: wait 2 seconds, then retry Attempt 4: wait 4 seconds, then retry Attempt 5: wait 8 seconds, then retry (then circuit opens)

Why? If API is slow, retrying immediately makes it worse Waiting gives API time to recover Delays are exponential (1s → 2s → 4s → 8s) So you don't hammer API while it's struggling

=== COMPLETE SOLUTION ===

Combine circuit breaker + exponential backoff:

class ResilientAPIClient: circuit_breaker = CircuitBreaker(failure_threshold=5, timeout=60s)

def call_with_retry(api_endpoint, request, max_attempts=5): for attempt in range(1, max_attempts + 1): try: # Circuit breaker protects against storms response = circuit_breaker.call_api(api_endpoint, request) return response

  except APIError as e:
    if attempt < max_attempts:
      # Exponential backoff (1s, 2s, 4s, 8s, 16s)
      wait_time = 2 ^ (attempt - 1)
      sleep(wait_time)
      continue
    else:
      # All retries exhausted, fail gracefully
      return graceful_failure("API unavailable after 5 attempts")
  
  except CircuitBreakerOpen:
    # Circuit is open, don't retry
    return graceful_failure("API is broken (circuit open)")

=== IMPLEMENTATION CHECKLIST ===

[ ] Week 1: Plan ├─ [ ] Identify all external API calls (Stripe, Google, etc) ├─ [ ] List current retry logic (immediate? exponential?) ├─ [ ] Assess risk (are you using naive retries?) └─ [ ] Decision: Implement circuit breaker? (if yes, continue)

[ ] Week 2: Implement ├─ [ ] Code circuit breaker (4-6 hours) ├─ [ ] Add exponential backoff (2-3 hours) ├─ [ ] Test locally (2 hours) ├─ [ ] Deploy to staging (1 hour) └─ [ ] Total effort: ~10 hours

[ ] Week 3: Test ├─ [ ] Simulate API failure (turn off API temporarily) ├─ [ ] Verify circuit breaker opens (stops retrying) ├─ [ ] Verify exponential backoff works (increasing delays) ├─ [ ] Monitor API calls (should drop 50%+ on failure) └─ [ ] Decision: Deploy to production?

[ ] Week 4: Deploy ├─ [ ] Roll out to production (1% traffic first) ├─ [ ] Monitor metrics (API call count, timeouts, latency) ├─ [ ] Increase to 50% traffic (no issues?) ├─ [ ] Roll out to 100% traffic └─ [ ] Declare victory (retry storms prevented)

=== EXPECTED OUTCOME ===

After implementing circuit breaker + exponential backoff: ├─ API call reduction: 50-80% (fewer retries) ├─ Timeout reduction: 60-90% (fail fast instead of retry) ├─ Reliability improvement: 95%+ (handles API failures gracefully) ├─ Cost reduction: 20-30% (fewer API calls = less overages) ├─ Scalability: 10x more concurrent agents without breaking APIs └─ Customer experience: Faster failures (instead of hanging)

---

Checklist: Seu agente tem proteção contra retry storms?

Avalie seu retry logic

=== RETRY STORM RISK ASSESSMENT ===

[ ] Current retry logic ├─ [ ] Do you have retries? (if no: add them) ├─ [ ] Are retries immediate? (if yes: risky) ├─ [ ] Do you have exponential backoff? (if no: add it) ├─ [ ] Do you have circuit breaker? (if no: add it) ├─ [ ] Do you have max attempts limit? (if no: dangerous) └─ [ ] Verdict: Safe or risky?

[ ] Symptoms of retry storms ├─ [ ] API costs higher than expected? (sign: over-retrying) ├─ [ ] Getting rate limited by API provider? (sign: too many calls) ├─ [ ] Frequent timeouts? (sign: API overloaded by your retries) ├─ [ ] Cascading failures (one API breaks, everything breaks)? (storm symptom) ├─ [ ] No idea why failures are happening? (invisible problem) └─ [ ] If yes to any: You probably have retry storms

[ ] Scale assessment ├─ [ ] Concurrent agents: <10 (low risk, storms unlikely) ├─ [ ] Concurrent agents: 10-100 (medium risk, possible storms) ├─ [ ] Concurrent agents: 100-1000 (high risk, storms likely) ├─ [ ] Concurrent agents: 1000+ (critical risk, storms certain) └─ [ ] Your scale: _____ (be honest)

[ ] API dependency assessment ├─ [ ] How many external APIs do you call? (>3: high complexity) ├─ [ ] Which are critical? (payment = critical, analytics = not) ├─ [ ] How often do they fail? (rarely vs often) ├─ [ ] What happens if one fails? (cascade vs isolated) └─ [ ] Verdict: Simple or complex?

=== SCORING ===

Risk score: ├─ No circuit breaker + immediate retries + high scale = CRITICAL RISK ├─ No circuit breaker + exponential backoff + high scale = HIGH RISK ├─ Circuit breaker + exponential backoff + high scale = LOW RISK ├─ Any setup + low scale = MEDIUM RISK (but OK for now) └─ If CRITICAL or HIGH: Implement circuit breaker THIS WEEK

=== DECISION ===

If CRITICAL RISK: └─ STOP EVERYTHING, implement circuit breaker NOW (prevent disasters)

If HIGH RISK: └─ IMPLEMENT THIS MONTH (prevent future problems)

If MEDIUM RISK: └─ IMPLEMENT THIS QUARTER (prepare for scale)

If LOW RISK: └─ MONITOR (you're already safe, but stay vigilant)

---

Conclusão: Retry logic bem implementada = estabilidade. Naive retries = disaster.

O que Uber descobriu sobre retry storms:

1. Immediate retries são perigosas (parecem seguras, quebram tudo)

   • Antes: "Retries são bom (resilência)"
   • Depois: "Naive retries são ruin (criam storms)"
   • Implicação: "Sua implementação atual é provavelmente perigosa."

2. Retry storms são invisíveis (você não vê, mas estão acontecendo)

   • Antes: "Timeouts ocasionais (aceitável)"
   • Depois: "Timeouts frequentes (seu agente é culpado)"
   • Implicação: "Você pode estar destruindo APIs sem saber."

3. Circuit breaker é solução simples (4 horas de código, 100 horas de economia)

   • Antes: "Implementar é complexo"
   • Depois: "Implementar é trivial"
   • Implicação: "Sem desculpa pra não implementar."

4. Scale amplifica o problema (1 agent = ok, 100 agents = disaster)

   • Antes: "Funciona na minha máquina"
   • Depois: "Quebra em produção com múltiplos agentes"
   • Implicação: "Você precisa de circuit breaker ANTES de escalar."

5. Seu agente está causando dano (silenciosamente, agora)

   • Antes: "Meu agente é seguro"
   • Depois: "Meu agente pode estar quebrando APIs externas"
   • Implicação: "Implementar circuit breaker é prioridade #1."

Sua decisão hoje:

• Ignore (hope you don't hit scale)
• Evaluate (check if you have retry storms)
• Implement (add circuit breaker this week)

Recomendação: Check seu código HOJE. Se tiver naive retries, implement circuit breaker THIS WEEK. Você pode estar causando problemas silenciosamente.

Na OpenClaw:

Ajudamos SaaS builders implementar retry protection:

• Retry assessment: Seu agente tem retry storms? (audit)
• Architecture review: Está using circuit breaker? (analysis)
• Implementation guide: Como adicionar circuit breaker? (engineering)
• Testing strategy: Como verificar se funciona? (validation)
• Monitoring setup: Como detectar retry storms em produção? (observability)
• Scaling preparation: Como manter estabilidade com 100+ agents? (reliability)
• Incident response: Se der ruim, como recuperar rápido? (reliability)

Your agents can either cause silent disasters (now) or prevent them (with circuit breaker).

Choice: Naive retries or resilient architecture?

Retry Storm Prevention | Circuit Breaker Implementation | API Reliability →