Agent em demo (funciona). Em produção (quebra silenciosamente)

Agent em demo (funciona). Em produção (quebra silenciosamente)

What just happened (October 2026):

TinyFish published agentic search architecture guide:

"Problem: Demo agents work great (controlled environment)."

"Reality: Production agents fail (incomplete data, cost explosion, timeout loops)."

"Root cause: Demos don't test edge cases (bad retrieval, retries, resource limits)."

"Solution: Production agentic search architecture (handles all edge cases)."

"Key insight: Agent success ≠ model quality. Agent success = architecture quality."

What this means for your company:

Current situation (your agent deployed to production):

Setup (typical SaaS agent): Agent: Customer research agent (WhatsApp) User: Sales rep needs competitor intel Workflow: Agent searches web, summarizes findings, returns report

Demo environment (what you tested): Query: "What's Competitor X's pricing?" Result: Search finds pricing page immediately Agent: Returns accurate pricing summary (2 seconds) Demo success: ✓

Production reality (day 1): Query: "What's Competitor X's latest product roadmap?" Result: Search returns 50 pages (incomplete data, roadmap scattered) Agent (first attempt): Summarizes partial data (missing key features) Sales rep: Makes pitch based on incomplete intel (loses deal) Agent cost: R$0.50 (one search) Your cost: R$50,000 (lost deal from bad intel)

Production reality (day 2, same agent): Query: "What's the CEO's background?" Result: Search returns generic results (no relevant data) Agent (first attempt): No answer Agent (retry 1): Tries different search terms (still nothing) Agent (retry 2): Tries broader search (still incomplete) Agent (retry 3): Still failing Agent (retry 4): Still failing Agent (retry 5-50): Stuck in retry loop (not coded to stop) Agent: Makes 50 API calls (costs R$25) Agent: Still returns no answer (timeout, 2 minutes) Sales rep: "Your agent is broken" Your cost: R$25 + reputation damage

Production reality (day 3): Query: "Analyze all competitors' recent announcements" Agent: "Sure, searching..." Agent: Does 500 searches (comprehensive research) Agent: Total cost: R$250 (just for ONE query) Your margin: You charge customer R$100/month for agent Your loss: -R$150 (agent costs more than you earn) You disable the feature (silent failure = customer doesn't know why)

Pattern (the real problem): Demo = controlled, perfect data, quick answers Production = messy data, incomplete results, retry loops, cost explosions Agent architecture: Not designed for production Result: Works in demo, fails silently in production

---

Why agents fail in production (3 hidden failure modes)

Failure 1: Incomplete retrieval (agent hallucinate with partial data)

Search returns incomplete evidence, agent doesn't know it

Example: Customer research agent User query: "What features does Competitor X have?"

Ideal scenario (demo): Search result: Complete features page (pricing, specs, integrations) Agent: Reads full page, returns accurate summary Result: ✓ Correct

Real scenario (production): Search result 1: Main website (homepage only) Search result 2: Old blog post (6 months old, outdated features) Search result 3: Twitter mention (missing details) Agent: Combines all 3, hallucinates missing features Agent thinks: "I have enough data" (actually has 40% of truth) Agent returns: "Competitor has features X, Y, Z" (Z is hallucinated)

User: Makes business decision based on hallucinated feature
User: Later discovers feature Z doesn't exist
User: Loses deal / makes wrong strategy
Your cost: R$100k+ deal loss

Without architecture fix: Agent doesn't know it failed (silent failure) User doesn't know agent hallucinated (until it's too late) You lose customer trust

With production architecture: Agent checks: "Is my evidence complete?" (confidence scoring) Agent decides: "Need better data, search again" Agent runs targeted follow-up search Agent gets: Complete feature list Agent returns: Accurate, confident summary

Failure 2: Infinite retry loops (cost explosion)

Agent keeps retrying failed searches (no stopping condition)

Example: Lead research agent User query: "Find budget for this company's marketing spending"

Ideal scenario (demo): Search: Finds financial report (budget clearly stated) Agent: Returns answer Cost: R$0.10 (one search)

Real scenario (production): Search 1: No relevant results Agent logic: "I failed, retry" Search 2: Still nothing Agent logic: "Still failed, retry with different query" Search 3: Different angle, still nothing Agent logic: "Try again..." Search 4-50: More retries...

Root cause: Company doesn't publicly disclose marketing budget
Agent doesn't know this (keeps searching anyway)
Result: 50 searches, still no answer, R$5 cost, 5 minutes wasted

Customer impact: User waited 5 minutes for "no answer" User frustrated (agent seemed "broken")

Your cost: R$5 per failed query 10 users × 2 failed queries/day × 20 business days = 400 failed queries = R$2,000/month waste (and customer churn)

Without architecture fix: Agent has no retry limit (keeps searching forever or until timeout) No backoff strategy (each retry uses same failed approach) No early exit (doesn't detect "impossible query")

With production architecture: Agent checks: "Have I tried N approaches? If yes, stop" Agent checks: "Is this query even answerable?" (early exit if no) Agent decides: "I need human help" or "This isn't publicly available" Agent costs: R$0.20 (2-3 targeted searches, then stops) Agent experience: User gets answer quickly or honest "not available"

Failure 3: Cost explosion (agent too expensive in production)

Agent designed for "accuracy over cost", breaks your margins

Example: Competitive intelligence agent Your pricing: R$100/month per customer Agent capability: Comprehensive competitor research (searches everything)

Demo scenario: User: "Research top 3 competitors" Agent: Does 10 searches (depth = good, cost = R$1.00) Demo works: Fast, accurate, cheap

Production scenario: Your customers: 100 users Each user: 5 research queries/month Total queries: 500/month Agent searches: 10 per query = 5,000 searches/month Cost per search: R$0.10 (API + compute) Total cost: 5,000 × R$0.10 = R$500/month

Your revenue: 100 users × R$100/month = R$10,000/month
Your agent cost: R$500/month
Your margin: R$9,500/month (looks good!)

Hidden problem (when adoption scales): Your marketing is successful Customers: 1,000 users (10x growth) Your revenue: 1,000 × R$100 = R$100,000/month (great!) Your agent cost: 1,000 × R$500 = R$500,000/month (oh no...)

Your margin: -R$400,000/month (you're paying to run agent)
You can't afford to scale
You disable agent (silent failure, customers lose feature)

Root cause: Agent designed for: Accuracy (search until confident) Agent designed for: Comprehensiveness (search everything) Agent NOT designed for: Cost efficiency (search minimal needed)

Without architecture fix: Agent searches: 10 queries per request (costs R$1.00) User pays: R$8/month for agent (R$100/user / 12) Agent costs you: R$1.00 (agent costs 12.5% of user fee) Scale to 10,000 users: Agent costs = R$5M/month (unsustainable)

With production architecture: Agent searches: 2-3 queries per request (costs R$0.20-0.30) Agent is cost-aware: Stops early if confidence reached Agent cost: 0.3% of user fee (much better) Scale to 10,000 users: Agent costs = R$300k/month (sustainable)

---

The production architecture (what TinyFish teaches)

Architecture pillar 1: Smart retrieval (incomplete data detection)

Agent knows when it has enough data (confidence scoring)

Technique: Evidence scoring

Traditional agent: Search → Summarize → Return No quality check = hallucinate with incomplete data

Production agent: Search → Score evidence (30% complete? 70%? 100%?) If incomplete: Search again (targeted follow-up) If complete: Summarize and return

Implementation: After each search, agent asks itself: "On a scale 0-100, how complete is my data?" "What's missing?" "Do I need more data?"

Agent decides:
  Score 80-100: "Confident, return answer"
  Score 50-79: "Partially confident, search for missing pieces"
  Score 0-49: "Insufficient, need completely different approach"
  

Example: Q: "What's Company X's revenue?"

Search 1: Finds 2023 revenue (R$100M)
Agent score: 60% (have 2023, but missing 2024-2025 trend)
Decision: Search again for recent results

Search 2: Finds 2024 guidance (R$120M projected)
Agent score: 85% (have recent trend)
Decision: Return answer with confidence "Current estimated R$120M"

Cost: 2 searches (R$0.20), confident answer
Without scoring: 5+ searches (R$0.50), possibly hallucinated answer

Architecture pillar 2: Smart retries (stopping conditions)

Agent knows when to stop retrying (backoff + early exit)

Technique: Retry strategy

Traditional agent: Query failed → Retry Retry failed → Retry again ...forever (until timeout or cost limit)

Production agent: Query failed → Retry (with different approach) Retry failed → Check if query is answerable If not answerable: Stop (return "not available") If answerable: Try again with new strategy Max 3-5 retries total (then give up)

Implementation: Retry 1: Initial search If fails → Classify why: "No results found?" → Try different search terms "Results irrelevant?" → Try broader search "Query impossible?" → Stop (early exit)

Retry 2: Use different keywords
  If fails → Different angle
  
Retry 3: Broader search
  If fails → STOP (query not answerable)
  Return: "Data not publicly available"
  

Benefit: Stops infinite loops (max 3 retries) Different approach each retry (not repeating failure) Early detection of impossible queries (don't waste time) Cost: Bounded (max R$0.30 per query, not R$5)

Architecture pillar 3: Cost limits (budget-aware agent)

Agent respects budget constraints (stops when limit reached)

Technique: Cost budgeting

Traditional agent: User query: "Research this thoroughly" Agent: Does 10 searches (costs R$1.00) Agent: Returns great answer, but expensive Scale: 1,000 queries = R$1,000 cost (unsustainable)

Production agent: User query: "Research this thoroughly" Agent budget: R$0.20 per query (max) Agent: "I have R$0.20, that's ~2 searches" Agent: Search 1 (R$0.10) + Search 2 (R$0.10) = R$0.20 budget used Agent: Stop (budget reached), return best answer found Cost: R$0.20 (sustainable)

Implementation: Before agent starts: Define budget per query: R$0.10, R$0.20, R$0.50 (based on use case)

During execution:
  Agent tracks cost: "I've spent R$0.12, have R$0.08 left"
  Agent decides: "That's only 1 more search, do it wisely"
  
Smart optimization:
  Instead of: 10 broad searches (hits all angles)
  Do: 2-3 targeted searches (highest ROI)
  

Benefit: Cost predictable (know max spend per query) Margins protected (agent costs ≤ revenue per user) Scales without breaking economics

---

How to implement production architecture (3 steps)

Step 1: Define agent SLOs (Service Level Objectives)

Before building, define what "good" looks like

Examples (for different use cases):

Use case 1: Lead research agent Response time: <30 seconds Accuracy: >80% (human acceptable) Cost per query: <R$0.20 Retry limit: 3 (max) Confidence required: >70% before returning

Use case 2: Competitive intelligence agent Response time: <2 minutes (user will wait) Accuracy: >90% (business decision critical) Cost per query: <R$0.50 Retry limit: 5 (more thorough) Confidence required: >85% before returning

Use case 3: Customer support agent Response time: <5 seconds (chat expectation) Accuracy: >95% (support quality critical) Cost per query: <R$0.05 (high volume, low margin) Retry limit: 1 (no time for retries) Confidence required: >90% or escalate to human

Your SLO definition: [ ] Response time: ___ seconds [ ] Accuracy target: % (how sure before answering?) [ ] Cost budget: R$_ per query [ ] Max retries: ___ attempts [ ] Confidence threshold: __% before returning answer

Step 2: Implement smart search (evidence scoring)

After each search, agent judges data quality

Pseudo-code (simplified):

function agent_search(query, max_budget): budget_remaining = max_budget confidence = 0 data = []

while confidence < 85 AND budget_remaining > 0 AND retries < 3:
  search_results = search(query)
  budget_remaining -= cost_of_search
  
  # Score the evidence
  confidence = score_evidence(search_results, query)
    # Returns 0-100 (0=useless, 100=perfect)
  
  data.append(search_results)
  
  if confidence < 70:
    # Refine query for next search
    query = refine_query(query, search_results)
  elif confidence < 85:
    # Targeted follow-up
    query = ask_for_missing_info(query, data)
  else:
    # Good enough, stop
    break
  
  retries += 1

return summarize(data)

Key insight: Agent knows when to stop (confidence threshold)

vs traditional agent that searches forever

Step 3: Monitor production (cost + quality)

Track both metrics (cost per query, accuracy rate)

Dashboard metrics:

[ ] Cost per query (track trends) Target: R$0.10-0.50 (depends on use case) Alert if: > R$1.00 (something wrong)

[ ] Accuracy rate (user satisfaction) Target: >80% (agent answer matches reality) Alert if: < 70% (users report bad answers)

[ ] Retry rate (how often agent retries) Target: <20% of queries need retry Alert if: >50% (queries are hard, need rethink)

[ ] Response time (user experience) Target: <30 seconds Alert if: >2 minutes (slow, consider reducing depth)

[ ] Budget overage rate (queries exceeding budget) Target: <5% of queries exceed budget Alert if: >20% (budget is too tight, increase it)

Action items: If cost too high: Reduce search depth, tighten budget, use cheaper searches If accuracy too low: Increase confidence threshold, more retries, better prompts If response slow: Reduce search depth, parallelize searches, cache results

---

What to do now (this week)

Audit your current agent

[ ] List all agents you've deployed [ ] For each agent, measure: - Average cost per query (R$?) - Success rate (% of queries answered) - Response time (average seconds) - User complaint rate (% reported issues)

[ ] Identify failure patterns: "Most failures due to incomplete data?" "Retry loops taking forever?" "Cost higher than expected?"

[ ] Pick one agent to optimize (start with highest ROI)

Implement one production pattern

[ ] Choose one pattern to add: Option 1: Evidence scoring (detect incomplete data) Option 2: Retry strategy (stop infinite loops) Option 3: Cost budgeting (control spending)

[ ] Implement on test agent [ ] Measure improvement Before: Cost R$0.50/query, 3 retries avg After: Cost R$0.20/query, 1 retry avg

[ ] Roll out to production (if successful)

Define SLOs for next agent

[ ] Before deploying new agent: [ ] Define response time requirement [ ] Define accuracy target [ ] Define cost budget per query [ ] Define max retries [ ] Define confidence threshold

[ ] Build agent with these SLOs in mind [ ] Monitor against SLOs in production

---

The bottom line: Architecture beats model quality

What most companies get wrong:

They think: "Better model (Claude vs GPT) = better agent" Reality: "Better architecture = better agent (at scale)"

They deploy: Agent with high-quality model but no production safeguards Result: Works in demo, fails in production

They measure: Agent accuracy in tests They ignore: Agent cost, retry loops, incomplete data in production

They learn: Too late (customer complains, cost explodes)

What production teams do:

They define: SLOs (cost, speed, accuracy, retries) They implement: Smart architecture (evidence scoring, retry limits, budgets) They monitor: Real production metrics (not just test accuracy) They iterate: Based on production data (not test results)

Result: Agents that work in demo AND production Costs that stay within budget Users that trust the agent Company that scales profitably

→ OpenClaw: Agent Architecture, Production Reliability, Search Optimization, Cost Control, Agent Monitoring

Your agent works in demo but breaks in production (incomplete retrieval, retry loops, cost explosion). Production architecture = evidence scoring + smart retries + cost budgeting. Start with SLOs (define expectations), implement one pattern, monitor real metrics. Demo success ≠ production success. 🔧📊✅