Seu agente tá overpromissado (realidade vs hype)

Seu agente tá overpromissado (realidade vs hype)

Notícia: Andrej Karpathy (ex-diretor IA OpenAI, lenda da IA) postou: "The gap in shared understanding of LLM capability is widening." Significado: Execs, investidores, clientes ACHAM que agentes LLM podem fazer X. Realidade: Agentes podem fazer 30% de X. Gap é ENORME e CRESCENDO.

Implicação: Seu agente tá sofrendo disso AGORA. Você prometeu: "Agente resolverá 80% das requisições." Realidade: Agente resolve 40%. Cliente: "Vocês mentiram." Você: "Não, LLM é assim mesmo." Cliente: "Outras empresas disseram 80%." Resultado: Cliente vai embora. CHURN.**

Problema: Hype vs Reality é o maior KILLER de agentes SaaS agora. Não é technology (tech tá bom). É EXPECTATIVA MANAGEMENT. Você tá vendendo promessas que LLM não consegue manter. Cliente descobre depois = raiva + refund.**

Verdade: LLMs são INCRÍVEIS em algumas coisas (entender context, gerar texto). Mas SÃO RUINS em outras (math exato, dados recentes, reasoning complexo). Você PRECISA entender: Onde agente é BOM (venda isso). Onde é RUIM (não prometa isso). Se não fizer: CHURN MASSIVO.**

REALIDADE (KARPATHY'S INSIGHT): ├─ What execs THINK agentes podem fazer: │ ├─ "Resolver 90% de customer service tickets (autonomously)" │ ├─ "Make decisions based on complex business rules" │ ├─ "Guarantee accuracy (zero hallucinations)" │ ├─ "Handle edge cases (without human fallback)" │ ├─ "Learn from feedback (continuously improve)" │ └─ Implication: "We can fire most support staff, agente faz tudo" │ ├─ What agentes ACTUALLY can do: │ ├─ "Resolve 40-60% of tickets (with caveats)" │ ├─ "Apply simple rules (not complex logic)" │ ├─ "Hallucinate confidently (50% false info seems true)" │ ├─ "Fail on edge cases (needs human escalation)" │ ├─ "Improve slightly with fine-tuning (not continuous learning)" │ └─ Implication: "We still need humans, agente is helper only" │ └─ The gap: ├─ Exec expectation: 90% automation ├─ Reality: 40-60% automation ├─ Delta: 30-50% gap (HUGE) ├─ When discovered: Customer RAGE ("You sold me fantasy") ├─ When discovered: CHURN (customer leaves) ├─ When discovered: REFUND (if SaaS) └─ Your problem: You didn't manage expectations

EXAMPLE (REAL-WORLD CHURN CYCLE): ├─ Month 1: You pitch agente │ ├─ Your promise: "80% automation, reduce support team by 60%" │ ├─ Exec: "Perfect, we'll fire half our support staff" │ ├─ Exec signs: R$ 100K contract │ └─ Reality: Not mentioned │ ├─ Month 2: Deployment │ ├─ Agente running on 1000 tickets/day │ ├─ Resolution rate: 45% (autonomous) │ ├─ Escalation rate: 55% (needs human) │ ├─ Exec: "Wait, we said 80%?" │ └─ Tension: Growing │ ├─ Month 3: Reality hits │ ├─ Exec fires 30% of support staff (as planned) │ ├─ Agente + remaining staff handle load │ ├─ But: 55% escalation = staff overwhelmed │ ├─ Customers: "Support is slower now!" │ ├─ Exec: "This agente made things worse" │ └─ Blame: On you │ ├─ Month 4: Cancellation │ ├─ Exec: "We're canceling, agente doesn't work" │ ├─ You: "But LLM fundamentally has these limits..." │ ├─ Exec: "Competitor said 80%, why can't you?" │ ├─ You: "Competitor is lying (but won't say that)" │ ├─ Result: CHURN │ └─ Root cause: Expectation mismatch (not tech failure)

WHY THIS HAPPENS: ├─ Sales team: "Say whatever closes deal" (no tech knowledge) ├─ Exec: Believes hype (thinks AI is magic) ├─ Reality: LLM has known limitations (tech team knows) ├─ Gap: Sales promises > Tech reality ├─ Result: Mismatch discovered after purchase ├─ Outcome: Customer angry (felt deceived) └─ Lesson: Manage expectations (say truth, not hype)

KARPATHY'S POINT: ├─ Problem: Gap is WIDENING (not stable) ├─ Why: Hype increasing faster than capability ├─ Effect: Misalignment getting WORSE ├─ Timeline: Next 12 months = MASSIVE churn (when gap discovered) ├─ Who suffers: Companies that oversell ├─ Who survives: Companies that manage expectations correctly └─ Your choice: Oversell = short-term revenue, long-term CHURN OR: Undersell = conservative, but sustainable

---

Entender: O que agentes REALMENTE conseguem fazer

What LLMs are actually good at

THINGS LLMS DO WELL: ├─ Natural language understanding │ ├─ Example: "I'm angry" → "Customer is upset" │ ├─ Example: "What's my balance?" → "Customer wants account info" │ ├─ Accuracy: 85%+ (really good) │ └─ Use: Classify customer intent (then route to right system) │ ├─ Text generation + formatting │ ├─ Example: "Generate apology email" → Good result │ ├─ Example: "Summarize customer history" → Works well │ ├─ Accuracy: 80%+ (good for creative tasks) │ └─ Use: Draft responses (humans review + send) │ ├─ Simple reasoning (few steps) │ ├─ Example: "If customer bought X, they might like Y" │ ├─ Example: "Customer is angry + purchased recently = refund offer" │ ├─ Accuracy: 70%+ (okay, but not perfect) │ └─ Use: Suggest next actions (not autonomous decisions) │ ├─ Context-aware responses │ ├─ Example: Read customer history, adjust tone │ ├─ Example: Remember previous conversation context │ ├─ Accuracy: 75%+ (good) │ └─ Use: Personalized support (feel more human) │ └─ Knowledge retrieval ├─ Example: "How do I reset password?" → Find documentation ├─ Example: "What's our refund policy?" → Retrieve policy ├─ Accuracy: 70-80% (good if data is up-to-date) └─ Use: Answer common questions (faster than human lookup)

THINGS LLMS ARE TERRIBLE AT: ├─ Exact arithmetic │ ├─ Example: "100 + 500 = ?" → Often wrong (confident + wrong) │ ├─ Accuracy: 30% (worse than random) │ ├─ Problem: LLMs guess next token (not calculate) │ └─ Use: NEVER for math (use calculator instead) │ ├─ Complex multi-step reasoning │ ├─ Example: "Given these 10 rules, what should we do?" → Fails │ ├─ Example: "Analyze this contract for risk" → Misses nuances │ ├─ Accuracy: 50% (might as well guess) │ ├─ Problem: LLM loses track of logic after 3-4 steps │ └─ Use: NOT autonomous (needs human review) │ ├─ Current/recent information │ ├─ Example: "What happened today?" → LLM doesn't know │ ├─ Example: "Customer's account status (real-time)?" → Wrong │ ├─ Accuracy: 0% (training data is old) │ ├─ Problem: LLM trained on old data (April 2024 max) │ └─ Use: Combine with API calls (fetch real-time data first) │ ├─ Consistency across time │ ├─ Example: "Same question, asked twice" → Different answers │ ├─ Example: "Contradicts earlier answer" → Doesn't notice │ ├─ Accuracy: 40% (not reliable) │ ├─ Problem: LLM is probabilistic (random) │ └─ Use: Add rules + validation (force consistency) │ ├─ Ethical/safety decisions │ ├─ Example: "Is this request legal?" → Might say yes (wrong) │ ├─ Example: "Should we process this refund?" → Can hallucinate logic │ ├─ Accuracy: 60% (risky) │ ├─ Problem: LLM will rationalize anything │ └─ Use: NEVER alone (needs human sign-off) │ └─ Admitting uncertainty ├─ Example: "I don't know" → LLM invents answer instead ├─ Example: Hallucination: Confident + false (worst combo) ├─ Accuracy: 20% (LLM says "I know" when doesn't) ├─ Problem: LLM trained to be helpful (invents if needed) └─ Use: Add detection (flag uncertain answers for human review)

THE REAL CAPABILITY MATRIX: ├─ Strong (85%+): Intent understanding, text generation, context ├─ Medium (70-75%): Simple reasoning, retrieval, personalization ├─ Weak (40-60%): Complex logic, edge cases, consistency ├─ Terrible (0-30%): Math, current info, safety decisions, admitting uncertainty └─ Implication: Agente is HELPER (not autonomous solution)

TRUTHFUL PITCH (vs overpromise): ├─ TRUTH: "Agente will: │ ├─ Understand what customers want (85% accuracy) │ ├─ Suggest appropriate responses (80% good) │ ├─ Reduce response time (faster initial reply) │ ├─ Free up humans for complex cases (escalation) │ ├─ Result: 30-40% reduction in support costs (not 70%) │ └─ Outcome: Better customer experience (not full automation)" │ └─ OVERPROMISE (what most do): "Agente will: ├─ Resolve 80%+ of tickets (WRONG: 40-60%) ├─ Make autonomous decisions (WRONG: needs human approval) ├─ Handle all edge cases (WRONG: escalates 30-50%) ├─ Never hallucinate (WRONG: hallucinates constantly) ├─ Learn from feedback (WRONG: needs fine-tuning) └─ Outcome: Customer fires half staff, agente fails, CHURN"

---

Como alinhar expectativa com realidade

Strategy 1: Honest capability assessment (before sales)

IDEIA: ├─ Test agente on YOUR use cases (not generic benchmarks) ├─ Measure: What % of tickets can agente handle autonomously? ├─ Document: Realistic number (not hopeful number) ├─ Use: Ground sales pitch in REALITY ├─ Benefit: No expectation mismatch (customer gets what promised) └─ Result: Happy customer (not surprised later)

IMPLEMENTATION: ├─ Step 1: Create test set │ ├─ Sample: 100-200 real customer tickets (from your customers) │ ├─ Variety: Mix easy + hard + edge cases │ ├─ Labeling: For each, mark correct resolution (ground truth) │ ├─ Effort: 2-4 hours (worth it) │ └─ Output: Test set (use for all measurements) │ ├─ Step 2: Test agente │ ├─ Run: Deploy agente on test set │ ├─ Measure: % tickets resolved autonomously (no escalation) │ ├─ Measure: % resolved correctly (quality check) │ ├─ Measure: % escalations (needs human) │ ├─ Measure: % hallucinations (wrong confident answer) │ ├─ Example result: "45% autonomous, 40% correct, 55% escalation" │ └─ Document: THIS IS YOUR NUMBER (not hope, reality) │ ├─ Step 3: Calculate business impact │ ├─ Formula: (Autonomous % × Avg cost per ticket) = savings │ ├─ Example: (45% × R$ 50 per ticket) = R$ 22.50 per ticket saved │ ├─ Example: If 1000 tickets/month = R$ 22.5K saved │ ├─ Cost: Agente costs R$ 40K/month (example) │ ├─ Math: Saves R$ 22.5K, costs R$ 40K = LOSS (not ROI) │ ├─ Implication: "Agente is not profitable for this customer" │ └─ Decision: Don't sell to this customer (wrong fit) │ ├─ Step 4: Segment your market │ ├─ Segment A: High-volume commodity support (good for agente) │ │ ├─ Example: 80% of tickets are simple (password resets) │ │ ├─ Agente autonomy: 70-80% │ │ ├─ ROI: Positive (worth it) │ │ └─ Pitch: "Agente will reduce costs 50%" │ │ │ ├─ Segment B: Mixed support (partial agente fit) │ │ ├─ Example: 50% simple, 50% complex │ │ ├─ Agente autonomy: 45-55% │ │ ├─ ROI: Marginal (maybe not worth it) │ │ └─ Pitch: "Agente will reduce costs 25%, free up humans" │ │ │ └─ Segment C: Complex support (bad for agente) │ ├─ Example: 80% of tickets need human judgment │ ├─ Agente autonomy: 20-30% │ ├─ ROI: Negative (not worth it) │ └─ Pitch: "Don't sell to this segment (honest)" │ └─ Result: ├─ You now know: Who is good customer (A) vs bad (C) ├─ Sales pitch: Based on reality (not hype) ├─ Customer expectation: Aligned with actual results ├─ Outcome: Happy customer (got what promised) └─ Churn: Eliminated (no surprise disappointment)

EXAMPLE (CUSTOMER SERVICE AGENTE): ├─ Test set: 150 real tickets (from your current customers) │ ├─ Types: Password resets, billing questions, feature requests, bugs │ ├─ Difficulty: 50 easy, 70 medium, 30 hard │ └─ Ground truth: Labeled by your support team │ ├─ Agente test results: │ ├─ Autonomous resolution: 45% (can resolve alone) │ ├─ Quality (correct answer): 85% of autonomous │ ├─ Escalations: 55% (needs human) │ ├─ Hallucinations: 8% (false confident answers) │ ├─ Average response time: 3 seconds (good) │ └─ Breakdown by type: │ ├─ Password resets: 90% autonomous (LLM good at this) │ ├─ Billing questions: 50% autonomous (needs data lookup) │ ├─ Feature requests: 20% autonomous (complex reasoning) │ └─ Bugs: 10% autonomous (edge cases) │ └─ Business calculation: ├─ Current: 1000 tickets/day, 5 support staff, R$ 100K/month salary ├─ With agente (45% autonomous): │ ├─ Handled by agente: 450 tickets/day (no human time) │ ├─ Escalated to humans: 550 tickets/day (needs human) │ ├─ Humans now free: 450/1000 = 45% time saved │ ├─ Salary reduction: 5 × 45% = 2.25 FTE (1-2 staff can be cut) │ ├─ Savings: R$ 25K-50K/month │ ├─ Agente cost: R$ 40K/month (example) │ ├─ Net ROI: -R$ 10K to +R$ 10K (marginal, risky) │ └─ Recommendation: "Agente will improve service, cost-neutral" │ └─ Honest pitch: ├─ "Agente will handle 45% of tickets (not 80%)" ├─ "You can reduce staff by 1-2 people (not 3-4)" ├─ "ROI is marginal, but service improves (faster response)" ├─ "If ticket volume grows 20%, agente pays for itself" └─ "Realistic expectation = happy customer"

Strategy 2: Set escalation expectations (from day 1)

IDEIA: ├─ Customer WILL get escalations (55% of tickets) ├─ Tell them upfront (not surprise them later) ├─ Frame as feature (not failure) ├─ Set SLA for escalations (how fast human responds) ├─ Benefit: No disappointment when escalation happens └─ Result: Customer accepts limitations (expected)

IMPLEMENTATION: ├─ Step 1: Educate customer (before deployment) │ ├─ Message: "Agente is helper, not replacement" │ ├─ Message: "When agente can't handle, human takes over (quickly)" │ ├─ Message: "Escalations are NORMAL (not failure)" │ ├─ Message: "We're designing escalations as feature" │ ├─ Expectation: Set "45% autonomous" (not 80%) │ └─ Timeline: Discuss in sales, before contract signed │ ├─ Step 2: Define escalation SLA │ ├─ SLA: "Escalations handled by human within 2 hours" │ ├─ SLA: "Complex cases = human takes over (no delay)" │ ├─ SLA: "Escalation doesn't lose context (agente notes passed)" │ ├─ SLA: "Human sees agente analysis (faster resolution)" │ └─ Result: Escalation is SMOOTH (not jarring) │ ├─ Step 3: Dashboard for transparency │ ├─ Show: % autonomous vs escalated (real-time) │ ├─ Show: Average resolution time (total, including escalations) │ ├─ Show: Customer satisfaction (autonomous vs escalated) │ ├─ Show: Cost per ticket (with agente) │ ├─ Benefit: Customer can SEE value (not just feel disappointed) │ └─ Result: Customer trusts you (transparent) │ └─ Framing escalations as positive: ├─ "Escalation = intelligent routing (easy stuff automated)" ├─ "Human now focused on complex cases (better outcome)" ├─ "Customer gets both speed AND accuracy (not one or other)" ├─ "Agente learns what should escalate (improves over time)" └─ Result: Customer sees escalation as feature, not bug

EXAMPLE DASHBOARD:

Customer Support Dashboard ═════════════════════════════════════════════

Today's Tickets: 87 ├─ Autonomous (Agente): 39 (45%) ✓ ├─ Escalated (Human): 48 (55%) → 2h avg response time └─ Total resolution time: 4.2h (24% faster than before)

Customer Satisfaction: ├─ Autonomous tickets: 4.1/5 ⭐⭐⭐⭐ ├─ Escalated tickets: 4.3/5 ⭐⭐⭐⭐ (humans are better) └─ Overall: 4.2/5 ⭐⭐⭐⭐

Cost per ticket: ├─ Before agente: R$ 100 (human only) ├─ With agente: R$ 85 (agente + human escalations) └─ Savings: 15% (R$ 15K/month)

System health: ├─ Hallucination rate: 6% (flagged for review) ├─ Escalation accuracy: 95% (correctly identified hard cases) ├─ Human satisfaction: "Agente escalations are well-prepared" └─ Trend: Improving (more autonomous over time)

Strategy 3: Set learning expectations (continuous improvement, not magic)

IDEIA: ├─ Agente WILL improve, but SLOWLY (not overnight) ├─ Improvement = fine-tuning + better prompts, NOT magic learning ├─ Timeline: 3-6 months to see meaningful improvement ├─ Benefit: Customer doesn't expect instant perfection └─ Result: Realistic expectations (achieve them, keep customer)

IMPLEMENTATION: ├─ Step 1: Explain improvement process │ ├─ "Month 1: Baseline (agente learns your data)" │ ├─ "Month 2: Optimization (we improve prompts)" │ ├─ "Month 3: Fine-tuning (agente trained on your tickets)" │ ├─ "Month 4-6: Continuous improvement (small gains each week)" │ ├─ "After 6 months: Autonomous rate up 5-10% (not 50%)" │ └─ Expectation: Set realistic (incremental, not dramatic) │ ├─ Step 2: Monthly review (show progress) │ ├─ Month 1: "Autonomous 45%" (baseline, no change expected) │ ├─ Month 2: "Autonomous 46%" (small improvement) │ ├─ Month 3: "Autonomous 48%" (optimization starting) │ ├─ Month 4: "Autonomous 50%" (fine-tuning working) │ ├─ Month 5: "Autonomous 52%" │ ├─ Month 6: "Autonomous 53%" (8% improvement total) │ └─ Message: "Small gains add up, worth it" │ └─ Set contract terms: ├─ SLA: "We guarantee 3-month trial" ├─ Improvement goal: "5-10% autonomous improvement by month 6" ├─ If not achieved: "We refund/adjust pricing" ├─ Benefit: You're confident (you'll hit goal) ├─ Customer: Incentivized to wait (knows timeline) └─ Result: Win-win (no premature cancellation)

EXAMPLE 6-MONTH JOURNEY: ├─ Month 0 (before): "Agente will improve our support" │ ├─ Baseline: 45% autonomous │ ├─ Expectation: "In 6 months, maybe 50%" (realistic) │ └─ Timeline: Customer agrees to 6-month trial │ ├─ Month 1: Onboarding │ ├─ Result: 45% autonomous (stable, expected) │ ├─ Reason: Agente learning your data │ ├─ Update: "Right on track, optimization starts month 2" │ └─ Sentiment: Satisfied (things working as expected) │ ├─ Month 2: Prompting optimization │ ├─ Result: 46% autonomous (+1%) │ ├─ Reason: Improved prompt templates │ ├─ Update: "Small gains visible, fine-tuning next month" │ └─ Sentiment: Happy (seeing improvement) │ ├─ Month 3: Fine-tuning │ ├─ Result: 48% autonomous (+2% more) │ ├─ Reason: Model fine-tuned on customer's tickets │ ├─ Update: "Fine-tuning working, acceleration continues" │ └─ Sentiment: Great (faster improvement) │ ├─ Month 4-6: Continuous optimization │ ├─ Result: 52-53% autonomous (+4-8% total) │ ├─ Reason: Continuous tweaks + escalation feedback loop │ ├─ Update: "Goal of 50% met (actually exceeded)" │ └─ Sentiment: Excellent (exceeded expectations) │ └─ Contract renewal: ├─ Customer attitude: "This is worth it, let's continue" ├─ Reason: Expectations were SET, and DELIVERED ├─ No surprise: Customer knew timeline + numbers ├─ Result: Happy renewal (not cancellation) └─ Lesson: Honest expectations = retention

---

Conclusão: Gerenciar expectativa é chave

Fatos:

✓ Karpathy disse: Gap é widening (execs expect too much) ✓ Reality: LLMs 40-60% autonomous (not 80%+) ✓ Problem: Sales overselling (gap between promise + delivery) ✓ Result: Customer discovers reality = CHURN ✓ Timeline: 2-3 months until disappointment ✓ Cost: Customer leaves, refund, reputation damage ✓ Solution: Honest assessment + transparent communication ✓ Benefit: Aligned expectations = happy customer ✓ Truth: Agente is HELPER (not replacement) ✓ Pitch: "45% autonomous, 55% escalation" (if true) ✓ vs: "80% autonomous, fire half staff" (if false = CHURN) ✓ Lesson: Integrity > Revenue (long-term)

ACTION ITEMS (THIS WEEK):

1. TODAY: Test YOUR agente on real tickets
2. THIS WEEK: Measure actual autonomy % (not hopeful %)
3. THIS WEEK: Document realistic number (ground truth)
4. THIS WEEK: Update sales pitch (based on reality)
5. NEXT WEEK: Brief sales team (honest messaging)
6. NEXT WEEK: Create customer onboarding (set expectations)
7. NEXT WEEK: Build escalation SLA (transparent)
8. MONTH 2: Create dashboard (customer sees progress)
9. MONTH 2: Review contract terms (set improvement goals)
10. ONGOING: Monitor for expectation mismatch (fix early)

Problema resolvido quando: └─ Sales pitch: Based on measured reality (not hype) └─ Customer expectation: Aligned with actual capability └─ Escalation process: Smooth (expected, not jarring) └─ Dashboard: Transparent (customer sees value) └─ Communication: Honest (build trust) └─ Improvement: Incremental (realistic timeline) └─ Contract renewal: Happy (not churn) └─ Customer satisfaction: High (exceeded expectations by being honest) └─ Revenue: Sustainable (long-term, not one-time)

→ OpenClaw: Expectation Management Framework (Honest Agente Pitch)

LLM capability gap crescendo. Seu agente? Overpromissado. Meça realidade (45% autonomy?), comunique honestamente (diga 45%, não 80%), gerencie escalações (smooth, expected), renove contrato (happy customer, not churn). Integridade > Revenue. 🎯