Agente IA sem QA (customers veem ruim antes você)

Agente IA sem QA (customers veem ruim antes você)

Você é founder/CEO de SaaS.

Seu SaaS: agente IA em produção (WhatsApp, suporte, vendas).

Seu cenário (muito comum):

• Your agente: Deployed (live with customers)
• Your assumption: "Agente está bom (testei antes de deploy)"
• Your reality: "Agente às vezes falha (customers reclamam, você não soube)"
  • Problem 1: "Customer got bad answer (agente halucinou)"
  • Problem 2: "You found out 3 DAYS later (from support ticket)"
  • Problem 3: "Customer left (bad experience damaged trust)"
  • Problem 4: "You didn't catch it (no QA system monitoring)"
• Your nightmare: "Agente degrading in production, customers suffering, I didn't notice. Churn happened while I was sleeping."

Your core problem:

• Agente quality degrades LIVE (model drift, prompt decay, integration issues)
• You have NO SYSTEM monitoring quality (flying blind)
• Customers experience bad agent first, you find out last
• By then: damage done (churn, negative reviews, trust lost)

The wake-up call (DiDi, September 2026):

• What DiDi did: Built automated QA system for contact center agents
• Why they did it: Couldn't manually QA thousands of conversations daily
• How they did it: Used LLM (Bedrock) to auto-evaluate agent responses
• What they measure: Quality, compliance, sentiment, accuracy (automated)
• What they found: QA automation catches issues BEFORE customers see them
• Result: Better customer experience, faster problem detection, scalable QA
• Your implication: "If DiDi needed auto-QA, I definitely need it."

---

Why manual QA fails (the illusion of control)

The manual QA trap

Your current process (if you have QA):

Setup: ├─ Deploy agente (live) ├─ Check: Manual spot-checks (read 5-10 conversations) ├─ Assumption: "If those 5 are good, agente is good" ├─ Frequency: Maybe weekly (when you remember) ├─ Coverage: ~0.01% of conversations (5 out of 50,000) └─ Problem: Sampling bias (you pick the easy cases)

What you miss: ├─ Edge cases: Rare scenarios agente fails (you didn't test them) ├─ Drift: Agente quality degrading over time (you don't monitor) ├─ Patterns: Systematic failures (specific question type always fails) ├─ Volume: Most conversations untouched (no visibility) ├─ Timing: Bad responses happen NOW, you find out LATER └─ Result: Flying blind (agente could be terrible, you don't know)

Real example: ├─ Monday: Deploy agente (tested, looks good) ├─ Tuesday-Thursday: Agente silently failing (you don't know) ├─ Friday: Customer complaint ("Agente is terrible") ├─ Your response: "Wait, let me check..." (investigating) ├─ Reality: 4 days of bad customer experience (already happened) └─ Damage: Customer lost, negative review, trust damaged

Why manual QA doesn't scale

Math:

Volume: ├─ Your agente: 1,000 conversations/day (low estimate) ├─ Manual QA: 10 conversations/day (realistic) ├─ Coverage: 1% (you check 10 out of 1,000) ├─ Miss rate: 99% (you don't see 990 conversations) └─ Implication: Agente could be terrible, you'd never know

Frequency: ├─ Your time: Limited (CEO/CTO can't spend 8 hours QA-ing) ├─ Realistic: Maybe 1 hour/week (spot checks) ├─ Conversations checked: ~20 (out of 7,000 weekly) ├─ Coverage: 0.3% (you see less than 1%) └─ Implication: Most conversations completely unreviewed

Economics: ├─ Manual QA cost: $5-10/conversation (if outsourced) ├─ Your volume: 1,000 conversations/day = R$ 5-10K/day in QA cost ├─ Your budget: Can't afford full coverage (too expensive) └─ Reality: Partial coverage (only check critical conversations)

Gap: ├─ What you check: Critical conversations, obvious failures ├─ What you miss: Subtle degradation, edge cases, patterns ├─ Result: You're optimistic about quality (but data is limited) └─ Risk: Major problems undetected until customer complains

---

The DiDi solution (automated QA with LLM)

What DiDi built (architecture)

Their system:

Input: ├─ Every conversation (1000s/day) ├─ Agent response (what agente said) ├─ Context (customer question, conversation history) └─ Ground truth (what correct answer should be)

Process: ├─ Step 1: LLM evaluates response ("Is this response good?") ├─ Step 2: Score quality (1-10, pass/fail) ├─ Step 3: Flag issues (hallucination, off-topic, rude, etc) ├─ Step 4: Store results (database) └─ Step 5: Alert if problem (real-time or batch)

Output: ├─ Quality score per conversation (automated) ├─ Quality trends (daily/weekly) ├─ Problem flags (hallucination: 5%, off-topic: 2%, etc) ├─ Anomalies (if quality drops suddenly) └─ Dashboards (visibility, monitoring)

Benefit: ├─ Coverage: 100% of conversations (not just spot-checks) ├─ Frequency: Real-time evaluation (catch issues now) ├─ Cost: LLM evaluation cheap (vs manual QA) ├─ Scale: Works for 1,000s of conversations/day ├─ Transparency: Data-driven (not gut-feel) └─ Actionable: Pinpoint what's broken, fix fast

Why LLM-based QA works (for contact center agents)

The logic:

Assumption: ├─ LLM is good at judging if another LLM's response is good ├─ Why? Both understand language, context, quality criteria ├─ Advantage: LLM can evaluate fast + consistent + cheap

What LLM QA can measure: ├─ Accuracy: Does response answer the question correctly? ├─ Relevance: Is response on-topic (not tangent)? ├─ Tone: Is response appropriate (professional, empathetic)? ├─ Completeness: Does response address all customer needs? ├─ Compliance: Does response follow policies/guidelines? ├─ Sentiment: Is response positive or negative? ├─ Hallucination: Does response make up facts? └─ Grammar: Is response well-written?

Example: ├─ Customer: "Como faço pra resetar minha senha?" ├─ Agent response: "Você pode ir em Settings → Reset Password → done." ├─ LLM QA: "Accuracy: 100%, Tone: Professional, Completeness: Good." ├─ Score: 95/100 (PASS) ├─ Action: No alert (agente responding well)

Counter-example: ├─ Customer: "Como faço pra resetar minha senha?" ├─ Agent response: "O Bitcoin é muito volátil nesses dias." ├─ LLM QA: "Accuracy: 0%, Relevance: 0%, Hallucination: True." ├─ Score: 5/100 (FAIL) ├─ Action: Alert (agente is broken)

DiDi's scope (real-world scale)

Their deployment:

Languages: ├─ Spanish (Latin America) ├─ Portuguese (Brazil) └─ Both critical for customer experience

Business lines: ├─ Ride-hailing (DiDi + taxi) ├─ Food delivery (DiDi Food) ├─ Financial services (DiDi Pay) └─ All need quality QA (customer trust critical)

Scale: ├─ Conversations: 1000s per day (per business line) ├─ Agents: Multiple (different skills, teams) ├─ Coverage: 100% (all conversations evaluated) ├─ Frequency: Real-time or batch (daily) └─ Impact: Prevents customer dissatisfaction across 3 businesses

Migration: ├─ Before: Manual QA + third-party tool (opaque, expensive) ├─ After: Automated LLM QA (transparent, owned, scalable) ├─ Result: Better control, better visibility, better economics

---

How to implement agent QA (for your SaaS)

Step 1: Define QA criteria (what makes a good response)

Start with basics:

For each use case, define scoring:

├─ Accuracy (does response answer correctly?) │ ├─ Scale: 0-10 (0 = wrong, 10 = perfect) │ ├─ Threshold: 8+ = pass │ └─ Weight: 40% (most important) │ ├─ Relevance (is response on-topic?) │ ├─ Scale: 0-10 (0 = off-topic, 10 = perfectly relevant) │ ├─ Threshold: 8+ = pass │ └─ Weight: 20% │ ├─ Tone (is response appropriate?) │ ├─ Scale: 0-10 (0 = rude, 10 = perfect) │ ├─ Threshold: 7+ = pass │ └─ Weight: 20% │ ├─ Completeness (does response address all needs?) │ ├─ Scale: 0-10 (0 = incomplete, 10 = complete) │ ├─ Threshold: 7+ = pass │ └─ Weight: 15% │ └─ Compliance (does response follow policies?) ├─ Scale: 0-10 (0 = violates policy, 10 = compliant) ├─ Threshold: 10 = must pass (no exceptions) └─ Weight: 5% (but critical)

Overall score: ├─ Formula: (Accuracy×0.4) + (Relevance×0.2) + (Tone×0.2) + (Completeness×0.15) + (Compliance×0.05) ├─ Pass: ≥80 (else flag for review) ├─ Fail: <80 (investigate why) └─ Trend: Track daily average (if trending down = problem)

Step 2: Set up LLM-based evaluation

Implementation options:

Option A: DIY with LLM API (cheapest)

Setup: ├─ Tool: OpenAI API, Claude API, or local LLM ├─ Prompt: Define QA criteria (accuracy, relevance, tone, etc) ├─ Input: Customer question + agent response ├─ Output: Score (0-10) + reasoning ├─ Frequency: Real-time or batch (every conversation) ├─ Cost: ~$0.01-0.05 per evaluation (very cheap) └─ Effort: 1-2 weeks to build

Example prompt:

You are a quality assurance evaluator for customer support.

Evaluate the agent response on these criteria:

1. Accuracy: Does it correctly answer the customer question? (0-10)
2. Relevance: Is it on-topic and focused? (0-10)
3. Tone: Is it professional and empathetic? (0-10)
4. Completeness: Does it address all customer needs? (0-10)
5. Compliance: Does it follow company policies? (0-10)

Customer question: {customer_question} Agent response: {agent_response} Company policies: {policies}

Respond in JSON: {"accuracy": X, "relevance": X, "tone": X, "completeness": X, "compliance": X, "overall_score": X, "pass_fail": "PASS/FAIL", "reasoning": "..."}

Option B: AWS Bedrock (like DiDi)

Setup: ├─ Service: Amazon Bedrock (managed LLM service) ├─ Models: Claude, Llama, Mistral (your choice) ├─ Integration: API call per conversation ├─ Cost: Similar to DIY ($0.01-0.05 per evaluation) ├─ Advantage: Managed, scalable, multimodel └─ Effort: 1-2 weeks (more mature than DIY)

Benefit: ├─ Reliability: AWS manages infrastructure ├─ Scale: Handles 1000s of evaluations/day ├─ Flexibility: Switch models easily ├─ Monitoring: Built-in dashboards └─ Similar to DiDi approach (proven at scale)

Option C: Third-party QA tool (turnkey)

Setup: ├─ Service: Specialized agent QA platforms (e.g., Talkwalker, MonkeyLearn) ├─ Integration: API or UI ├─ Cost: Higher ($0.10-0.50 per evaluation) ├─ Advantage: Pre-built, validated, domain-specific └─ Effort: 1-2 weeks (plug-and-play)

Benefit: ├─ Expertise: Built for contact centers ├─ Features: Pre-configured for common metrics ├─ Support: Vendor support included └─ Trade-off: Less control, more cost

Step 3: Monitor + alert (catch problems early)

Dashboard setup:

Real-time metrics: ├─ Today's quality score: 85% (average) ├─ Pass rate: 92% (% of conversations passing QA) ├─ Trending: ↓ (down 3% from yesterday—watch this) ├─ Problems: Hallucination (5%), off-topic (2%), tone issues (1%) └─ Top issue: Hallucination (focus on fixing this)

Alerts: ├─ Quality drops below 80%: Email + Slack notification ├─ Specific conversation fails: Log for manual review ├─ Trend drops >5% day-over-day: Investigate (model drift? prompt issues?) ├─ Compliance violation: Immediate escalation └─ Frequency: Check daily (or hourly if high-volume)

Actions: ├─ Low quality: Investigate cause (model, prompt, data) ├─ Systematic failures: Retrain or fix prompt ├─ One-off failures: Manual review, document pattern └─ Compliance: Stop agente, fix, redeploy

---

Your QA implementation checklist

This week:

☐ Define QA criteria for your agente ├─ What makes a good response? (accuracy, tone, relevance, etc) ├─ Document scoring system (0-10, pass threshold) ├─ Weight each criteria (accuracy 40%, tone 20%, etc) ├─ Document edge cases (what should agente do?) └─ Owner: Product/Customer Success

☐ Choose QA implementation ├─ Option A: DIY with LLM API (cheapest, if you have eng) ├─ Option B: AWS Bedrock (managed, proven scale) ├─ Option C: Third-party tool (easiest, but pricier) ├─ Decision: Cost vs effort trade-off └─ Owner: CTO/Engineering

☐ Estimate scope ├─ Conversations/day: ___ ├─ Cost per evaluation: $0.01-0.50 ├─ Total QA budget/month: ___ ├─ Feasibility: Is budget OK? └─ Owner: Finance/CTO

Next sprint:

☐ Build QA system (MVP) ├─ Choose tool (API, Bedrock, or third-party) ├─ Implement scoring logic (code or config) ├─ Run on sample conversations (10-50) ├─ Validate scores (are they correct?) ├─ Adjust criteria if needed └─ Owner: Engineering

☐ Set up monitoring ├─ Dashboard: Real-time quality score ├─ Alerts: Email/Slack if quality drops ├─ Logging: Store results (database) ├─ Access: Who can see dashboard? └─ Owner: Engineering/Product

☐ Deploy QA to production ├─ Start: Sample of conversations (10%) ├─ Monitor: Daily check for issues ├─ Scale: Gradually move to 100% ├─ Adjust: Fix any QA logic issues └─ Owner: Engineering/Product

Ongoing (every week):

☐ Review QA dashboard ├─ Overall quality: Trending up or down? ├─ Problem areas: What's failing most? ├─ Patterns: Specific question types failing? ├─ Action: Fix top issues (retrain, prompt fix, etc) └─ Frequency: Weekly review (or daily if low quality)

☐ Investigate failures ├─ Sample failing conversations (read 5-10) ├─ Root cause: Why did agente fail? ├─ Fixable: Prompt change, training data, config? ├─ Action: Make fix, test, deploy └─ Owner: Engineering/Product

☐ Refine QA criteria ├─ Are scores accurate? (do they match your intuition?) ├─ Missing anything? (new criteria to add?) ├─ Adjust weights? (if some criteria matter more) └─ Every 2-4 weeks: Update scoring logic

---

Conclusion: QA automation catches problems before customers do

Signal (DiDi insight):

• Contact center agents need QA (quality control)
• Manual QA doesn't scale (too expensive, too slow)
• LLM-based auto-QA is fast + cheap + scalable
• 100% coverage (not just spot-checks)
• Real-time detection (catch problems now, not later)
• Lesson: If DiDi needed this, you do too

Your situation now:

• Your agente is live (customers using it)
• You hope quality is good (but you don't know for sure)
• You can't manually QA every conversation (not scalable)
• Customers might be getting bad responses (you don't know)
• By the time you find out (too late): churn happened

Your options:

Option 1: No QA (risky)

• Agente quality unknown (flying blind)
• Customers experience bad responses (undetected)
• Churn happens (you find out late)
• Risk: -R$ 50-200K/month (from undetected quality issues)
• Recommendation: HIGH RISK (avoid)

Option 2: Manual QA (limited)

• Spot-check conversations (10-20/week)
• Coverage: 1% (miss 99% of conversations)
• Issues: Detected slowly (not real-time)
• Cost: $1-5K/month (if outsourced)
• Limitation: Doesn't scale (can't QA everything)
• Recommendation: BETTER THAN NOTHING (but not ideal)

Option 3: Automated LLM-based QA (recommended)

• Every conversation evaluated (100% coverage)
• Real-time detection (issues caught now)
• Cost: $500-1,500/month (for typical volume)
• Scalable: Works for 1000s of conversations/day
• Data-driven: Quality metrics + alerts
• Benefit: Catch problems BEFORE customers suffer
• ROI: Save R$ 50-200K/month (from prevented churn)
• Recommendation: BEST APPROACH (do this)

At OpenClaw, we help SaaS teams implement agent QA:

• DEFINE: QA criteria (what makes good response?)
• BUILD: Auto-evaluation system (LLM-based scoring)
• MONITOR: Real-time dashboards (quality trends)
• ALERT: Automated alerts (catch issues early)
• INVESTIGATE: Root cause analysis (why did agente fail?)
• FIX: Iterate (retrain, prompt fix, config change)

Result: Agente quality visible. Problems caught fast. Customers happy. Churn prevented.

Seu agente IA está em produção?

Você sabe a qualidade dele (dados reais, não gut-feel)?

Você tem sistema de QA monitorando conversations (100% coverage)?

Você quer detectar problemas ANTES do customer reclamar?

Você quer saber exatamente qual % de respostas passam em QA?

Se não tem ou quer expert guidance (define QA criteria, implement auto-evaluation, set up monitoring, catch issues early, prevent churn):

Implementar QA Automático AGORA (define criteria, build LLM-based QA, monitor real-time, catch issues early, prevent churn, protect revenue) →