Seu agente IA é psicótico (gera fake news, mente pra cliente)

Seu agente IA é psicótico (gera fake news, mente pra cliente)

Você é founder/CEO de SaaS.

Seu SaaS: agente IA em produção (WhatsApp, suporte, vendas).

Seu agente hoje: Responde perguntas, vende, resolve tickets.

Seu assumption (DANGEROUS):

• "Agente só responde o que sabe (não inventa)"
• "Se agente não sabe, escala pra humano (nunca mente)"
• "Clientes confiam em agente porque é confiável"
• "Hallucinations são raros (edge case)"
• "Se agente erre, cliente vai avisá-lo (feedback loop)"

Your reality (breaking news):

• AI psychosis = agente mente convincentemente

  • Meaning: Inventa fatos (sounds real, is fake)
  • Meaning: Cita fontes inexistentes ("Lei 123/2026 diz que..." — não existe)
  • Meaning: Gera números aleatórios ("Seu saldo é R$ 45.789,32" — pulled from air)
  • Meaning: Faz promessas falsas ("Você tem direito a R$ 5K refund" — you don't)
  • Meaning: Muito prolífico (mente MUITO, não pouco)
  • Result: Cliente acredita, age, sofre dano

• What research shows (Defining AI Psychosis):

  • AI psychosis = confident hallucinations (wrong facts stated with certainty)
  • Prolific = happens often (not rare, happens in 10-30% of responses)
  • Convincing = sounds real (user can't tell lie from truth)
  • Dangerous = customer acts on false information
  • Invisible = you don't know it's happening (no error detection)
  • Question: Is your agente psychotic (you don't know)?

The signal (September 2024):

• Research: "Defining AI Psychosis, Part 2: Prolific AI Psychosis"
• Insight: Hallucinations are common + convincing + hard to detect
• Implication: Your agente might be lying to customers (right now)
• Risk: CNPJ fine (false information = fraud), customer lawsuit, reputational damage
• Opportunity: Implement detection + prevention

Your problem (quantified):

AI psychosis scenario (customer journey):

1. Customer asks via WhatsApp: "Posso devolver o produto fora do prazo?"

2. Your agente responds (confident, sounds real): "Sim, você tem 90 dias de direito de devolução mesmo após prazo. Isso está protegido pela Lei 123/2026 de Proteção ao Consumidor. Você pode iniciar o processo agora através do link: [link]"

3. What's wrong: ├─ "Lei 123/2026" doesn't exist (agente invented it) ├─ "90 days" might be wrong (your policy is 30 days) ├─ Link might be malformed (agente made it up) ├─ Promise is false (customer has 30 days, not 90) └─ Sounds 100% real (customer believes immediately)

4. Customer acts: ├─ Buys item again (thinking they have 90 days to return) ├─ Tries to return after 45 days (outside 30-day window) ├─ You refuse return (policy is 30 days) ├─ Customer is furious (agente promised 90 days) └─ Customer complains on social media

5. Damage: ├─ Customer churn: Lost customer forever ├─ Reputation: "Bad customer service, lied to me" ├─ Regulatory: CNPJ complaint (false information = fraud) ├─ Legal: Lawsuit (agente made promise you can't keep) ├─ Scale: If 10% of agente responses are psychotic: │ ├─ 1,000 customers/day × 10% = 100 false promises/day │ ├─ 100 × 30 days = 3,000 angry customers/month │ ├─ 3,000 × R$ 500 chargeback = R$ 1.5M loss/month │ └─ Risk = existential (agente is destroying business) └─ Timeline: You discover when CNPJ investigates (too late)

6. Root cause: You didn't detect hallucination └─ Agente sounded confident (you trusted it) └─ No quality check (hallucination passed through) └─ No guardrail (agente could promise anything) └─ No feedback loop (customer had to tell you it's wrong)

---

Why AI psychosis is worse than human psychosis (and why it matters)

The problem: Confident hallucinations are invisible

Human agent vs AI agent (when they make mistakes):

Human agent makes mistake: ├─ "Você tem 90 dias pra devolver" (wrong) ├─ But: You notice immediately │ ├─ "Wait, that doesn't match our policy" │ ├─ "Let me check with manager" │ └─ "Actually, it's 30 days, my mistake" ├─ Consequence: Corrected before damage ├─ Cost: Zero (caught before customer acts) └─ Pattern: Humans self-correct (have context awareness)

AI agent makes mistake (hallucination): ├─ "Você tem 90 dias pra devolver" (wrong, invented) ├─ But: You don't notice (no human in loop) │ ├─ Agente sounds confident │ ├─ Agente cites fake law ("Lei 123/2026") │ ├─ Agente provides link (real-looking but fake) │ └─ Looks completely legit ├─ Consequence: Sent to customer (damage happens) ├─ Cost: R$ 500-5K per incident (chargeback, reputation) ├─ Pattern: AI doesn't self-correct (no awareness)

Key difference: ├─ Human: "I think I made a mistake" (self-aware) ├─ AI: "I am 100% confident" (not aware it's wrong) ├─ Result: Human stops before harm, AI causes harm └─ Implication: AI mistakes are more dangerous

Why hallucinations are prolific (not rare):

Why AI hallucinates (frequently):

1. Training data problem ├─ LLM trained on internet (includes false info) ├─ LLM doesn't distinguish real from fake ├─ LLM just predicts next word (no fact-checking) └─ Result: Can output false facts confidently

2. Prompt problem ├─ You ask: "What's my account balance?" ├─ Agente has no access to database ├─ Agente hallucinates: "Your balance is R$ 45,789.32" ├─ Sounds real (numbers are specific) ├─ Completely false (agente invented number) └─ Result: False but confident

3. Knowledge cutoff problem ├─ You ask: "What's the new ICMS tax rate for 2026?" ├─ Agente trained on 2024 data (doesn't know 2026) ├─ Agente hallucinates: "The new rate is 19%" ├─ Might be wrong (actual rate is 21%) ├─ Agente doesn't know it's outdated └─ Result: Confidently outdated

4. Probability problem ├─ LLM = probability model (picks likely next word) ├─ "What law protects consumer return rights?" ├─ LLM picks: "Lei 123/2026" (sounds law-like, doesn't exist) ├─ LLM confidence: 95% (high) ├─ Actually exists: 0% (never existed) └─ Result: High confidence, zero truth

Frequency: Research shows hallucinations in 10-30% of responses ├─ FAQ (known answers): 2-5% hallucination rate (low) ├─ Complex questions: 20-40% hallucination rate (high) ├─ Knowledge questions: 10-30% hallucination rate (medium) └─ If your agente answers 1,000 customer questions/day: ├─ 100-300 contain hallucinations (per day) ├─ 3,000-9,000 per month (invisible) └─ Risk = continuous (not edge case)

Why you can't detect hallucinations (and why this matters)

The detection problem:

How to detect if agente hallucinated:

1. Read output ├─ "Você tem 90 dias pra devolver" ├─ Sounds plausible (90 days is reasonable) ├─ Hard to know if true (depends on policy) └─ Most people: Can't tell (sounds real)

2. Fact-check against policy ├─ Your policy: 30 days ├─ Agente output: 90 days ├─ Mismatch detected (this is hallucination) ├─ But: You need to check policy EVERY response ├─ 1,000 responses/day × 2 min check = 33 hours (impossible) └─ Result: You can't fact-check everything

3. Customer tells you it's wrong ├─ Customer: "Your agente promised 90 days" ├─ You: "No, it's 30 days" ├─ Customer: Angry (feels deceived) ├─ Discovery: Late (after customer affected) └─ Cost: Reputation damage, chargeback

4. AI tells you if it's confident ├─ Agente outputs: "I am 95% confident in this answer" ├─ Reality: 95% confident but 0% correct (hallucinating) ├─ Confidence ≠ correctness (false signal) └─ You can't trust confidence score

Conclusion: You can't detect hallucinations in real-time ├─ Too many responses to check ├─ Hallucinations sound real (hard to spot) ├─ Agente won't tell you (no self-awareness) ├─ Customer has to tell you (too late) └─ Result: Invisible damage happening right now

---

How to detect AI psychosis (before it damages business)

Detection strategy 1: Fact-checking layer

Add verification step (before sending to customer):

Architecture (with fact-checking):

1. Customer asks question └─ Input: "Posso devolver fora do prazo?"

2. Agente generates response └─ Output: "Sim, você tem 90 dias de direito de devolução"

3. Fact-checker evaluates response (NEW) ├─ Check 1: Is answer in FAQ/knowledge base? │ ├─ Search KB: "90 days" │ ├─ Result: Not found (doesn't match policy) │ └─ Flag: Potential hallucination │ ├─ Check 2: Does answer match policy database? │ ├─ Query DB: Return policy │ ├─ Result: 30 days (not 90) │ └─ Flag: HALLUCINATION DETECTED │ ├─ Check 3: Is answer citing sources? │ ├─ Agente said: "Lei 123/2026" │ ├─ Verify: Law doesn't exist │ └─ Flag: FAKE SOURCE │ └─ Final decision: ├─ If all checks pass: Send response to customer ├─ If any flag: Hold response, escalate to human └─ Result: Customer never sees hallucination

4. Safe response sent └─ (Only if fact-checked and verified)

Cost: Fact-checker API (R$ 100-500/month) Benefit: Prevents 90% of hallucination damage (before it reaches customer) ROI: Prevents R$ 500-5K loss per incident (1M+ payback)

Implementation options:

1. Rule-based checks (simple, fast) ├─ Check if answer contains policy facts ├─ Verify sources (must be in approved list) ├─ Flag if number mentioned (needs DB verification) ├─ Cost: R$ 5K-10K (engineering) ├─ Accuracy: 70% (misses subtle hallucinations) └─ Best for: FAQ, known patterns

2. Semantic checks (moderate, slower) ├─ Use another LLM to verify first LLM ├─ "Does this answer match our policy?" ├─ "Is this law/source real?" ├─ Cost: R$ 500-1K/month (API calls) ├─ Accuracy: 85% (better detection) └─ Best for: Mixed workload

3. Knowledge base grounding (best, most complex) ├─ Agente only answers from KB (no hallucination possible) ├─ Retrieval-augmented generation (RAG) ├─ "Answer only using these documents" ├─ Cost: R$ 20K-50K (architecture + KB setup) ├─ Accuracy: 98% (no hallucination from KB) └─ Best for: Critical applications (finance, legal, medical)

Detection strategy 2: Confidence calibration

Trust agente's confidence (but verify when uncertain):

Confidence-based routing:

1. Agente generates response with confidence score ├─ Response: "Você tem 90 dias" ├─ Confidence: 92% └─ Actually correct: 0% (hallucination, but sounds confident)

2. Problem: Confidence ≠ correctness ├─ Overconfident hallucinations are common ├─ "I am 95% sure" doesn't mean correct └─ Can't trust confidence alone

3. Solution: Cross-check only low-confidence responses ├─ If confidence > 85%: Send directly (assume correct) ├─ If confidence < 85%: Fact-check before sending ├─ If confidence < 50%: Escalate to human └─ Result: Reduce fact-checking load (only uncertain answers)

4. Better solution: Calibrated confidence ├─ Measure agente's actual accuracy (ground truth) ├─ Example: Agente says 95% confident ├─ Actual accuracy of "95% confident" answers: 70% ├─ Calibration: Discount confidence by 25 points ├─ Adjusted confidence: 70% (closer to reality) └─ Use calibrated score for routing

Benefit: Skip fact-checking for high-confidence correct answers (save resources) Risk: Overconfident hallucinations bypass check (still dangerous) Recommendation: Combine with fact-checking (confidence is signal, not guarantee)

Detection strategy 3: Monitoring & alerts

Track hallucinations in production (catch them early):

Monitoring setup:

1. Log all agente responses ├─ What agente said ├─ Confidence score ├─ Customer feedback (if any) ├─ Human review result └─ Actual outcome (was answer correct?)

2. Measure hallucination rate ├─ Track: % of responses that were hallucinations ├─ Example: 3% of responses contained false facts ├─ Trend: Is it increasing or decreasing? ├─ Alert: If > 5%, something is wrong └─ Action: Pause agente, investigate

3. Identify hallucination patterns ├─ "Which questions cause most hallucinations?" ├─ Example: Complex legal questions (40% hallucination) ├─ Example: Future predictions (60% hallucination) ├─ Example: Specific numbers (30% hallucination) └─ Action: Flag high-risk questions for human review

4. Customer feedback loop ├─ Prompt: "Was agente's answer helpful?" ├─ If NO: Mark as potential hallucination ├─ Review: Human checks marked responses ├─ Learn: Improve agente from feedback └─ Alert: If >10% negative, escalate

5. Set up alerts ├─ Alert if hallucination rate > 5% ├─ Alert if customer complaint mentions agente ├─ Alert if CNPJ complaint about false information ├─ Alert if agente cites law/source that doesn't exist └─ Action: Investigate + fix immediately

Cost: Monitoring + logging (R$ 2K-5K/month) Benefit: Early detection (catch problems before scale) ROI: Prevents massive damage (reputation, legal, regulatory)

---

How to prevent AI psychosis (architecture strategies)

Strategy 1: Retrieval-augmented generation (RAG)

Force agente to answer ONLY from approved sources:

RAG architecture:

1. Customer asks: "Posso devolver fora do prazo?"

2. Agente retrieves relevant documents ├─ Search: "return policy, return window, devolver" ├─ Find: "Returns Policy v2.1" (document in KB) ├─ Extract: "30 days from purchase" (relevant text) └─ Guarantee: Answer comes from official document

3. Agente generates response from document ├─ Input: "Based on this document, answer the question" ├─ Document: "Returns Policy v2.1: 30 days from purchase" ├─ Response: "Você tem 30 dias para devolver a partir da compra" ├─ Quality: Can't hallucinate (answer is grounded) └─ Source: Document is cited (customer can verify)

4. Customer gets grounded, verified answer ├─ Answer: "Você tem 30 dias..." (correct) ├─ Source: "Returns Policy v2.1" (verifiable) ├─ Confidence: 100% (document is truth) └─ Risk: Zero (can't lie if answer comes from KB)

Benefit: Zero hallucinations (impossible to invent when grounded in docs) Cost: R$ 20K-50K (RAG setup + KB maintenance) Best for: Policy Q&A, FAQ, documentation-heavy support Limitation: Only works for documented information (can't handle novel questions)

Strategy 2: Structured outputs

Constrain agente to safe response format:

Structured output constraint:

Instead of free-text: ├─ "You can return items because of a special law" ├─ (Can hallucinate reasons, cite fake laws)

Use structured format: ├─ Answer: [YES / NO / ESCALATE_TO_HUMAN] ├─ Reason: [POLICY_MATCH / SPECIAL_CASE / UNKNOWN] ├─ Policy_section: [ID from approved policy] ├─ Reference: [Document URL] ├─ Confidence: [0-100]

Example: { "answer": "NO", "reason": "POLICY_MATCH", "policy_section": "RETURNS_POLICY_v2.1_SECTION_3", "reference": "https://company.com/policy#returns", "confidence": 92, "message": "Você tem 30 dias para devolver a partir da compra conforme nossa política de devoluções." }

Benefit: Forces agente to cite sources (can be verified) Benefit: Constrains output (harder to hallucinate) Cost: R$ 5K-10K (schema design + validation) Best for: Customer-facing responses Limitation: Reduces flexibility (can't handle complex scenarios)

Strategy 3: Human-in-the-loop (HITL)

Have human review high-risk responses:

Human review workflow:

1. Agente generates response └─ Output: "Você tem 90 dias para devolver"

2. Risk classifier evaluates risk ├─ Risk level: HIGH (makes promise about policy) ├─ Reason: Financial/legal implication ├─ Action: Require human review └─ Queue: Add to human review queue

3. Human reviews response ├─ Read agente output ├─ Check against policy database ├─ Decision: Approve or reject ├─ If approve: Send to customer ├─ If reject: Edit + send corrected version └─ Learn: Improve agente based on correction

4. Customer gets verified response ├─ Response was reviewed by human ├─ Guaranteed accurate (human checked) ├─ Cost: Human time (R$ 50-100 per response) └─ Quality: 100% (human verified)

Benefit: Catches hallucinations before customer sees them Cost: R$ 10K-50K/month (depends on volume) Best for: Critical interactions (sales, policy, compliance) Limitation: Doesn't scale (humans get overwhelmed)

Optimization: Risk-based HITL ├─ High risk (policy, numbers, promises): 100% human review ├─ Medium risk (explanations, generic): 20% human review (sampling) ├─ Low risk (FAQ, known answers): 0% human review (auto-approve) └─ Result: Scale human effort (only review what matters)

---

Conclusion: AI psychosis is a silent crisis (fix it before it spreads)

The problem (summary):

• Agente hallucinates (confident false facts)
• Prolific (happens 10-30% of time)
• Invisible (you don't notice until customer complains)
• Damaging (R$ 500K-5M risk if at scale)
• Compliance (CNPJ fine for false information = fraud)

Your decision (3 paths):

1. Do nothing (hope for best)

   • Risk: High (hallucinations spreading daily)
   • Cost of inaction: R$ 500K-5M (fines, lawsuits, churn)
   • Detection: Late (customer tells you)
   • Recommendation: Don't do this

2. Monitor + catch (moderate effort)

   • Fact-checker layer (R$ 5K-10K setup + R$ 500/month)
   • Monitoring + alerts (R$ 2K-5K/month)
   • Human review (sampling, 20% of risky responses)
   • Cost: R$ 10K setup + R$ 3K-7K/month
   • Benefit: Catches 80-90% of hallucinations
   • Timeline: 4-6 weeks to implement
   • ROI: Prevents R$ 500K+ loss (100x payback)

3. Build robust system (high effort, best)

   • RAG (knowledge base grounding): R$ 30K-50K
   • Structured outputs: R$ 5K-10K
   • HITL (human review for high-risk): R$ 10K-50K/month
   • Monitoring + alerting: R$ 2K-5K/month
   • Total: R$ 50K setup + R$ 15K-60K/month
   • Benefit: 99%+ prevention (nearly impossible to hallucinate)
   • Timeline: 2-3 months to implement
   • ROI: Enables enterprise trust (can scale confidently)

At OpenClaw, we help SaaS build psychosis-resistant agents (detection + prevention):

• AUDIT: Current hallucination rate (how often is agente lying?)
• DETECT: Fact-checking layer (catch hallucinations before customer sees)
• PREVENT: RAG + structured outputs (make hallucinations impossible)
• MONITOR: Hallucination tracking + alerts (early warning system)
• REVIEW: HITL workflow (human safety net for critical responses)
• VERIFY: Quality assurance (prove agente is honest)
• SCALE: Enterprise-grade system (production-safe)

Result: Agente que é confiável por design (não por esperança). Clientes confiam porque você garantiu qualidade. Zero fake news. Zero legal risk.

Seu agente está psicótico (alucinando)?

Você sabe a taxa de hallucination (ou assume é baixa)?

Você detecta mentiras antes de chegarem no cliente (ou descobre tarde)?

Você tem guardrails (ou agente pode prometer qualquer coisa)?

Você quer agente honesto (100% verificado, zero fake news)?

Se quer expert guidance (hallucination detection, prevention, RAG, HITL, monitoring, quality assurance, enterprise-grade):

Agente IA Psychosis-Proof | Hallucination Detection & Prevention | RAG | HITL | Quality Assurance →