Seu agente de automação é fraco (culpa do harness, não do LLM)

Seu agente de automação é fraco (culpa do harness, não do LLM).

Você é founder de SaaS.

Seu agente de automação:

• Faz tarefas (código, workflows, integração de APIs)
• Your assumption: "Agente fraco? Upgrade pro modelo melhor (GPT-5, Claude-4)."
• Reality: "Pesquisa empirical descobriu: Harness design = 80% do sucesso (not model)."
• Your blind spot: ├─ Modelo: 20% da qualidade do agente ├─ Harness (arquitetura): 80% da qualidade do agente ├─ You focusing: Modelo melhor (mais caro) ├─ You ignoring: Harness melhor (free, só estrutura) ├─ Result: "Pagando caro por upgrade inútil (problema é arquitetura)." └─ Implicação: "Sua estratégia de melhoria é backward (solving wrong problem)."

Pesquisadores de Carnegie Mellon/Stanford descobriram:

"Agent harness design = 80% da performance. Model quality = 20%. Implicação: Melhorar harness (prompts, tool integration, feedback loops) = 4x mais impacto que upgrade de modelo. Teste: Mesma tarefa, mesmo modelo, harness diferente: Performance varia 60-90% (apenas pela arquitetura)."

Translation to your SaaS:

• Old belief: "Better model = better agent (spend on GPT-5)."
• New reality: "Better harness = better agent (improve architecture free)."
• Implication: "You're wasting money on models (problem is harness)."
• Opportunity: "Fix harness = 4x improvement (same cost, better results)."

---

O Problema: Você investe em modelo, ignora arquitetura

Por que harness design importa mais que modelo

=== THE HARNESS DISCOVERY ===

What research found: ├─ Agent performance = f(model, harness) ├─ Model contribution: ~20% of variance ├─ Harness contribution: ~80% of variance ├─ Implication: "Harness >> model (by 4x)." └─ Surprise: "Everyone invests in model, ignores harness."

What is "harness"? ├─ Prompts (how you ask model) ├─ Tool integration (what tools agent has access to) ├─ Feedback loops (how agent learns from mistakes) ├─ Execution environment (how agent runs code) ├─ Error handling (how agent recovers from failures) ├─ Reasoning structure (how agent reasons through problems) ├─ Memory management (how agent remembers context) ├─ Safety guardrails (how agent avoids mistakes) └─ Monitoring (how you measure agent performance).

Example breakdown: ├─ GPT-4 (model): $0.03 per 1K tokens ├─ But: Without good harness = 30% success rate ├─ Claude-3 (model): $0.015 per 1K tokens ├─ With: Good harness = 85% success rate ├─ Result: "Cheaper model + better harness = 2.8x better (lower cost)." └─ Your situation: "You using expensive model with bad harness (wasting money)."

=== HARNESS COMPONENTS THAT MATTER ===

1. Prompt Engineering (how you ask the model) ├─ Bad harness: │ └─ "Write code to process CSV file" ├─ Good harness: │ ├─ "You are code generator for CSV processing" │ ├─ "Your goal: Write Python code (pandas library)" │ ├─ "Input: CSV with columns [name, age, email]" │ ├─ "Output: Deduplicated CSV (by email)" │ ├─ "Error handling: Log to file if email invalid" │ ├─ "Testing: Include pytest test cases" │ ├─ "Example:" │ │ └─ "# Example test case (shows expected format)" │ └─ "Edge cases: Handle empty file, missing columns" ├─ Impact: 30% → 85% success (just better prompt) └─ Cost: Free (same model, better prompt).

2. Tool Integration (what agent can access) ├─ Bad harness: │ └─ Agent has 1 tool: "execute_code" ├─ Good harness: │ ├─ Agent has: "read_file", "write_file", "execute_code" │ ├─ Agent has: "test_code", "lint_code", "debug_code" │ ├─ Agent has: "git_commit", "github_pr", "slack_notify" │ ├─ Agent has: "database_query", "api_call", "cache_check" │ └─ Agent has: "ask_human" (when stuck) ├─ Impact: Agent can solve 3x more problems (more tools) └─ Cost: Free (just better architecture).

3. Feedback Loops (how agent improves from mistakes) ├─ Bad harness: │ ├─ Agent runs code → code crashes │ ├─ Agent doesn't see error │ ├─ Agent repeats mistake next time │ └─ Result: Agent stuck (can't learn) ├─ Good harness: │ ├─ Agent runs code → code crashes │ ├─ Agent sees error (stderr, exit code) │ ├─ Agent analyzes error ("TypeError: list not iterable") │ ├─ Agent fixes code (change logic) │ ├─ Agent retries (up to 3 times) │ ├─ Agent learns (next time won't repeat mistake) │ └─ Result: Agent improves (self-correcting) ├─ Impact: Error recovery 5x better (feedback loop) └─ Cost: Free (just better architecture).

4. Execution Environment (how agent runs code safely) ├─ Bad harness: │ ├─ Agent runs code in main process │ ├─ Code crashes → entire system crashes │ ├─ Code infinite loop → system hangs │ ├─ Code malicious → steals data │ └─ Result: Risky (no isolation) ├─ Good harness: │ ├─ Agent runs code in sandbox (isolated container) │ ├─ Code crash → only sandbox crashes (system safe) │ ├─ Code timeout → killed after 30 seconds (prevent hang) │ ├─ Code permissions → read-only access (prevent steal) │ ├─ Code monitoring → CPU, memory limits (prevent resource abuse) │ └─ Result: Safe (isolated execution) ├─ Impact: Reliability 10x better (no crashes) └─ Cost: ~$100/month (sandboxing service).

5. Reasoning Structure (how agent thinks through problems) ├─ Bad harness: │ └─ "Generate code" → Agent writes code directly ├─ Good harness (Chain-of-Thought): │ ├─ "Step 1: Understand problem" → Agent analyzes │ ├─ "Step 2: Plan approach" → Agent outlines solution │ ├─ "Step 3: Write code" → Agent writes code │ ├─ "Step 4: Test code" → Agent tests │ ├─ "Step 5: Optimize" → Agent optimizes │ └─ Result: Better code (thoughtful) ├─ Good harness (Tree-of-Thought): │ ├─ "Approach A: Use pandas" → Generate code → Test │ ├─ "Approach B: Use polars" → Generate code → Test │ ├─ "Compare: Which is faster?" → Choose best │ └─ Result: Optimal solution (explored options) ├─ Impact: Code quality 60% better (better reasoning) └─ Cost: Free (just better structure).

=== REAL WORLD EXAMPLE ===

Scenario: Agent generates code to process CSV file (deduplicate)

BAD HARNESS (30% success): ├─ Prompt: "Write Python code to remove duplicate rows from CSV" ├─ Result: │ ├─ 70% of outputs: Syntax errors, logic bugs, missing imports │ ├─ 30% of outputs: Working code │ └─ Reason: Model guessing (no structure)

GOOD HARNESS (85% success): ├─ Prompt: [detailed instruction] + examples + error handling ├─ Tool: read_file, write_file, execute_code, test_code ├─ Feedback: If code fails, show error, agent fixes ├─ Sandbox: Run in isolated container (safe) ├─ Reasoning: Chain-of-Thought (step by step) ├─ Result: │ ├─ 85% of outputs: Working code │ ├─ 10% of outputs: Works but suboptimal │ ├─ 5% of outputs: Fails (agent knows + retries) │ └─ Reason: Structure guides model (better output)

COST COMPARISON: ├─ Bad harness + GPT-4 Turbo: $0.03 per 1K tokens │ ├─ Success rate: 30% (need 10 tries for 3 successes) │ ├─ Effective cost: $0.10 per successful output │ └─ Time: 10 minutes (many retries) ├─ Good harness + GPT-4 Turbo: $0.03 per 1K tokens │ ├─ Success rate: 85% (need 1.2 tries) │ ├─ Effective cost: $0.035 per successful output │ └─ Time: 2 minutes (faster, fewer retries) ├─ Good harness + Claude-3.5 Sonnet: $0.015 per 1K tokens │ ├─ Success rate: 85% (same harness) │ ├─ Effective cost: $0.018 per successful output │ └─ Time: 2 minutes (faster, cheaper) └─ WINNING COMBO: Good harness + cheaper model = 5x cheaper + 2.8x faster.

=== WHY YOU'RE GETTING THIS WRONG ===

Common mistakes: ├─ Mistake 1: "My agent is bad → I need GPT-5" │ ├─ Reality: Problem is harness (not model) │ ├─ Fix: Improve prompt + tools + feedback loops │ ├─ Result: 4x improvement (free) │ └─ Then: Maybe upgrade model (if still needed) ├─ Mistake 2: "More prompts = better agent" │ ├─ Reality: Better structured prompts = better agent │ ├─ Fix: Use Chain-of-Thought, examples, constraints │ ├─ Result: 3x improvement (same token count) │ └─ Then: Optimize further (if needed) ├─ Mistake 3: "Agent runs code directly (no sandbox)" │ ├─ Reality: Risky (crashes, hangs, exploits) │ ├─ Fix: Use sandboxed execution (isolated container) │ ├─ Result: 10x reliability improvement (no crashes) │ └─ Then: Sleep better (know system is safe) ├─ Mistake 4: "Agent makes mistake → No feedback" │ ├─ Reality: Agent can't learn (stuck) │ ├─ Fix: Show error, let agent retry │ ├─ Result: 5x improvement (self-correcting) │ └─ Then: Agent improves over time ├─ Mistake 5: "Agent has 1 tool (execute_code only)" │ ├─ Reality: Agent can't solve complex problems (limited) │ ├─ Fix: Add more tools (read/write, test, git, etc) │ ├─ Result: 3x more problems solved (more tools) │ └─ Then: Agent is actually useful (not toy) └─ Mistake 6: "No monitoring of agent performance" ├─ Reality: Can't improve what you don't measure ├─ Fix: Track success rate, error types, execution time ├─ Result: Know where to improve (data-driven) └─ Then: Iterate quickly (optimize what matters).

---

Como otimizar agent harness (estratégias práticas)

1. Prompt Engineering (melhorar perguntas)

BAD PROMPT: "Write code to handle user authentication"

GOOD PROMPT: "You are authentication engineer writing production code.

Task: Implement user login with email/password.

Requirements: ├─ Framework: FastAPI (Python) ├─ Database: PostgreSQL ├─ Password: Hash with bcrypt (min 12 rounds) ├─ Session: JWT token (24-hour expiry) ├─ Validation: Email format, min 8-char password ├─ Error handling: Invalid credentials → log attempt ├─ Security: Rate limit 5 attempts per minute └─ Testing: Include pytest test cases (success + failure)

Example input: POST /auth/login {"email": "user@example.com", "password": "secret123"}

Example output: {"token": "eyJ0eXAi...", "expires_in": 86400}

Edge cases to handle: ├─ Email not found ├─ Password incorrect ├─ Rate limit exceeded └─ Database error

Write code now:"

IMPACT: ├─ Bad prompt: 20% success rate ├─ Good prompt: 80% success rate (same model) └─ Improvement: 4x better (just better structure).

2. Tool Integration (mais opções pra agent)

BAD HARNESS (1 tool): ├─ Agent has: execute_code (that's it) ├─ Problem: Agent can't read files, can't test, can't commit ├─ Result: Limited (can't do complete workflow)

GOOD HARNESS (10+ tools): ├─ read_file(path) → read any file ├─ write_file(path, content) → write file ├─ execute_code(code, timeout=30) → run code safely ├─ test_code(path) → run tests, show results ├─ lint_code(path) → check code quality ├─ git_commit(message) → commit changes ├─ github_pr(branch, title, description) → create PR ├─ slack_notify(channel, message) → send notification ├─ database_query(sql) → query database ├─ api_call(url, method, headers, body) → call API ├─ ask_human(question) → when stuck, ask human └─ Result: Powerful (can do complete automation).

IMPACT: ├─ With 1 tool: Agent can solve 10% of problems ├─ With 10 tools: Agent can solve 70% of problems (7x better) └─ Cost: Free (just better architecture).

3. Feedback Loops (agent aprende de erros)

BAD HARNESS (no feedback): ├─ Agent writes code → code crashes ├─ You see error (agent doesn't) ├─ Agent repeats mistake next time

GOOD HARNESS (with feedback): ├─ Agent writes code ├─ Execute in sandbox ├─ Code crashes → capture error ├─ Show error to agent: "SyntaxError: invalid syntax line 5" ├─ Agent analyzes error ├─ Agent fixes code ├─ Retry (up to 3 times) ├─ If success → continue ├─ If fails 3x → ask human (escalate) ├─ Result: Self-correcting (agent improves).

IMPACT: ├─ No feedback: First-try success 30% ├─ With feedback: First-try success 80% (2.6x better) ├─ With retry: Final success 95% (3.2x better) └─ Cost: ~5 minutes coding (worth it).

4. Execution Environment (sandbox para segurança)

BAD HARNESS (no sandbox): ├─ Code runs in main process ├─ Code crash → system crash ├─ Code infinite loop → system hang ├─ Code malicious → full access ├─ Result: Risky (not production-ready).

GOOD HARNESS (with sandbox): ├─ Code runs in isolated container ├─ Container crash → system safe ├─ Timeout 30 seconds → kill process ├─ Read-only filesystem → can't modify data ├─ Memory limit 500MB → prevent abuse ├─ CPU limit 2 cores → fair usage ├─ Network: Whitelist only safe endpoints ├─ Result: Safe (production-ready).

IMPACT: ├─ Reliability: 10x better (no crashes) ├─ Security: 100x better (isolated) ├─ Cost: ~$100/month (sandboxing service) OR free (self-hosted).

5. Monitoring & Measurement (saber o que melhorar)

METRICS TO TRACK: ├─ Success rate (% of tasks completed successfully) ├─ First-try success (% succeeding on first attempt) ├─ Retry rate (how many retries needed) ├─ Error types (what fails most: syntax, logic, timeout?) ├─ Execution time (how long per task) ├─ Token usage (cost per task) ├─ User satisfaction (do humans like the output?) └─ Quality score (code style, performance, maintainability).

EXAMPLE DASHBOARD: ├─ Last 7 days: │ ├─ Success rate: 65% (goal: 85%) │ ├─ First-try: 40% (goal: 70%) │ ├─ Most common error: "IndentationError" (20% of failures) │ ├─ Avg execution time: 45 seconds │ ├─ Avg cost: $0.12 per task │ ├─ User satisfaction: 7.2/10 (goal: 8.5) │ └─ Action: Fix indentation handling in prompt → retry

RESULT: ├─ Data-driven improvements (know what to optimize) ├─ Faster iteration (test changes, measure impact) ├─ Continuous improvement (compound gains over time) └─ Cost reduction (fewer retries = lower cost).

---

Roadmap: Como implementar harness otimizado

Week 1: Audit (conhecer baseline)

• Medir success rate atual (qual % sucede?)
• Identificar padrões de erro (que tipo de erro mais common?)
• Coletar feedback de usuários (satisfeitos?)
• Documentar atual harness (prompt, tools, feedback loops)

Week 2-3: Quick wins (melhorias rápidas)

• Melhorar prompt (Chain-of-Thought, exemplos)
• Adicionar 3-5 tools essenciais (read_file, write_file, test)
• Implementar feedback loops básicas (show error, retry)
• Setup monitoring (track success rate)

Week 4-6: Scale up (melhorias maiores)

• Adicionar mais tools (git, api_call, ask_human)
• Implementar sandbox (segurança)
• Optimize prompts (baseado em error patterns)
• A/B test diferentes harness designs

Ongoing: Continuous improvement

• Monitorar metrics (success rate, user satisfaction)
• Iterar em prompts (baseado em erros mais comuns)
• Adicionar tools conforme necessário
• Treinar team (como usar agent melhor)

---

Conclusão: Harness = nova competência crítica

O que pesquisa revelou:

1. Harness design = 80% do sucesso (não o modelo)

   • Implicação: "Você está investindo no lugar errado (modelo, quando deveria ser harness)."
   • Action: "Stop upgrading modelo. Start optimizing harness."

2. Boas práticas existem (Chain-of-Thought, feedback loops, sandboxing)

   • Implicação: "Não é mágico. É engenharia (estrutura, design)."
   • Action: "Apply best practices now (not waiting for better model)."

3. Custo-benefício é excelente (melhorias grátis ou baratas)

   • Implicação: "Melhorar harness = 4x resultado, zero custo."
   • Action: "Nenhuma desculpa para não fazer agora."

4. Mensurável (track success, optimize data-driven)

   • Implicação: "Sabe exatamente o que melhorar."
   • Action: "Iterar rápido (teste, mede, melhora)."

5. Competência defensiva (competitors that know this ganham)

   • Implicação: "Se você não faz, competitors que fazem vão te vencer."
   • Action: "Aprenda agora (ou fique atrás)."

Sua decisão hoje:

• Continuar gastando em upgrades de modelo (perdendo dinheiro)
• Começar a otimizar harness (ganhar 4x com zero custo)

Recomendação: AUDIT seu agente harness NOW. Identifique gaps. Implemente Chain-of-Thought. Adicione tools. Setup feedback loops. Mensure tudo. THEN (só depois): Consider model upgrade (if still needed).

Na OpenClaw:

Ajudamos SaaS builders otimizar agent harness:

• Harness audit: Qual é seu score atual? (assessment)
• Prompt optimization: Como estruturar melhor? (engineering)
• Tool integration: Quais tools adicionar? (architecture)
• Feedback loops: Como agent aprende? (learning)
• Monitoring setup: Como medir progress? (measurement)
• Benchmarking: Você está competitivo? (comparison)
• Scaling: Como ter 100s de agents? (operations)
• Cost optimization: Como reduzir spend? (efficiency)

Your agent's ceiling is set by harness, not by model. Optimize harness first. Upgrade model last.

Agent Harness Optimization | Prompt Engineering | Architecture →