Seu agente de código gera bugs? Culpa do harness, não do LLM

Seu agente de código gera bugs? Culpa do harness, não do LLM

Você é founder de SaaS.

Seu agente de código:

• Gera código automaticamente (API endpoints, data models, utils)
• Seu modelo: Claude, GPT, Mistral (you picked best LLM)
• Your assumption: "Melhor modelo = melhor código. Ponto."
• Reality: Qualidade do código depende mais do harness (framework) que do modelo
• Evidence: HarnessTax research: Harness quality = 30-40% do resultado final
• Implication: "Seu LLM é excelente. Mas seu harness (framework) é medíocre."
• Result: "Agent gera código que parece OK, mas tem bugs invisíveis"
• Customer experience: "Código gerado quebra em produção. Culpa sua (não do LLM)."
• Your discovery: "Nunca pensei no harness. Assumo modelo é tudo."
• Real answer: "Modelo é 60%. Harness é 40%. Framework importa MUITO."

Seu problema AGORA:

• HarnessTax (research framework) publicou: "Harness matters (a LOT)"
• Finding: "Agent output quality está 30-40% dependente do harness design"
• Implication: "Trocar modelo de GPT-4 → Claude não resolve problema se harness é ruim"
• Your realization: "Meu agente gera código ruim. Culpa do harness? (não do modelo)"
• Bigger problem: "Competitor com melhor harness gera código 40% melhor que meu (mesmo modelo)"
• Timeline: "Harness improvement = architectural work (4-8 semanas)"
• Your opportunity: "Implementar better harness (antes de competitor fazer)."

O que HarnessTax está sinalizando:

"Agent harness (framework, prompt structure, context management) determines 30-40% of code quality. Most teams focus on model selection (GPT vs Claude) but ignore harness. If your harness is poorly designed, your agent will generate mediocre code even if you're using GPT-4. Conversely, good harness + smaller model can outperform bad harness + larger model. Framework matters."

---

O problema: Harness determina qualidade do agente (não o modelo)

Como harness afeta code quality (invisível até exploração)

=== SCENARIO: Seu SaaS de automação de código ===

Your coding agent setup (typical): ├─ Model: Claude 3.5 Sonnet (or GPT-4) ├─ Task: "Generate API endpoint" ├─ Prompt: "Write a function that does X" ├─ Context: "Here's the existing codebase" ├─ Output: Code (looks good, but quality depends on harness) └─ Your assumption: "Model quality = output quality. Done."

=== WHAT IS A HARNESS? ===

Harness = Framework for agent reasoning ├─ Includes: │ ├─ Prompt structure (how you frame the task) │ ├─ Context management (what info you give agent) │ ├─ Output formatting (how agent returns code) │ ├─ Constraint enforcement (rules agent must follow) │ ├─ Error handling (what if agent fails?) │ ├─ Validation (check output before returning) │ └─ Feedback loop (learn from mistakes) ├─ Harness is NOT the model (Claude, GPT) ├─ Harness is NOT the prompt (instruction) ├─ Harness IS the architecture that connects them └─ Result: Two teams, same model, different harness → Different quality

=== EXAMPLE: GPT-4 with bad harness vs Claude with good harness ===

Scenario A: GPT-4 + bad harness ├─ Model: GPT-4 (best available) ├─ Prompt: Generic ("Generate a function") ├─ Context: Incomplete (missing codebase style guide) ├─ Validation: None (output not checked) ├─ Error handling: None (fails silently) ├─ Result: │ ├─ Generated function has logic bug (index off-by-one) │ ├─ Style doesn't match codebase (inconsistent) │ ├─ No error handling (crashes on edge case) │ ├─ Bug discovered: 3 days later (in production) │ └─ Quality score: 62% (looks OK, but has bugs) ├─ Cost to fix: Manual code review + debugging (4 hours) └─ Outcome: FAILURE (produced buggy code)

Scenario B: Claude 3.5 Sonnet + good harness ├─ Model: Claude 3.5 Sonnet (smaller than GPT-4) ├─ Prompt: Structured ("Generate function with X constraints") ├─ Context: Complete (codebase style guide + type definitions) ├─ Validation: Automatic (check syntax, type hints, edge cases) ├─ Error handling: Comprehensive (handles nulls, errors, edge cases) ├─ Result: │ ├─ Generated function has no logic bugs │ ├─ Style matches codebase (consistent) │ ├─ Proper error handling (fails gracefully) │ ├─ Passes validation: Immediately │ └─ Quality score: 89% (production-ready) ├─ Cost to fix: None (code is ready) └─ Outcome: SUCCESS (produced good code)

=== THE GAP ===

Team A (GPT-4 + bad harness): ├─ Model quality: 95/100 ├─ Harness quality: 50/100 ├─ Output quality: (95 * 0.6) + (50 * 0.4) = 57 + 20 = 77/100 └─ Result: Mediocre output (despite best model)

Team B (Claude 3.5 + good harness): ├─ Model quality: 85/100 ├─ Harness quality: 90/100 ├─ Output quality: (85 * 0.6) + (90 * 0.4) = 51 + 36 = 87/100 └─ Result: Better output (with smaller model, better harness)

=== YOUR SITUATION ===

Your current setup (probably): ├─ Model: Claude or GPT-4 (good) ├─ Harness: Generic prompt + basic context (bad) ├─ Estimated output quality: 65-75/100 ├─ Problem: Generate code, but lots of bugs ├─ Customer complaint: "Your AI generates code that doesn't work" ├─ Your response: "But we're using best model!" ├─ Reality: "Your harness is holding you back (not the model)" └─ Solution: Improve harness architecture

---

What makes a good harness (vs bad harness)

5 components that determine harness quality

=== COMPONENT 1: Prompt Structure ===

Bad harness prompt: ├─ Instruction: "Write a function that does X" ├─ Context: None ├─ Constraints: None ├─ Expected format: None ├─ Error handling: Not mentioned └─ Result: Agent has freedom to interpret (generates inconsistent code)

Good harness prompt: ├─ Instruction: "You are a senior Python engineer" ├─ Context: "Our codebase uses [style guide]. Here are examples." ├─ Constraints: │ ├─ "Use type hints for all functions" │ ├─ "Follow snake_case naming (not camelCase)" │ ├─ "Include docstrings (Google style)" │ ├─ "Handle all error cases (no silent failures)" │ ├─ "Write 2-3 test cases (edge cases)" │ └─ "Max function length: 50 lines" ├─ Expected format: "python\n[code]\n" ├─ Error handling: "If implementation is unclear, ask clarifying questions" └─ Result: Agent has clear constraints (generates consistent, high-quality code)

Quality impact: 15-25% improvement just from better prompt structure

=== COMPONENT 2: Context Management ===

Bad harness context: ├─ What you give agent: "Generate function for user authentication" ├─ Context provided: Nothing else ├─ Missing: Codebase style, existing patterns, dependencies, edge cases ├─ Result: Agent guesses (generates code that doesn't fit codebase) ├─ Example: Agent generates OOP code, but codebase is functional └─ Impact: Code doesn't integrate (requires manual rewrite)

Good harness context: ├─ What you give agent: "Generate function for user authentication" ├─ Context provided: │ ├─ Codebase style guide (what you value) │ ├─ Authentication patterns (existing code examples) │ ├─ Type definitions (what types to use) │ ├─ Dependencies available (what libraries are imported) │ ├─ Error handling patterns (how errors are handled) │ ├─ Edge cases to consider (e.g., SQL injection, XSS) │ ├─ Performance constraints (latency, memory) │ └─ Security requirements (what's required) ├─ Result: Agent understands context (generates code that fits perfectly) ├─ Example: Agent generates functional code, matches existing patterns └─ Impact: Code is production-ready (minimal changes needed)

Quality impact: 20-30% improvement just from better context

=== COMPONENT 3: Output Validation ===

Bad harness validation: ├─ What you do: Take agent output, use it as-is ├─ Validation: None ├─ Check syntax? No ├─ Check types? No ├─ Check edge cases? No ├─ Run tests? No ├─ Result: Buggy code reaches customer (or deployed, breaks in production) └─ Cost: Customer complaint + debug + fix (4-8 hours per bug)

Good harness validation: ├─ What you do: Take agent output, validate before returning ├─ Validation steps: │ ├─ Syntax check (parse code, catch syntax errors) │ ├─ Type check (validate type hints are correct) │ ├─ Style check (ensure consistent with codebase) │ ├─ Test execution (run generated test cases) │ ├─ Edge case check (look for obvious bugs) │ ├─ Security check (look for common vulnerabilities) │ └─ Lint (check for code quality issues) ├─ If validation fails: │ ├─ Option 1: Return to agent for fixing ("Your code has X, fix it") │ ├─ Option 2: Return error to user ("Cannot generate, here's why") │ └─ Option 3: Request clarification ("Can you clarify X?") ├─ Result: Only good code reaches customer └─ Cost: Prevention (5 minutes validation) vs fix (4-8 hours later)

Quality impact: 25-35% improvement just from validation

=== COMPONENT 4: Error Handling & Feedback ===

Bad harness error handling: ├─ Agent fails: "I can't generate code for this task" ├─ User sees: Generic error (no info why) ├─ Retry: User has to restart ├─ Learning: Agent doesn't learn (same mistake next time) └─ Result: User frustration + low success rate

Good harness error handling: ├─ Agent fails: "I can't generate code for this task" ├─ What you do: │ ├─ Capture failure reason ("Missing X information", "Too complex", etc) │ ├─ Provide feedback to agent ("Try with more details") │ ├─ Guide user ("Please provide X", "Simplify task to Y", etc) │ ├─ Log failure ("Agent failed on task X. Reason: Y") │ ├─ Learn from failure ("Next time, ask for more context") │ └─ Retry with improvements (Automatically or with user help) ├─ Result: User understands what happened, knows how to fix it ├─ Success rate: Improves over time (agent learns) └─ Cost: Prevention of repeat failures

Quality impact: 10-15% improvement from better error handling

=== COMPONENT 5: Constraint Enforcement ===

Bad harness constraints: ├─ Constraints: None (or vague in prompt) ├─ Example: "Write efficient code" (what's efficient? depends on context) ├─ Result: Agent interprets freely (generates inefficient code) ├─ Example output: │ ├─ O(n²) when O(n) is better │ ├─ 500 lines when 50 lines is cleaner │ ├─ Unnecessary dependencies (bloats code) │ └─ No error handling (doesn't follow constraint) └─ Impact: Technical debt

Good harness constraints: ├─ Constraints: Hard boundaries ├─ Example: │ ├─ "Max function length: 50 lines (HARD LIMIT)" │ ├─ "Time complexity: O(n) or better (REQUIRED)" │ ├─ "Use only: [list of approved libraries] (LOCKED)" │ ├─ "Must include error handling (REQUIRED)" │ ├─ "Must have test cases (REQUIRED)" │ └─ "Must pass linting (REQUIRED)" ├─ Enforcement: Validated before returning to user ├─ If violated: Agent tries again (or requests help) └─ Impact: Consistent, predictable output quality

Quality impact: 15-20% improvement from constraint enforcement

---

How to improve your harness (practical steps)

4-step framework to upgrade harness quality

Step 1: Audit current harness (what are you using now?)

☐ Question 1: What's your current harness? ├─ Do you have a system prompt? (or just send task?) ├─ What context do you provide? (codebase info, style guide, examples?) ├─ Do you validate output? (syntax check, type check?) ├─ Do you handle errors? (capture why agent fails?) ├─ Do you enforce constraints? (hard limits on code?) └─ Score: Count how many YES

☐ Question 2: Score your harness ├─ If 0-1 YES: Harness quality = ~40% (Very bad) ├─ If 1-2 YES: Harness quality = ~55% (Bad) ├─ If 2-3 YES: Harness quality = ~70% (OK) ├─ If 3-4 YES: Harness quality = ~80% (Good) ├─ If 5 YES: Harness quality = ~90% (Excellent) └─ Your score = Current output quality

☐ Question 3: What's your output quality? ├─ Track: % of generated code that is production-ready (no fixes needed) ├─ Current: If harness score is 40-70%, output quality is probably 60-75% ├─ Target: Output quality should be >90% (production-ready) ├─ Gap: If current <90%, harness needs improvement └─ Priority: Fix harness

☐ My checklist (step 1): ├─ [ ] Audited current harness (5 components) ├─ [ ] Scored each component (0-5) ├─ [ ] Estimated output quality % (60-90%?) ├─ [ ] Identified worst components (focus areas) └─ [ ] Ready for improvement (next step)

Step 2: Design better harness (component by component)

☐ Improve Component 1: System Prompt ├─ Current prompt: [your generic prompt] ├─ New prompt (add): │ ├─ Context: "You are a senior engineer" │ ├─ Task: "Generate code that: [specific requirements]" │ ├─ Style: "Follow [codebase style]: [examples]" │ ├─ Constraints: "[Hard limits on code]" │ ├─ Testing: "Include [N] test cases for edge cases" │ └─ Error handling: "Handle all error cases: [examples]" ├─ Expected improvement: +15-25% output quality └─ Effort: 2-4 hours

☐ Improve Component 2: Context ├─ Current context: [none or minimal] ├─ New context (add): │ ├─ Codebase style guide (link to doc or inline) │ ├─ Code examples (3-5 existing functions that match style) │ ├─ Type definitions (what types to use) │ ├─ Architecture patterns (how to structure code) │ ├─ Dependencies available (what libraries are allowed) │ ├─ Edge cases to consider (common mistakes) │ └─ Performance/security requirements (constraints) ├─ Expected improvement: +20-30% output quality └─ Effort: 4-8 hours (compile good context)

☐ Improve Component 3: Validation ├─ Current validation: [none] ├─ New validation (add): │ ├─ Syntax check (parse code, catch errors) │ ├─ Type validation (check type hints) │ ├─ Style check (lint against codebase style) │ ├─ Test execution (run generated tests) │ ├─ Edge case check (look for obvious bugs) │ └─ Security scan (look for vulnerabilities) ├─ On failure: Return to agent for fixing ("Your code has X. Fix it.") ├─ Expected improvement: +25-35% output quality └─ Effort: 8-16 hours (implement validators)

☐ Improve Component 4: Error Handling ├─ Current error handling: [generic error message] ├─ New error handling (add): │ ├─ Capture why agent failed (categorize error) │ ├─ Provide feedback to agent (specific, actionable) │ ├─ Guide user (what to do next) │ ├─ Log for learning (track patterns) │ ├─ Retry with improvements (auto or manual) │ └─ Track success rate (monitor improvement) ├─ Expected improvement: +10-15% output quality └─ Effort: 4-8 hours (implement error handling)

☐ Improve Component 5: Constraints ├─ Current constraints: [vague or missing] ├─ New constraints (make hard limits): │ ├─ Max function length: [N] lines (HARD LIMIT) │ ├─ Time complexity: O(?) or better (REQUIRED) │ ├─ Approved libraries only: [list] (LOCKED) │ ├─ Must include error handling (REQUIRED) │ ├─ Must have tests (REQUIRED) │ ├─ Must pass linting (REQUIRED) │ └─ [Your domain-specific constraints] ├─ Enforcement: Validate before returning ├─ Expected improvement: +15-20% output quality └─ Effort: 4-6 hours (define + enforce)

☐ My checklist (step 2): ├─ [ ] Improved system prompt (added context + constraints) ├─ [ ] Collected better context (style guide + examples) ├─ [ ] Designed validation steps (syntax + type + style + tests) ├─ [ ] Designed error handling (feedback + retry logic) ├─ [ ] Defined hard constraints (limits + requirements) └─ [ ] Ready for implementation (next step)

Step 3: Implement new harness (code it)

☐ Implementation: ├─ Update system prompt (new constraints + context) ├─ Add context injection (pass codebase info to agent) ├─ Implement validators (syntax, type, style, security) ├─ Add error handling (capture + categorize failures) ├─ Implement retry logic (agent tries again with feedback) ├─ Add monitoring (track quality metrics) └─ Deploy to staging (test before production)

☐ Testing (before production): ├─ Test 1: Same tasks, new harness. Compare output quality. │ ├─ Metric: % of code that is production-ready │ ├─ Baseline (old harness): 65% │ ├─ Target (new harness): 85%+ │ ├─ If not met: Debug and iterate │ └─ Acceptance: 85%+ ├─ Test 2: Edge cases. Can new harness handle them? │ ├─ Test complex tasks (new harness should give feedback, not fail) │ ├─ Test error scenarios (does error handling work?) │ ├─ Test constraints (does validation enforce limits?) │ └─ Acceptance: 95%+ of cases handled correctly ├─ Test 3: Performance. Does validation add latency? │ ├─ Baseline (old harness): 2 seconds to generate code │ ├─ Expected (new harness): 3-4 seconds (validation overhead) │ ├─ If >5 seconds: Optimize validators │ └─ Acceptance: <5 seconds total └─ Result: Ready for production

☐ My checklist (step 3): ├─ [ ] Implemented all 5 harness components ├─ [ ] Deployed to staging environment ├─ [ ] Tested output quality (85%+ production-ready) ├─ [ ] Tested edge cases (95%+ handled) ├─ [ ] Tested performance (<5 seconds) └─ [ ] Ready for production deployment (next step)

Step 4: Monitor & iterate (continuous improvement)

☐ Monitoring (after deployment): ├─ Daily: Track output quality % (production-ready code) │ ├─ Metric: "% of generated code that required zero fixes" │ ├─ Baseline (old harness): 65% │ ├─ Expected (new harness): 85%+ │ ├─ If <85%: Debug harness, identify weakness │ └─ Action: Iterate, improve ├─ Weekly: Track error patterns │ ├─ Metric: "What % of failures are due to [each component]?" │ ├─ Example: 30% context-related, 20% constraint-related, 50% unknown │ ├─ Focus improvement on highest % (context improvement priority) │ └─ Action: Fix weakest component first ├─ Monthly: Full harness audit │ ├─ Review: Are all 5 components still effective? │ ├─ Gather feedback: From users + team │ ├─ Iterate: Improve based on findings │ └─ Target: Maintain 85%+ quality (or improve) └─ Ongoing: Version your harness ├─ Harness v1.0 (baseline): 65% quality ├─ Harness v1.1 (prompt improvement): 75% quality ├─ Harness v1.2 (context improvement): 82% quality ├─ Harness v1.3 (validation + error handling): 88% quality ├─ Harness v1.4 (constraint enforcement): 91% quality └─ Target: Keep iterating until 95%+

☐ My checklist (step 4): ├─ [ ] Deployed new harness to production ├─ [ ] Monitoring output quality daily (target 85%+) ├─ [ ] Tracking error patterns weekly ├─ [ ] Iterating based on data (weekly improvements) ├─ [ ] Versioning harness (v1.0, v1.1, etc) └─ [ ] On track to reach 95%+ quality (3-6 months)

---

Conclusão: Harness quality determines agent output quality

O que HarnessTax está sinalizando:

1. Harness matters MORE than you think (30-40% of quality)

   • You think: "Model quality = output quality. GPT-4 is best. Done."
   • Reality: "Harness (framework) is 40% of quality. Model is 60%."
   • Implication: "Better harness + smaller model > worse harness + bigger model."

2. Most teams ignore harness (focus on model only)

   • Mistake: Pick best model, use generic prompt, ignore framework
   • Result: Mediocre output (despite best model)
   • Winner: Competitor with better harness (using smaller model)
   • Competitive gap: 30-40% output quality difference (from harness alone)

3. Harness improvement is quick (4-8 weeks)

   • Design better prompt: 2-4 hours
   • Collect good context: 4-8 hours
   • Implement validation: 8-16 hours
   • Add error handling: 4-8 hours
   • Enforce constraints: 4-6 hours
   • Total: 25-50 engineering hours (vs continuous model research)
   • ROI: 20-30% output quality improvement (quick win)

4. Output quality is now competitive advantage (not model brand)

   • Companies focus on: "We use Claude vs GPT-4" (irrelevant)
   • Reality: Output quality depends on harness (invisible to customers)
   • Competitive edge: Better harness = better code = happier customers
   • Timeline: Implement harness improvements NOW (before competitor does)

5. Harness is permanent (not model version-locked)

   • Model updates: Come from vendor (you don't control)
   • Harness: Your IP (you control, iterates over time)
   • Result: Good harness = sustainable advantage
   • Implication: Invest in harness (better ROI than model research)

Seu checklist (faça esta semana):

• Você sabe qual é seu harness? (prompt + context + validation + error handling + constraints)
• Você mede output quality %? (production-ready code sem fixes)
• Seu output quality é 85%+? (ou <75%, meaning harness is weak)
• Você valida código antes de entregar? (ou confía no modelo)
• Você tem contexto rico (codebase style + examples)? (ou genérico)

Se respondeu NÃO a mais de 2, seu harness está FRACO.

Na OpenClaw:

Ajudamos SaaS builders a implementar better harnesses pra agents de código:

• Harness audit: Qual é seu score atual? (40-90%?)
• Best practices: Como estruturar system prompt + context? (framework)
• Validation layer: Como implementar production-ready checks? (code)
• Error handling: Como capturar + aproveitar falhas? (feedback loops)
• Constraint enforcement: Como garantir code quality standards? (hard limits)
• Continuous improvement: Como iterar e melhorar over time? (versioning)

Você pode continuar com harness genérico (e competitor vai ganhar com harness melhor).

Ou você pode implementar better harness AGORA (em 4-8 semanas) e estar 30-40% à frente.

Better Harness for Coding Agents | Code Quality Framework | Agent Output Optimization →