Seu agente não foi benchmarked? Está quebrado.

Seu agente não foi benchmarked? Está quebrado.

Você é founder de SaaS.

Seu agente de IA:

• "Works on my machine" (demo funciona)
• 50 linhas de código (simples, elegante)
• Your assumption: "If it works locally, it's production-ready."
• Reality: "Local ≠ Production (very different)."
• Your blind spots: ├─ Latency: Demo: 200ms. Production: 5000ms (25x slower). ├─ Error rate: Demo: 0%. Production: 15% (users hit edge cases). ├─ Throughput: Demo: 1 user. Production: 100 users (agent locks up). ├─ Edge cases: Demo: Happy path only. Production: Chaos. └─ Result: "Agent breaks in production (you didn't test)."

Terminal-Bench just made benchmarking easy:

"Run Harbor evals on Vercel Sandbox. Each test executes in isolated microVM. Parallelize beyond local machine. Finally: easy agent benchmarking."

Translation to your SaaS:

• Old way: Test agent locally (limited, not realistic)
• New way: Benchmark on Vercel Sandbox (production-like, parallel, easy)
• Discovery: "Agent is 10x slower in production than I thought."
• Action: "Need to optimize before shipping."
• Result: "Production-ready agent (tested, not hoped)."

---

O Problema: Agent benchmarking é tabela redonda (mas ninguém faz)

Por que agentes quebram em produção

=== THE TESTING GAP ===

Your agent testing today: ├─ Manual testing │ ├─ You write prompt: "Hello" │ ├─ Agent responds: "Hi there!" │ ├─ You think: "Works. Ready for production." │ └─ Reality: You tested 1 input (happy path) ├─ Local testing │ ├─ Your machine: 16GB RAM, fast CPU │ ├─ Production: Shared VM, network latency │ ├─ Your test: 1 concurrent user │ ├─ Production: 100 concurrent users │ └─ Result: "Completely different performance" ├─ No edge case testing │ ├─ What if user provides malformed input? │ ├─ What if agent times out? │ ├─ What if external API is down? │ ├─ What if customer spams agent (DOS)? │ └─ Result: "Agent breaks on first edge case" └─ No benchmark comparison ├─ Is agent faster/slower than competitor? ├─ Is agent reliable as SaaS standard? ├─ How does agent scale to 1K users? └─ Result: "You have no baseline. Can't measure progress."

=== WHY THIS IS DANGEROUS ===

1. Silent failures ├─ Agent doesn't crash (it just gets slow) ├─ Customers see: "Timeouts" (blame you, not agent) ├─ You see: "Logs look fine" (don't understand why slow) └─ Result: "Customer loses trust (can't explain delay)."

2. Scale blindness ├─ Agent works for 1 customer ├─ Agent works for 10 customers ├─ Agent breaks at 100 customers (nobody knew limit) ├─ You scramble (customer already angry) └─ Result: "Churn (customer expected reliability)."

3. Competition disadvantage ├─ Competitor benchmarks agent (knows performance) ├─ You don't benchmark (flying blind) ├─ Customer asks: "How fast is your agent?" ├─ You answer: "Uh... fast?" (sounds unsure) ├─ Customer picks competitor (they had numbers) └─ Result: "Lost deal (couldn't back up claims)."

4. Technical debt ├─ You ship untested agent (seemed fine) ├─ 6 months later: Agent is slow (accumulated tech debt) ├─ Customers complaining (performance degraded) ├─ You investigate: "Why is it slow?" (no baseline to compare) ├─ Fix takes months (you don't know what to optimize) └─ Result: "Missed revenue (fixing instead of building)."

5. Liability ├─ SLA: "Agent responds in <5 seconds" ├─ Reality: Agent sometimes takes 30 seconds ├─ Customer sues: "Breach of SLA" ├─ Your defense: "We tested it locally" (weak) └─ Result: "Legal issue (untested claims)."

=== THE BENCHMARKING SOLUTION ===

Terminal-Bench solves this: ├─ What: Standardized agent benchmarks ├─ Benchmarks: │ ├─ SWE-bench: Software engineering tasks (coding agents) │ ├─ tau3-bench: Multi-step reasoning (agent planning) │ ├─ OSWorld: OS interaction (system automation agents) │ └─ Terminal-Bench: Shell commands (CLI agents) ├─ Before: Need to setup infrastructure (hard) ├─ After: Run on Vercel Sandbox (easy, no setup) ├─ Parallelization: Run 100 trials in parallel (fast) └─ Result: "Easy agent benchmarking (measure before ship)."

---

A Verdade Incômoda: Você não sabe se seu agente é production-ready

Como agent benchmarking revela problemas invisíveis

=== BEFORE BENCHMARKING ===

Your agent: ├─ Manual test: "Works on my machine" ├─ Demo: Responds in 200ms ├─ Assumption: "Production-ready" ├─ You ship it ├─ Customer uses it ├─ Customer sees: 5000ms response time (25x slower) ├─ Customer complains: "Your agent is slow" ├─ You investigate: "I don't know why. Looked fast in testing." ├─ Customer loses trust: "Your SaaS is unreliable" └─ Result: "Churn (untested agent broke production)."

=== AFTER BENCHMARKING (TERMINAL-BENCH) ===

Your agent: ├─ Benchmark: Terminal-Bench on Vercel Sandbox ├─ Results: │ ├─ Latency: 50th percentile = 200ms (median) │ ├─ Latency: 95th percentile = 5000ms (tail) │ ├─ Latency: 99th percentile = 10000ms (extreme tail) │ ├─ Success rate: 95% (5% errors) │ ├─ Throughput: 100 concurrent requests OK │ ├─ Throughput: 1000 concurrent requests → timeouts │ └─ Breakdown: Which steps are slow? ├─ Discovery: "Agent is slow in production (95th percentile is bad)." ├─ Action: "Optimize before shipping." ├─ Improvement: "Optimize slow step. Now 95th = 800ms (10x better)." ├─ Verification: "Re-benchmark. Confirmed improvement." ├─ Customer experience: "Agent is fast (tested, not guessed)." └─ Result: "Production-ready agent (verified, not assumed)."

=== WHAT TERMINAL-BENCH REVEALS ===

1. Latency distribution ├─ Median latency: 200ms (typical) ├─ 95th percentile: 5000ms (slow users experience this) ├─ 99th percentile: 10000ms (very slow users) └─ Insight: "Median is fast but tail is slow. Need to optimize tail."

2. Success rate ├─ Success: 95% (5% of requests fail) ├─ Failures: Edge cases (malformed input, timeout, etc) ├─ Breakdown: Which inputs cause failures? └─ Insight: "Need to handle edge cases (don't ignore 5% failure)."

3. Throughput limits ├─ 100 concurrent users: OK ├─ 1000 concurrent users: Timeouts (agent saturated) ├─ Limit: Agent can't serve >500 concurrent users └─ Insight: "You can't scale beyond 500 concurrent users. Need to optimize or use multiple instances."

4. Bottleneck analysis ├─ Step 1 (parse input): 10ms ├─ Step 2 (call model): 4000ms (slow!) ├─ Step 3 (format output): 100ms ├─ Root cause: Model inference is bottleneck └─ Insight: "Optimize model speed (not parsing or formatting)."

5. Comparison to baseline ├─ Your agent: 200ms median ├─ Competitor agent: 100ms median ├─ Industry standard: 150ms median ├─ Your agent: 33% slower than competitor └─ Insight: "Need to catch up (or your agent loses customers)."

=== THE BENCHMARKING WORKFLOW ===

Step 1: Baseline (before optimization) ├─ [ ] Run Terminal-Bench on current agent ├─ [ ] Measure: Latency, success rate, throughput ├─ [ ] Document: Baseline metrics └─ Output: "You know current performance."

Step 2: Identify bottlenecks ├─ [ ] Analyze: Which steps are slow? ├─ [ ] Profile: Where is time spent? ├─ [ ] Breakdown: Model inference? API calls? Data processing? └─ Output: "You know what to optimize."

Step 3: Optimize ├─ [ ] Hypothesis: "If I optimize X, performance improves Y%" ├─ [ ] Change: Implement optimization ├─ [ ] Test: Local verification (quick check) ├─ [ ] Benchmark: Terminal-Bench (measure improvement) └─ Output: "You know if optimization worked."

Step 4: Iterate ├─ [ ] Repeat step 2-3 (identify new bottleneck) ├─ [ ] Track: Performance improvement over time ├─ [ ] Document: What optimizations helped most └─ Output: "Incremental improvements (measure each)."

Step 5: Production verification ├─ [ ] Monitor: Production performance (compare to benchmark) ├─ [ ] Alert: If production diverges from benchmark ├─ [ ] Re-benchmark: Quarterly (stay ahead of degradation) └─ Output: "Production stays performant (benchmarking is continuous)."

=== BENCHMARKING BEST PRACTICES ===

1. Measure meaningful metrics ├─ Latency distribution (median, 95th, 99th percentile) ├─ Success rate (not just binary pass/fail) ├─ Throughput (how many concurrent users?) ├─ Cost (tokens/inference = money) └─ Avoid: Single point metric ("average latency" hides tail).

2. Test realistic scenarios ├─ Happy path (normal input) ├─ Edge cases (malformed, empty, huge input) ├─ Error cases (API down, timeout, etc) ├─ Scale (1 user, 10, 100, 1000) └─ Avoid: Only testing happy path.

3. Compare to baseline ├─ Your agent: Measure ├─ Competitor agent: Measure ├─ Industry standard: Measure ├─ Regression check: Did optimization help or hurt? └─ Avoid: No comparison (can't tell if good/bad).

4. Document everything ├─ Benchmark date: When did you measure? ├─ Conditions: What hardware? What load? ├─ Results: Latency, success, throughput ├─ Changes: What optimized since last benchmark? ├─ Trend: Getting better or worse? └─ Avoid: Running benchmarks but not tracking results.

5. Continuous benchmarking ├─ Before ship: Must benchmark (not optional) ├─ After deploy: Monitor production vs benchmark ├─ Quarterly: Re-benchmark (catch degradation early) ├─ On change: Re-benchmark (verify optimization) └─ Avoid: One-time benchmark (performance changes over time).

---

Como começar com Terminal-Bench

Passo a passo: Benchmarking seu agente

=== SETUP (1-2 hours) ===

1. Install Harbor bash pip install harbor-cli

   └─ Harbor: Open-source evaluation harness

2. Choose benchmark ├─ [ ] SWE-bench (coding tasks) ├─ [ ] tau3-bench (reasoning tasks) ├─ [ ] OSWorld (system tasks) ├─ [ ] Terminal-Bench (CLI tasks) └─ Choose based on agent type

3. Setup Vercel account bash vercel login

   └─ Authenticate with Vercel

4. Configure agent ├─ [ ] Create agent entry point ├─ [ ] Define inputs (what can agent accept?) ├─ [ ] Define outputs (what should agent return?) └─ Output: Agent ready for benchmarking

=== RUN BENCHMARK (1-4 hours) ===

1. Run Terminal-Bench on Vercel bash harbor run
   --benchmark terminal-bench
   --agent my-agent
   --env vercel
   --parallel 10

   └─ Runs 100 trials in parallel on Vercel Sandbox

2. Collect results ├─ Latency histogram (distribution) ├─ Success rate (% passed) ├─ Throughput (max concurrent users) ├─ Error breakdown (what failed?) └─ Time to complete: Depends on benchmark size (1-4 hours)

3. Analyze results ├─ [ ] Median latency: Is it acceptable? ├─ [ ] 95th percentile: Are slow users getting too slow? ├─ [ ] 99th percentile: Are very slow users timing out? ├─ [ ] Success rate: Is 95%+ passing? ├─ [ ] Errors: Which inputs cause failures? └─ Output: You understand agent performance

4. Identify bottlenecks ├─ [ ] Which steps take longest? ├─ [ ] Which inputs fail most? ├─ [ ] Where should you optimize? └─ Output: Optimization roadmap

=== ITERATE (ongoing) ===

1. Optimize agent ├─ [ ] Target: Reduce 95th percentile latency ├─ [ ] Change: Implement optimization ├─ [ ] Test: Local verification └─ Output: Optimized agent

2. Re-benchmark bash harbor run
   --benchmark terminal-bench
   --agent my-agent-v2
   --env vercel
   --parallel 10

   └─ Compare to previous run (did optimization help?)

3. Track progress ├─ Benchmark 1: v1 agent (baseline) ├─ Benchmark 2: v2 agent (after optimization 1) ├─ Benchmark 3: v3 agent (after optimization 2) ├─ Trend: Performance improving? Degrading? └─ Output: You know what optimizations work

4. Monitor production ├─ Expected: 95th percentile = 800ms (from benchmark) ├─ Actual: Monitor production (is it matching benchmark?) ├─ Divergence: If production > benchmark, investigate └─ Output: Production stays performant

=== TOTAL TIME INVESTMENT ===

Initial setup: 2-4 hours (one time) First benchmark: 2-4 hours (first run) Re-benchmark: 2-4 hours (quarterly or on major change) Monitoring: 1 hour/month (check production vs benchmark)

Total year 1: ~30-40 hours (one-time + ongoing) ROI: Prevent 1 production incident (caused by untested agent) = saved R$ 50K-500K Payback: 1-3 months (first incident prevented)

---

Checklist: Is your agent benchmarked?

Assess your agent testing maturity

=== AGENT BENCHMARKING CHECKLIST ===

[ ] Pre-production ├─ [ ] Have you run any benchmarks on your agent? ├─ [ ] Do you have baseline metrics (latency, success rate)? ├─ [ ] Do you know 95th percentile latency? ├─ [ ] Do you know max throughput (concurrent users)? ├─ [ ] Have you tested edge cases? └─ [ ] If NO to any: Agent is untested. DON'T SHIP.

[ ] Comparison ├─ [ ] Do you know how fast competitor's agent is? ├─ [ ] Do you know industry standard latency? ├─ [ ] Can you explain why your agent is faster/slower? ├─ [ ] Have you made public SLA claims (e.g., <5 sec)? ├─ [ ] Can you verify your SLA in benchmark? └─ [ ] If NO to any: You're flying blind on competitive fit.

[ ] Bottleneck analysis ├─ [ ] Do you know which steps are slowest? ├─ [ ] Do you know why they're slow (profiling)? ├─ [ ] Have you attempted optimization? ├─ [ ] Did you re-benchmark after optimization? ├─ [ ] Are you tracking optimization results? └─ [ ] If NO to any: You don't know what to optimize.

[ ] Production verification ├─ [ ] Do you monitor production latency? ├─ [ ] Do you compare production to benchmark? ├─ [ ] Do you alert if production diverges from benchmark? ├─ [ ] Do you re-benchmark quarterly? ├─ [ ] Is benchmarking part of release process? └─ [ ] If NO to any: Production performance is unmeasured.

[ ] Documentation ├─ [ ] Do you have benchmark results documented? ├─ [ ] Do you have optimization roadmap? ├─ [ ] Do you have SLA commitment (to customers)? ├─ [ ] Can you explain agent performance to customers? ├─ [ ] Is benchmarking knowledge shared with team? └─ [ ] If NO to any: Knowledge is silos.

=== SCORING ===

Count YES answers: ├─ 20+ YES: You're mature (agent benchmarking is standard) ├─ 12-19 YES: Partially mature (some gaps) ├─ 4-11 YES: Weak maturity (significant risk) ├─ 0-3 YES: No maturity (critical risk)

=== DECISION ===

If 20+ YES: ├─ You're in good shape (keep benchmarking) ├─ Focus: Continuous improvement └─ Risk: Low (untested agent won't ship)

If 12-19 YES: ├─ You have gaps (add benchmarking) ├─ Priority: Add production monitoring └─ Risk: Medium (might ship with issues)

If 4-11 YES: ├─ You have major gaps (critical work needed) ├─ Priority: RUN TERMINAL-BENCH ASAP └─ Risk: High (agent is untested)

If 0-3 YES: ├─ You're completely unbenchmarked (dangerous) ├─ Priority: STOP. Benchmark before shipping. └─ Risk: Critical (untested agent will break production)

---

Conclusão: Agent benchmarking é novo padrão

O que Terminal-Bench + Vercel Sandbox mudou:

1. Benchmarking is now easy (no infra setup needed)

   • Before: Need to setup benchmark infrastructure (hard, expensive)
   • After: Run on Vercel Sandbox (free, easy, no setup)
   • Implication: "No excuse to skip benchmarking. Everyone can do it."

2. Parallel testing is fast (get results quickly)

   • Before: Run benchmarks sequentially (1 trial at a time, slow)
   • After: Run 100 trials in parallel (hours instead of days)
   • Implication: "You can iterate fast. Benchmark multiple times per week."

3. Production-like testing is possible (catch real issues)

   • Before: Test locally (different from production)
   • After: Test on real cloud infrastructure (production-like)
   • Implication: "Benchmark results predict production behavior."

4. Standardized benchmarks exist (compare to others)

   • Before: Each company creates own benchmarks (not comparable)
   • After: Use SWE-bench, tau3-bench, Terminal-Bench (standardized)
   • Implication: "You can compare agent performance to competitors/standards."

5. Benchmarking is now cost-effective (not just for big companies)

   • Before: Only big companies could afford benchmarking infra
   • After: Anyone can run benchmarks on Vercel (cheap/free)
   • Implication: "Even small SaaS can benchmark. Competitive advantage shrinks."

Your decision today:

• Skip benchmarking (hope agent is fast)
• Run Terminal-Bench (know agent is fast)
• Both (benchmark + monitor + optimize)

Recommendation: Start with Terminal-Bench. 4 hours of benchmarking = prevent 100 hours of production debugging. ROI is obvious.

Na OpenClaw:

Ajudamos SaaS builders benchmark agents:

• Agent benchmarking audit: How ready is your agent? (assessment)
• Terminal-Bench integration: Setup + run benchmarks (infrastructure)
• Performance optimization: Identify + fix bottlenecks (engineering)
• Continuous monitoring: Production vs benchmark (observability)
• Competitive analysis: How fast vs competitor agents? (benchmarking)
• SLA verification: Can you back up claims? (testing)
• Optimization roadmap: Priority order for improvements (strategy)

You can assume your agent is fast (hope).

Or you can measure it (know).

Choice: Hope or verify?

Agent Benchmarking | Terminal-Bench Setup | Performance Testing →