Seu agente desperdça tokens (17K runs revelam quais tools funcionam)

Seu agente desperdiça tokens (17K runs revelam quais tools funcionam)

Você é founder/CEO de SaaS.

Seu SaaS: agente IA (atendimento, vendas, suporte).

Sua atual arquitetura de agente:

• Tool selection: Hardcoded (agente sempre usa Tools A, B, C)
• Tool choice logic: "User asks → Use all available tools (whatever works)"
• Token usage: High (agente tries all tools, wastes tokens on failures)
• Cost per request: R$ 0.05-0.15 (expensive due to inefficient tool use)
• Assumption: "All tools are equally useful (use them all)"
• Reality: "Armature analyzed 17K agent runs (Claude/Codex/Cursor) and revealed which tools actually work best"

Armature study (17,000 agent runs analysis):

What Armature measured:

• Sample size: 17,000 real agent executions
• Models tested: Claude, OpenAI Codex, Cursor
• Question: Which tools do these agents actually choose? Which tools succeed vs fail?
• Finding: Tool selection patterns are predictable (some tools succeed 90%+, others fail 50%+)
• Implication: You can optimize agente tool selection (use high-success tools only, eliminate low-ROI tools)

Why this matters:

Old approach (your agente now):

1. Customer asks: "What's the weather in São Paulo?"
2. Agente thinks: "I have 5 tools available (API, search, cache, LLM, database)"
3. Agente tries: All 5 tools (wasting tokens)
4. Agente succeeds: Via API (but wasted 4 tool calls)
5. Token cost: R$ 0.15 (4 failed tools + 1 success)
6. Customer gets: Correct answer, but expensive

Optimized approach (post-Armature data):

1. Customer asks: "What's the weather in São Paulo?"
2. Agente thinks: "My data shows API succeeds 95% for weather queries"
3. Agente tries: API only (skip the 4 failed tools)
4. Agente succeeds: Via API (first attempt)
5. Token cost: R$ 0.02 (1 successful tool call)
6. Customer gets: Correct answer, 87% cheaper

Difference:

• Old approach: R$ 0.15 per request (inefficient)
• Optimized: R$ 0.02 per request (efficient)
• Savings: 87% cost reduction per request
• Annual impact: R$ 100K → R$ 13K (if 1M requests/year)

---

O problema (seu agente não sabe quais tools são eficientes)

Scenario 1: Your current agente (no tool optimization)

Current behavior:

Your agente tool selection: ├─ Tool availability: 5-10 tools (API, search, database, cache, etc) ├─ Tool selection logic: "Try all tools until one works" ├─ Success pattern: Unknown (you don't measure which tools succeed) ├─ Token waste: High (trying failing tools) ├─ Cost per request: R$ 0.05-0.15 (expensive) ├─ Customer experience: "Agente works, but slow" └─ Margin: Crushed (inefficient token use)

Problem:

• You don't know which tools succeed 90%+ (high-ROI)
• You don't know which tools fail 50%+ (low-ROI, money-wasting)
• Agente wastes tokens trying low-ROI tools
• Cost per request is unnecessarily high
• Competitors optimizing tool selection will undercut your pricing

Scenario 2: Competitor's agente (post-Armature tool optimization)

Optimized behavior (using Armature data):

Competitor agente tool selection: ├─ Tool availability: Same 5-10 tools ├─ Tool selection logic: "Use tools in priority order (highest success first)" ├─ Success pattern: Measured (API = 95%, search = 80%, database = 70%, cache = 60%) ├─ Tool selection: Only use high-ROI tools │ ├─ Query type = "weather" → Use API (95% success) │ ├─ Query type = "search" → Use search API (80% success) │ ├─ Query type = "customer data" → Use database (70% success) │ └─ Query type = "cached answer" → Use cache (60% success) ├─ Token waste: Low (only using high-success tools) ├─ Cost per request: R$ 0.01-0.03 (efficient) ├─ Customer experience: "Agente is fast and cheap" └─ Margin: Healthy (efficient token use)

Advantage:

• Competitor knows which tools succeed 90%+
• Competitor only uses high-ROI tools
• Agente wastes fewer tokens (lower cost)
• Cost per request is 70-87% lower than your agente
• Competitor can undercut your pricing by 50%+ (while keeping margin)

Market signal (Armature data = tool selection optimization is now measurable)

What Armature's 17K runs reveal:

1. Tool performance is NOT equal

   • Some tools succeed 90%+ (high-ROI)
   • Some tools fail 50%+ (low-ROI, money-wasting)
   • Using all tools equally = wasting money

2. Tool selection CAN be optimized

   • Measure which tools succeed for each query type
   • Prioritize high-success tools
   • Skip low-success tools
   • Result: 50-87% cost reduction

3. Competitors WILL optimize

   • Armature data is public (everyone can see tool patterns)
   • Competitors will build tool optimization into their agente
   • Market standard: Only use high-performing tools
   • Your agente (no optimization): Seen as inefficient

4. Urgency: NOW

   • Optimization is easy (measure + prioritize)
   • ROI is huge (70-87% cost reduction)
   • Window: 1-2 months (before market normalizes)
   • If you wait: You're playing catch-up

Implication: "Armature data just made tool optimization table stakes. If your agente doesn't optimize tool selection, you're money-wasting. Competitors will use this data to undercut your pricing. You need to optimize NOW or lose margin + market share."

---

A solução (optimize agente tool selection based on performance data)

Step 1: Measure tool performance (Week 1, R$ 5-10K)

Goal: Understand which tools succeed vs fail for your agente

How to measure:

1. Add logging to agente:

   • Log every tool call (which tool, which query type, success/failure)
   • Log latency (how long did each tool take?)
   • Log cost (how many tokens did each tool use?)
   • Log result quality (did the tool answer the customer's question?)

2. Collect historical data:

   • Analyze last 1-3 months of agente logs
   • Identify patterns: "Weather queries succeed via API 95% of time"
   • Identify failures: "Weather queries fail via cache 40% of time"
   • Build success matrix (Tool X → Query type Y → Success rate Z%)

3. Example matrix for your agente:

   Query type	Tool 1 (API)	Tool 2 (Search)	Tool 3 (DB)	Tool 4 (Cache)
   Weather	95%	30%	10%	5%
   Customer data	40%	20%	92%	80%
   Web search	20%	88%	15%	25%
   FAQ/cached	15%	30%	20%	91%

4. Cost calculation:

   • API call cost: R$ 0.05
   • Search cost: R$ 0.03
   • Database query cost: R$ 0.01
   • Cache lookup cost: R$ 0.001

5. ROI calculation (Weather query example):

   Your agente (no optimization):

   • Try all 4 tools: R$ 0.05 + R$ 0.03 + R$ 0.01 + R$ 0.001 = R$ 0.091
   • Wasted tries: 3 (search 30%, DB 10%, cache 5% all fail)
   • Cost per success: R$ 0.091 ÷ 0.95 = R$ 0.096 per query

   Optimized agente:

   • Use only API (95% success): R$ 0.05
   • Cost per success: R$ 0.05 ÷ 0.95 = R$ 0.053 per query
   • Savings: R$ 0.096 - R$ 0.053 = R$ 0.043 (45% cheaper)

Cost: R$ 5-10K (1 week, engineering time to add logging + analyze) Result: Success matrix for your agente (which tools work for which queries)

Step 2: Build tool optimization logic (Week 2, R$ 10-20K)

Goal: Agente selects high-performing tools based on query type

How to implement:

1. Add tool selection logic to agente: python def select_tools_for_query(query_type): # Use success matrix from Step 1 tools_by_success = { "weather": ["API" (95%), "search" (30%), "db" (10%), "cache" (5%)], "customer_data": ["db" (92%), "cache" (80%), "search" (20%), "api" (40%)], "web_search": ["search" (88%), "cache" (25%), "db" (15%), "api" (20%)], "faq": ["cache" (91%), "db" (20%), "search" (30%), "api" (15%)], } # Return tools in priority order (highest success first) return tools_by_success.get(query_type, all_tools_default_order)

   Example usage:

   def answer_query(query): query_type = detect_query_type(query) # "weather" or "customer_data" etc tools = select_tools_for_query(query_type) # Get prioritized tools for tool in tools: try: result = tool.call(query) # Try highest-success tool first if result: return result # Stop after first success except: continue # Try next tool if this one fails

2. Optimization strategies:

   • Strict: Only use top-1 tool (highest success rate)

     • Weather → Use API only (95% success)
     • Cost: Very low, but might fail 5% of time

   • Balanced: Use top-2 tools (retry if first fails)

     • Weather → Try API (95%), fallback to search (30%)
     • Cost: Low (few retries), reliability good (99% success)

   • Conservative: Use top-3 tools (multiple retries)

     • Weather → Try API (95%), search (30%), db (10%)
     • Cost: Medium (more retries), reliability very good (99.5% success)

3. Choose strategy based on your risk tolerance:

   • High margin tolerance = Strict (lowest cost)
   • Balanced margin + reliability = Balanced (recommended)
   • Quality critical = Conservative (reliability paramount)

Cost: R$ 10-20K (1 week, engineering time to implement tool selection logic) Result: Agente now prioritizes high-performing tools (70-87% cost reduction)

Step 3: Test + validate (Week 3, R$ 5-10K)

Goal: Confirm tool optimization works (same quality, lower cost)

How to test:

1. A/B test:

   • Group A: Old agente (tries all tools)
   • Group B: Optimized agente (prioritized tools)
   • Run both on same 1,000 queries
   • Measure: Success rate, latency, cost
   • Target: Group B ≥ 95% success, <50% cost of Group A

2. Validation metrics:

   • Success rate: (queries answered correctly) / (total queries) ≥ 95%
   • Cost reduction: Old cost / New cost ≥ 2x (at least 50% savings)
   • Latency: Old latency / New latency ≥ 1.2x (at least 20% faster)
   • Customer satisfaction: Same or better (faster response, same quality)

3. Example results:

   Old agente (no optimization):

   • Success rate: 96%
   • Cost per query: R$ 0.10
   • Latency: 2 seconds
   • Monthly cost (1M queries): R$ 100K

   Optimized agente:

   • Success rate: 95% (acceptable, <1% drop)
   • Cost per query: R$ 0.025 (75% cheaper!)
   • Latency: 1 second (50% faster!)
   • Monthly cost (1M queries): R$ 25K (75% savings!)

Cost: R$ 5-10K (1 week, QA testing) Result: Confirmed optimization works (75% cost reduction, 50% faster, same quality)

Step 4: Rollout + monitor (Week 4, R$ 5-10K)

Goal: Deploy optimized agente to production + monitor quality

How to rollout:

1. Gradual deployment:

   • Week 1: Optimized agente on 10% of traffic (shadow mode, monitor quality)
   • Week 2: Optimized agente on 50% of traffic (if quality good, continue)
   • Week 3: Optimized agente on 100% of traffic (full rollout)
   • Keep old agente as fallback (if optimized fails, fall back)

2. Continuous monitoring:

   • Track success rate (target: ≥95%)
   • Track cost savings (target: ≥70% reduction)
   • Track customer satisfaction (target: same or better)
   • Alert if success rate drops below 90% (something broke, rollback)

3. Optimization loop:

   • Every week, re-measure tool performance
   • Update success matrix (tools might change over time)
   • Adjust tool selection logic (keep optimizing)

Cost: R$ 5-10K (1 week, monitoring + ops) Result: Optimized agente live, 75% cost reduction, 50% faster

Total: 4 weeks, R$ 25-50K investment

---

Seu roadmap (4 weeks, R$ 25-50K = tool selection optimization + 70-87% cost reduction)

Week 1: Measure tool performance

• Add logging to agente (which tool, query type, success/failure)
• Analyze historical data (1-3 months of logs)
• Build success matrix (Tool X → Query type Y → Success rate Z%)
• Cost: R$ 5-10K
• Result: Tool performance data (which tools work for which queries)

Week 2: Build tool optimization logic

• Implement tool selection logic (prioritize high-performing tools)
• Choose optimization strategy (strict, balanced, or conservative)
• Deploy to staging environment
• Cost: R$ 10-20K
• Result: Agente now chooses high-performing tools first

Week 3: Test + validate

• A/B test: Old agente vs optimized agente
• Measure: Success rate, cost, latency
• Target: ≥95% success, ≥50% cost reduction, 20% faster
• Cost: R$ 5-10K
• Result: Confirmed optimization works (lower cost, same quality)

Week 4: Rollout + monitor

• Gradual deployment (10% → 50% → 100% traffic)
• Monitor: Success rate, cost savings, customer satisfaction
• Continuous optimization (re-measure weekly, update logic)
• Cost: R$ 5-10K
• Result: Optimized agente live, 70-87% cost reduction

Total: 4 weeks, R$ 25-50K, 70-87% cost reduction, 50% faster agente

---

Conclusão: Armature data reveals tool optimization opportunity

Signal (Armature measures 17K agent runs):

• Tool performance is NOT equal (some succeed 90%+, others fail 50%+)
• Tool selection CAN be optimized (prioritize high-performing tools)
• Cost reduction is HUGE (70-87% savings possible)
• Window is NOW (1-2 months before market normalizes)

Your current exposure:

• Agente wastes tokens trying low-performing tools
• Cost per request is unnecessarily high (R$ 0.05-0.15)
• Competitors will optimize tool selection (using Armature data)
• Market shifting to tool-optimized agentes
• Churn risk: High (if you don't optimize)

Suas opções:

Opção 1: Ignore tool optimization (status quo)

• Your agente: No tool optimization (wastes tokens)
• Cost: R$ 0.05-0.15 per request (expensive)
• Competitors: Using Armature data to optimize tools (70-87% cheaper)
• Market perception: Your agente is inefficient/expensive
• Churn: -10-20% (customers switch to optimized competitors)
• Margin: Crushed (you're paying 5-10x more per request)
• Outcome: Obsolete agente (inefficient tool use)

Opção 2: Optimize tool selection (4 weeks, R$ 25-50K) - RECOMMENDED

• Your agente: Optimized tool selection (data-driven)
• Cost: R$ 0.01-0.03 per request (70-87% cheaper)
• Competitors: Also optimizing (everyone has data)
• Market perception: Your agente is efficient/cheap
• Churn prevention: Zero (agente is now cost-competitive)
• Margin: Healthy (you're paying same as optimized competitors)
• Competitive advantage: 1-2 month lead (before market normalizes)
• Outcome: Best-in-class efficient agente

Your decision window: THIS WEEK

If you START tool optimization this week:

• You're optimized by end of September (4 weeks)
• You have 4-6 week competitive advantage (before competitors catch up)
• Revenue impact: Churn prevention + margin improvement (+15-25%)

If you wait until October:

• You're optimized by end of October (still viable)
• You have 2-4 week competitive advantage (competitors starting same time)
• Revenue impact: Some churn during optimization window (-5-10%)

If you wait until 2027:

• Tool optimization is already standard (everyone has it)
• You're playing catch-up (no competitive advantage)
• Revenue impact: Significant churn (-20-30%, customers switched)

At OpenClaw, ajudamos SaaS agentes optimize tool selection:

• TOOL PERFORMANCE ANALYSIS: Measure which tools succeed for each query type (using 17K runs data)
• SUCCESS MATRIX BUILDING: Create data-driven tool prioritization (Tool X → Query type Y → Success rate Z%)
• TOOL SELECTION LOGIC: Implement smart tool choosing (prioritize high-performing tools only)
• OPTIMIZATION STRATEGY: Choose strict vs balanced vs conservative (risk tolerance tradeoff)
• A/B TESTING: Validate optimization (old vs new agente, same quality, lower cost)
• GRADUAL ROLLOUT: Deploy optimized agente safely (10% → 50% → 100% traffic)
• MONITORING + ITERATION: Track cost savings + quality (continuously optimize)

Result: Seu agente agora otimiza tool selection (75% cost reduction, 50% mais rápido). Token usage cai (você usa só high-performing tools). Cost per request cai de R$ 0.10 para R$ 0.025 (75% savings). Você tem competitive advantage (1-2 month before market normalizes). Margin fica saudável (competitors têm que otimizar também, você já fez).

Seu agente desperdiça tokens?

Tenta todos os tools (mesmo os que falham)?

Custo de R$ 0.05-0.15 per request (caro)?

Quer agente que optimiza tool selection (data-driven, 75% cost reduction, 50% mais rápido)?

Quer competitive advantage (1-2 month antes que mercado normaliza)?

Se não sabe por onde começar OU quer implementação em <4 semanas:

Optimize tool selection agora (4 semanas, R$ 25-50K, 75% cost reduction, 50% mais rápido, data-driven, Armature analysis) →