Prompt vs Fine-tuning vs Custom Model: qual escolher?

Prompt vs Fine-tuning vs Custom Model: qual escolher?

Você é founder de SaaS.

Seu produto:

• Agente de IA (WhatsApp, web, mobile)
• Usa LLM (OpenAI, Anthropic, custom)
• Precisa que agente seja bom em task específica (atendimento ao cliente, vendas, análise de dados)

Seu dilema:

• Task: "Meu agente precisa analisar leads (potencial de compra)"
• Opção A: Prompt engineering (mudar prompt, tentar acertar)
• Opção B: Fine-tuning (treinar modelo com seus dados)
• Opção C: Custom model (treinar modelo from scratch, só seu)
• Your question: "Qual eu uso?"
• Your fear: "Se escolher errado, gasto R$50K em fine-tuning que não funciona"
• Your real problem: "Não sei quando cada opção é certa"

A notícia que responde:

AWS publicou guia: "Generative AI Customization Spectrum" — explicando exatamente quando usar cada nível de customização (sem over-engineer).

Ideia: Existe um spectrum (escala). Você deve escolher o MÍNIMO que resolve seu problema.

Maior erro de founder: Gastar muito (fine-tuning, custom model) quando simples prompt teria bastado.

---

O Spectrum: De simples pra complexo

3 níveis. Cada um custa diferente. Qual é certo?

=== THE SPECTRUM ===

Level 1: Prompt Engineering (MAIS SIMPLES) ├─ What: Mudar o prompt/instrução do modelo ├─ Example: "Analise este lead: [lead data]. Responda: alto/médio/baixo potencial." ├─ Cost: R$0 (você só escreve melhor prompt) ├─ Time: 1-2 dias (teste iterativo) ├─ Quality: 70-80% (bom pra tasks simples) ├─ Scaling: Fácil (mesmo prompt pra todos) └─ Risk: Baixo (testa antes de usar)

Level 2: Fine-tuning (INTERMEDIÁRIO) ├─ What: Treinar modelo com seus dados (ajustar pesos) ├─ Example: "Treinar ChatGPT com 1000 examples de leads seu = modelo especializado" ├─ Cost: R$5K-20K (depende de quantidade dados + iterações) ├─ Time: 1-2 semanas (coleta dados, treinamento, teste) ├─ Quality: 85-95% (muito bom pra task específica) ├─ Scaling: Moderado (precisa re-treinar se novo tipo de dado) └─ Risk: Moderado (custo é alto, requer dados de qualidade)

Level 3: Custom Model (MAIS COMPLEXO) ├─ What: Treinar modelo from scratch (novo modelo, só seu) ├─ Example: "Treinar modelo próprio de leads (não usar GPT, Claude, etc)" ├─ Cost: R$100K-500K+ (infraestrutura, computação, pesquisadores) ├─ Time: 3-6 meses (coleta de dados massiva, treinamento complexo) ├─ Quality: 95%+ (melhor possível, otimizado 100% pra seu caso) ├─ Scaling: Muito difícil (precisa gerenciar infraestrutura própria) └─ Risk: Alto (custo grande, tecnicamente complexo, pode não valer)

=== THE DECISION MATRIX ===

Use PROMPT ENGINEERING se: ├─ Task é simples (classificação, summary, resposta padrão) ├─ Accuracy 70-80% é bom o bastante ├─ Dados são públicos/genéricos (não precisa de conhecimento específico) ├─ Velocidade é importante (quero testar hoje) ├─ Budget é limitado (startup, bootstrapped) └─ Examples: "Classifique sentiment de review", "Resuma este texto", "Responda FAQ"

Use FINE-TUNING se: ├─ Task é específica do seu negócio (precisa de contexto) ├─ Accuracy 85-95% é crítico ├─ Você tem dados de qualidade (exemplos bons de input/output) ├─ Task vai rodar 1000x+ vezes por mês (vale o investimento) ├─ Budget permite (R$10K-30K) └─ Examples: "Classifique leads por potencial de venda", "Extraia dados de contrato", "Gere email de resposta personalizada"

Use CUSTOM MODEL se: ├─ Task é estratégica (core competência da empresa) ├─ Accuracy 95%+ é ESSENCIAL (custo de erro é alto) ├─ Você tem datasets massivos (100K+ exemplos) ├─ Task vai rodar 1M+ vezes/mês (economia de scale justifica) ├─ Budget é alto (R$100K+) E você tem time técnico └─ Examples: "Detecção de fraude em real-time", "Recomendação de produto", "Autonomous decision-making"

=== REAL EXAMPLES (SUA SITUAÇÃO) ===

Example 1: Lead scoring (classificar leads por potencial) ├─ Your task: "Analisar lead, dar score 0-100 (potencial de venda)" ├─ Start with: PROMPT ENGINEERING │ ├─ Prompt: "Você é expert em vendas. Analise: [lead]. Score: [0-100]. Justifique." │ ├─ Test: Roda 10 leads, vê se acerta │ ├─ Cost: R$0 │ ├─ Time: 1 dia │ └─ Accuracy: 70% (alguns scores wrong, mas ok pra começar) │ ├─ If accuracy is bad (< 70%): │ └─ Move to: FINE-TUNING │ ├─ Collect: 500-1000 examples (lead + score esperado) │ ├─ Train: ChatGPT FineTuned (2-3 dias) │ ├─ Cost: R$5K-10K │ ├─ Test: Accuracy agora 90%+ │ └─ Decision: Deploy (worth it if running 1000+ leads/mês) │ ├─ If accuracy still bad (< 90%) or running 1M+ leads/mês: │ └─ Consider: CUSTOM MODEL │ ├─ Only if: Revenue impact é huge (erro custa muito) │ ├─ Cost: R$100K+ │ ├─ Time: 3-6 months │ ├─ Accuracy: 98%+ │ └─ Decision: Only large companies (Nubank, Magazine Luiza, etc) │ └─ Most cases: PROMPT → FINE-TUNING (STOP) └─ Custom model is overkill for 99% of SaaS

Example 2: Customer support (responder perguntas de cliente) ├─ Your task: "Agente responde pergunta do cliente (FAQs, status, etc)" ├─ Start with: PROMPT ENGINEERING │ ├─ Prompt: "You are helpful support agent. Answer: [question]" │ ├─ Test: 20 questions, vê qualidade │ ├─ Cost: R$0 │ ├─ Time: 1 dia │ └─ Accuracy: 75% (respostas ok, mas às vezes missing context) │ ├─ If ok-ish (75%+): │ └─ STOP HERE! Don't fine-tune yet │ ├─ Why: Support is best-effort, customers forgive 25% wrong answers │ ├─ Cost savings: R$5K-10K (não gastar em fine-tuning) │ ├─ Time savings: Deploy today (não esperar 2 semanas) │ └─ Decision: Use prompt engineering, add human escalation if confused │ └─ If critical accuracy needed (< 2% error rate): └─ Then: FINE-TUNING ├─ Collect: 1000 Q&A examples (customer question + ideal answer) ├─ Train: 5-7 days ├─ Cost: R$10K-20K ├─ Accuracy: 98%+ └─ Decision: Worth it if support cost savings > training cost

Example 3: Email personalization (escrever email customizado) ├─ Your task: "Agente escreve email pra cliente (name, context personalized)" ├─ Start with: PROMPT ENGINEERING │ ├─ Prompt: "Write personalized email. Name: [name]. Context: [context]. Tone: professional." │ ├─ Test: 10 emails, pede feedback │ ├─ Cost: R$0 │ ├─ Time: 1 dia │ └─ Accuracy: 80% (emails bons, mas às vezes generic) │ ├─ If ok (80%+): │ └─ STOP HERE (best-effort, não precisa ser perfeito) │ ├─ Why: Email personalization é subjective (não há "certo" absoluto) │ ├─ Cost savings: R$5K-10K │ └─ Decision: Deploy, measure actual open rates, improve based on data │ └─ If measuring open rates and they're low (< 20%): └─ Then: FINE-TUNING ├─ Collect: 500 emails (your emails + open rates from past campaigns) ├─ Train: Model learns what emails get opened ├─ Cost: R$5K ├─ Accuracy: 5-10% improvement in open rates └─ Decision: Worth it if R$5K < ROI improvement (usually yes)

---

A verdadeira pergunta: Quando PARAR de fazer prompt engineering?

Maioria das SaaS para na prompt engineering. Essa é a verdade inconveniente.

=== THE REALITY CHECK ===

Statistics (estimated, based on AWS/Anthropic data): ├─ 90% of SaaS: Stop at prompt engineering (good enough) ├─ 8% of SaaS: Move to fine-tuning (justified ROI) ├─ 2% of SaaS: Build custom models (hyper-scale, hyper-critical) └─ Conclusion: You probably don't need fine-tuning OR custom model

=== THE PROMPT ENGINEERING PLAYBOOK ===

Step 1: Write baseline prompt (1 day) ├─ Start simple: "Analyze this [input]. Output: [format]." ├─ Test: 10-20 examples ├─ Measure: Accuracy, latency, cost └─ Decision: Is 70-80% accuracy good enough?

Step 2: Iterate prompt (3-5 days) ├─ Technique 1: Add examples (few-shot learning) │ └─ Before: "Analyze this lead." │ └─ After: "Analyze this lead. Examples: [3 good examples]. Now analyze: [input]." ├─ Technique 2: Add context (system prompt) │ └─ "You are expert in [domain]. You care about [metric]. Analyze: [input]." ├─ Technique 3: Add reasoning (chain-of-thought) │ └─ "Think step-by-step. First, identify [thing]. Then, analyze [thing]. Finally, output." ├─ Technique 4: Change model (if current is bad) │ └─ Try: Claude instead of GPT, Mistral instead of Claude (different strengths) └─ Measure: Accuracy should improve to 85-90%

Step 3: Decision point ├─ If accuracy is 85%+: STOP, deploy to production │ └─ You're done. Move to next feature. ├─ If accuracy is 70-84%: Consider fine-tuning OR accept 70-84% (depends on task) │ └─ Evaluate: Cost of fine-tuning vs benefit of 10-20% improvement ├─ If accuracy is <70%: Either fine-tune OR task might be wrong │ └─ Ask: "Is this task even possible with current approach?" └─ Most of the time: 85%+ accuracy is achievable with step 1-2

=== COST COMPARISON (REAL NUMBERS) ===

Your SaaS: 100K API calls/month (moderate usage)

Option 1: Prompt engineering only ├─ LLM cost: 100K calls × R$0.001/call = R$100/month ├─ Engineering time: 5 days (1 engineer) = R$5K ├─ Total initial cost: R$5K ├─ Monthly cost: R$100 ├─ Accuracy: 85% └─ ROI: Excellent (5K upfront, 100/month ongoing)

Option 2: Prompt engineering + fine-tuning ├─ LLM cost: Same R$100/month ├─ Engineering time: 5 days (prompt) + 10 days (fine-tuning) = R$15K ├─ Fine-tuning cost: R$10K (training, data prep) ├─ Total initial cost: R$25K ├─ Monthly cost: R$100 + R$500 (fine-tuned model overhead) ├─ Accuracy: 92% (+7% improvement) └─ ROI: Questionable (25K upfront, is +7% worth it?)

Option 3: Custom model ├─ LLM cost: N/A (your model) ├─ Engineering time: 5 days (planning) + 120 days (development) = R$125K ├─ Infrastructure cost: R$2K/month (GPUs, servers) ├─ Data labeling: R$20K (1000 examples × R$20) ├─ Total initial cost: R$145K ├─ Monthly cost: R$2K + personnel ├─ Accuracy: 98% (+13% improvement) └─ ROI: Only if massive scale (1M+ calls/month, saving R$10K/month)

=== DECISION FRAMEWORK (FLOWCHART) ===

Question 1: Is your task simple? (classification, summary, Q&A) ├─ YES → Use PROMPT ENGINEERING │ └─ Expected accuracy: 80-85% │ └─ Cost: R$0-5K │ └─ Time: 1-5 days │ └─ Stop here (don't over-engineer) │ └─ NO → Question 2: Do you have 500+ quality examples? ├─ NO → Use PROMPT ENGINEERING │ └─ You need data first, can't fine-tune yet │ └─ Cost: R$0-5K │ └─ Plan: Collect data over time, revisit later │ └─ YES → Question 3: Is accuracy gap (85% → 95%) worth R$10K? ├─ NO → Use PROMPT ENGINEERING │ └─ Benefit doesn't justify cost │ └─ Cost: R$0-5K │ └─ Decision: Accept 85% accuracy, move on │ └─ YES → Use FINE-TUNING └─ Expected accuracy: 90-95% └─ Cost: R$10K-20K └─ Time: 1-2 weeks └─ Decision: Deploy (you did the math, it's worth it)

=== RED FLAGS (Don't fine-tune if you see these) ===

❌ Red flag 1: "We want to fine-tune because everyone's doing it" └─ Reality: Most companies waste money on fine-tuning that prompt engineering would solve

❌ Red flag 2: "We need 99% accuracy but don't have examples" └─ Reality: Can't fine-tune without data. Start with prompt engineering, collect data, revisit

❌ Red flag 3: "We'll fine-tune to save on LLM costs" └─ Reality: Fine-tuning costs (infrastructure, training) > LLM savings for most SaaS

❌ Red flag 4: "We're building custom model to be different" └─ Reality: Custom models only make sense at hyper-scale (not for most startups)

✅ Green flag 1: "We have 1000+ examples and measured accuracy gap" └─ Action: Fine-tuning might be worth it

✅ Green flag 2: "Our task is failing at 60% accuracy and we've iterated prompt 20+ times" └─ Action: Fine-tuning could help, but first check if task is solvable

✅ Green flag 3: "We're running 1M+ LLM calls/month and accuracy costs us R$100K/month" └─ Action: Custom model ROI might work, get technical assessment

---

Como começar: Roadmap em 4 semanas

Estrutura clara. Sem over-engineering. Sem waste.

=== WEEK 1: BASELINE PROMPT ===

Day 1-2: Define task ├─ What exactly needs to happen? (be specific) ├─ Example: "Given a customer support ticket, output: (1) category, (2) urgency, (3) next action" ├─ Why: Vague tasks lead to vague prompts └─ Owner: You + product team

Day 3-4: Write initial prompt ├─ Start simple: "[Task description]. Output: [format]." ├─ Example: "Categorize this support ticket. Categories: billing/technical/general. Output JSON." ├─ Test: 5-10 examples manually (don't automate yet) ├─ Measure: How many correct? (rough accuracy) └─ Owner: You (1-2 hours)

Day 5: Iterate 1st time ├─ What failed? (analyze wrong answers) ├─ Improve prompt: Add missing context ├─ Test again: 5-10 new examples ├─ Measure: Better? Worse? Same? └─ Owner: You (1-2 hours)

=== WEEK 2: IMPROVE PROMPT ===

Day 1-3: Iterate prompt (techniques) ├─ Technique 1: Few-shot (add examples to prompt) ├─ Technique 2: Chain-of-thought (ask model to think step-by-step) ├─ Technique 3: System prompt (add context about model role) ├─ Technique 4: Format specification (be very explicit about output format) ├─ Test each: Pick best one └─ Owner: You (engineering) + QA (testing)

Day 4-5: Test at scale ├─ Run prompt on 100 examples (not 10) ├─ Measure accuracy: % correct ├─ Measure latency: avg time per call ├─ Measure cost: R$ per 1000 calls ├─ Decision: Is 85%+ accuracy? If yes → STOP, deploy. If no → continue iterating └─ Owner: Engineering

=== WEEK 3: DECISION POINT ===

Meeting: Prompt engineering worked? ├─ Accuracy >= 85%? │ ├─ YES → DEPLOY TO PRODUCTION │ │ ├─ Timeline: Next Monday (3 days) │ │ ├─ Monitoring: Track accuracy in production (A/B test if needed) │ │ └─ Success: Done! Move to next feature │ │ │ └─ NO (accuracy < 85%) → EVALUATE FINE-TUNING │ ├─ Question 1: Do we have 500+ examples? │ │ ├─ NO → Collect data first (next 2-4 weeks), revisit │ │ └─ YES → Question 2: Is fine-tuning ROI positive? │ │ ├─ NO → Accept 85% accuracy or pivot task │ │ └─ YES → Move to Week 4 (fine-tuning) │ └─ Owner: Product + engineering

=== WEEK 4: FINE-TUNING (IF NEEDED) ===

Day 1-2: Prepare data ├─ Collect 500-1000 examples (input + expected output) ├─ Clean data: Remove errors, ensure consistency ├─ Split: 80% train, 20% test ├─ Format: Prepare in fine-tuning format (JSONL) └─ Owner: Data engineering / QA

Day 3-4: Train ├─ Use AWS Bedrock / OpenAI fine-tuning API ├─ Train: 2-6 hours (depends on size) ├─ Cost: R$5K-10K ├─ Test: Evaluate on test set ├─ Compare: Fine-tuned vs baseline └─ Owner: Engineering

Day 5: Deploy ├─ If accuracy improved (90%+) → Deploy fine-tuned model ├─ If accuracy didn't improve → Keep baseline (fine-tuning failed, abort) ├─ Timeline: Production by end of day └─ Owner: Engineering

=== TIMELINE SUMMARY ===

Most likely path: PROMPT ENGINEERING ONLY ├─ Week 1: Baseline prompt ├─ Week 2: Improve prompt ├─ Week 3: Test & approve ├─ Week 4: Deploy ├─ Cost: R$0-5K ├─ Accuracy: 85-90% └─ Result: Good enough, move on

Rarer path: PROMPT + FINE-TUNING ├─ Week 1-2: Prompt engineering (baseline) ├─ Week 3: Data collection (if not ready) ├─ Week 4: Fine-tuning ├─ Cost: R$15K-20K ├─ Accuracy: 92-95% └─ Result: Worth it only if ROI is clear

Rarely: CUSTOM MODEL ├─ Only after proving fine-tuning worked ├─ Only if running 1M+ calls/month ├─ Timeline: 3-6 months ├─ Cost: R$100K+ └─ Result: Hyper-optimized, only for giants

---

Conclusão: O maior erro é over-engineering

A realidade (2025-2026):

• Prompt engineering resolve 90% de casos
• Fine-tuning resolve 9% (quando prompt não bastou)
• Custom models resolve 1% (hyper-scale only)
• Founder mistake: Jump to fine-tuning sem testar prompt primeiro
• Winner: Companies que fazem prompt bem (85% accuracy rápido)
• Loser: Companies que gastam R$100K em fine-tuning que não precisa

Seu roadmap (escolha agora):

┌────────────────────────────────────┐ │ OPÇÃO A: Jump to fine-tuning │ ├────────────────────────────────────┤ │ Cost: R$20K upfront (waste) │ │ Time: 3-4 weeks │ │ Risk: Might not work (data issues) │ │ Result: Overkill 90% of time │ └────────────────────────────────────┘

┌────────────────────────────────────┐ │ OPÇÃO B: Start with prompt NOW ✓ │ ├────────────────────────────────────┤ │ Cost: R$0-5K (no waste) │ │ Time: 1-2 weeks │ │ Risk: Low (easy to iterate) │ │ Result: 85% accuracy, move on │ └────────────────────────────────────┘

Na OpenClaw:

Ajudamos SaaS escolher nível certo de customização LLM:

• Task assessment: Qual é sua task realmente? É simples ou complexa?
• Prompt strategy: Como estruturar prompt para máxima accuracy (sem over-engineering)
• Accuracy benchmarking: Test actual accuracy (prompt vs fine-tuning vs custom)
• ROI calculation: Custa R$10K fine-tuning, mas vale a pena?
• Roadmap planning: Qual é próximo passo? Fine-tuning ou outro modelo?
• Production deployment: Como testar em produção sem quebrar usuários

Você quer saber se seu LLM task precisa fine-tuning (ou se prompt basta)?

Prompt Strategy | Accuracy Testing | ROI Calculation | Roadmap →