Prompting vs fine-tuning
Two ways to shape a model's output — when a better prompt wins, when you actually need to train it, and why RAG is usually the smarter middle.
By the end of this page, you'll have a decision rule for "prompt it, RAG it, or fine-tune it?" — and know why most teams should pick the first two.
The three levers, one goal
You want the model to behave a certain way. There are three ways to get there, in order of cost and permanence:
| Lever | What it changes | Cost | Reversible? |
|---|---|---|---|
| Prompting | The instructions the model sees | Free, seconds | Trivially |
| RAG | The knowledge it has access to | Low (embed + store) | Easy |
| Fine-tuning | The model’s weights — how it talks, formats, prioritizes | High (GPU + data + time) | Hard |
Prompting: the first and most underused lever
A prompt is a contract with the model. You can specify:
- Role — “You are a senior backend engineer.”
- Format — “Respond with valid JSON, no prose.”
- Constraints — “If unsure, say ‘I don’t know’.”
- Examples — a few input→output pairs (called few-shot) that show the model exactly the pattern you want.
Classify the sentiment of each line. Return JSON: {"label": "pos"|"neg"|"neu"}
Line: "The battery lasted two full days"
→ {"label": "pos"}
Line: "Shipped a day late, packaging fine"
→ {"label": "neu"}
Line: "Your support took three weeks to reply"
→ {"label": "neg"}
Line: "The onboarding email was clear and quick"
→
That last one is the model’s turn — and the three examples above did more for consistency than any amount of adjectives in a system prompt ever will.
Fine-tuning: what it actually does
Fine-tuning continues the model’s training on your examples. It’s the right tool when you need:
- A consistent output format (legal clause extraction, structured logs)
- A specific tone or register (brand voice, clinical language)
- Niche domain behavior (a model that “speaks” your internal jargon naturally)
It’s the wrong tool when you need:
- New facts → use RAG. Fine-tuning smears facts into weights; you can’t cite, audit, or delete them.
- Safety → use RLHF-style training or guardrails. Fine-tuning on your data won’t fix a model’s bias.
The decision rule
Ask, in order:
- Can I get it with a better prompt? → Do that. 80% of “the model does it wrong” is a prompt problem.
- Does it need knowledge it doesn’t have? → RAG. Fresh, citeable, cheap.
- Does it need a behavior that prompting can’t reliably enforce? → Fine-tune.
That order matters because each step is more expensive, less reversible, and harder to debug than the one before.
Watch out: "I fine-tuned it and it works" often means "I fine-tuned it and it works *for the examples in my test set*". Fine-tuning overfits to the patterns in your data faster than you'd expect. If it breaks on a slightly different input, you don't have a fix — you have a prompt problem you solved with a sledgehammer.
Try it: take a task where your AI output is "close but not right". Write two versions of the prompt: one with adjectives ("be precise, be thorough"), one with three concrete examples. Compare the consistency. You'll likely find the examples version wins — and now you know why fine-tuning is the *last* resort, not the first.