Knowledge distillation
Also known as: distillation
Knowledge distillation trains a small model (the student) to mimic a large model (the teacher). The student learns to reproduce the teacher’s outputs — and picks up a lot of its behavior at a fraction of the cost.
It’s a core reason small models keep getting surprisingly good:
- Teacher: huge, expensive, high-quality.
- Student: small, cheap, runs on your laptop.
- Result: 8B model that behaves a lot like a 70B one.
The trick that powers local AI: distillation is how you get "near-frontier quality on a laptop". The small model is shadowing the big one.