Small, specialised models are quietly winning

Frontier models are extraordinary and they are the right tool for open-ended reasoning. But a lot of production AI is not open-ended reasoning. It is classification, extraction, routing, summarising a known format, rewriting text, a step inside an agent. For those, the industry is discovering that a much smaller model, often open-weight and sometimes fine-tuned on a few thousand examples, does the job at a fraction of the cost and latency.

Why the shift is happening now

• Open-weight models in the small-to-mid range have caught up to where frontier models were 18 months ago
• Inference cost at scale became a real line item, and small models are an order of magnitude cheaper
• Latency matters for anything interactive, and small models respond in a fraction of the time
• Fine-tuning tooling got simple enough that a normal engineering team can do it
• On-device and in-VPC deployment solves data-residency concerns that block some clients entirely

The pattern that works: route, do not pick

The strongest architectures do not choose one model. They route: a cheap fast model handles the common, easy cases and escalates the hard or low-confidence ones to a larger model. Most traffic never touches the expensive model, and quality on the cases that need it stays high.

What it costs you

Running your own models means owning inference: GPUs or an inference provider, autoscaling, evaluation when you upgrade the base model, a rollback path. It is real platform work. The trade is predictable cost, lower latency, data control and no dependency on one vendor's pricing and availability.

A reasonable default

1. 1Prototype on a frontier model to prove the task is possible
2. 2Once it works, measure how hard the task actually is
3. 3Try the smallest model that could plausibly do it, with good context
4. 4If it is close, fine-tune on real examples and re-measure
5. 5Keep the frontier model as the escalation path for hard cases