for a while I thought choosing the model was a technical decision, the kind you settle by looking at a benchmark. I changed my mind watching the teams that are starting to bring AI in-house. The pattern I see is almost always the same: for any task, grab the biggest model available. Classify an email? Frontier model. Extract a tax ID from a PDF? Frontier model. Answer “yes” or “no” to a binary question? Frontier model.

It works. And that is exactly where the problem lives, because working and being the right choice are different things. The biggest model solves almost everything, so nobody stops to ask whether they needed it. The bill arrives three months later, when volume climbs and the inference invoice starts to rival the payroll of a small team.

The question almost nobody asks at the right moment is simple: what is the smallest model that solves this task at the quality the business requires? That is a business question, not a technology one. And it is what separates teams running AI from teams running expensive AI.

§ 01 / SizingWhat changes when the model is small.

SLM, short for small language model, is a model with orders of magnitude fewer parameters than a frontier LLM, trained or tuned to do a narrow set of things well. Calling it a “bad LLM” misreads what it is. It loses on open-ended reasoning, on tasks that need broad world knowledge, on long and ambiguous conversations. And it wins, comfortably, on four fronts that tend to decide a project:

  • Cost. Inference on a small model costs a fraction of the large one. When the task runs thousands or millions of times a day, whether triage, classification, or extraction, that difference stops being a detail and becomes the project viability.
  • Latency. A smaller model responds faster. In a synchronous flow, like a support interaction the customer is waiting on or a step inside a pipeline that has to close in milliseconds, the large model latency becomes a product constraint, and no amount of engineering gets around it.
  • Privacy and on-prem. A small model fits on hardware you control. For anyone with data that cannot leave the building, such as health, finance, and legal, running on-prem or in a private cloud stops being a preference and becomes a requirement. A frontier model via API, in those cases, is simply not in play.
  • Specific task and edge. If the work is narrow and well defined, a small model tuned for it tends to beat the generic large one. It comes out cheaper and, often, more accurate, because it does not carry the ambiguity of trying to serve everyone. And it fits where the large one cannot: on the device, at the edge, offline.

§ 02 / PatternsThe patterns that make this pay off.

Choosing between large and small is a false dilemma. The systems that hold up in real operation combine the two. Three patterns solve most cases.

Routing (or cascade). The small model answers first. It solves the easy case, which is usually most of the volume, and escalates to the large model only when the task demands it. The escalation rule can be the small model own confidence, the complexity of the input, or a cheap classifier in front of everything. In practice, you pay large-model prices only on the fraction of cases that truly need it. It is the highest-ROI pattern and the first one to consider.

Distillation. You use the large model as a teacher to generate high-quality examples and train a small model to reproduce that behavior on that task. The result is a lean model that inherits part of the large one competence on the slice that matters. It takes work and needs data, but it pays off when volume is high and the task is stable.

Fine-tuning an SLM. For a narrow, recurring task, like your way of classifying tickets, your extraction format, or your response tone, tuning a small model with your own examples delivers a consistency no prompt on the large model guarantees. You trade flexibility for predictability. In production, predictability is almost always worth more.

These patterns compose. A mature system usually has a tuned SLM in front, routing to the large model on the hard cases, and RAG bringing the company own context into the prompt in either one. The logic here is to size each part to the work it does.

§ 03 / When not to saveWhen the large model is still the answer.

Shrinking the model by reflex is the same mistake inverted. The large model remains the right choice, and skimping on it gets expensive, when the task is:

  • Open-ended, multi-step reasoning, where the path is not known in advance.
  • Broad world knowledge, with no own base to anchor it through RAG.
  • Low volume and high value per call. An analysis that runs ten times a day and grounds an expensive decision. Here the inference cost is irrelevant next to the cost of being wrong.
  • Prototype and discovery. Early on, before you know the shape of the task, use the large model to learn the problem. Optimizing for a small model before understanding the work is premature optimization, because you are tuning a model for a task that is still going to change.

The rule: start large to understand, switch to small to operate.

§ 04 / Anti-patternsThe anti-patterns.

Three errors show up every time, and all three cost money:

  1. Large model on a high-volume binary task. Using a frontier model to say “spam / not spam” a million times a day is burning budget out of laziness to measure.
  2. Fine-tuning before exhausting the prompt. Tuning a model takes work, creates a dependency, and ages. Much of what the team wants to solve with fine-tuning is solved with a better prompt, a few examples (few-shot), and RAG. Fine-tuning is the last lever, not the first.
  3. Optimizing cost without measuring quality. Swapping the large model for the small one and not measuring whether quality dropped is like cutting cost by firing someone without looking at what they did. Savings that degrade the result are debt taken on without anyone noticing.

§ 05 / DecisionHow to decide.

Deciding which model to use is a business decision disguised as a technical one. It comes down to crossing three axes: volume (how many times the task runs), value per call (how much one wrong answer costs), and constraint (latency, privacy, hardware). High volume and low value per call push toward the small model. Low volume and high value per call justify the large one. A hard privacy or latency constraint decides on its own.

Before choosing the model for any new task, run this checklist:

  • What is the task, exactly? Narrow and repetitive favors a small model; open and variable favors a large one.
  • What is the daily volume? High volume turns a per-call cost difference into a viability difference.
  • How much does one wrong answer cost? An expensive error justifies paying more per call.
  • Is there a latency constraint? A synchronous flow with a user waiting weighs against the large model.
  • Can the data leave the building? If not, the small on-prem model is the only path, and the discussion ends here.
  • Do I have a measured quality baseline? Without a baseline, any model swap is a bet, not a decision.
  • Can I start with routing? Small model in front, large one on what is left, is almost always the best first step.
  • Am I at the understand stage or the operate stage? Understanding calls for large. Operating at scale calls for the smallest that solves it.
Noûs principle
We size the model to the task, not to the hype. Before choosing, we measure volume, value per call, and constraint. We start large to understand the problem and move to the smallest model that holds the required quality, with routing up front and the large one only where it is irreplaceable.

The biggest model is an excellent, expensive tool. Treating every task as if it deserved the most expensive tool is what happens when nobody stopped to decide. Sizing the model to the work is what makes AI fit the budget and survive day-to-day. It is engineering with a business head, which is where the whole thing should start.

end  ·  field note #46  ·  noûs / aug 26