Organizing Execution

Model Routing: Matching the Executor to the Task

How to match each step of a workflow to a suitable model configuration by its result, risk, verifiability, tools, permissions and constraints, and how to…

Chapter 43Organizing Execution

A team may have access to several models, products and ways of calling them. The most expensive model is not necessarily needed for every step, and the cheapest per token will not necessarily produce the cheapest accepted result. Routing is the decision about which execution configuration suits a particular piece of work, given its requirements, its risk and how well it can be verified.

Learning outcome. After this unit you will be able to distinguish a model from the product that runs it; choose a configuration that takes into account the task, the data, the tools and the consequences of an error; define a safe escalation route; compare full cost and quality on representative tasks; and explain the limits of a small test and how changeable model catalogues are.

Prerequisites. This unit builds on context and its constraints from Context as a Limited Resource, sources of truth from Source of Truth, specialisation and human accountability from Human Teams, Expertise and Specialist Agents, and parallel work streams from Parallelism. Metrics and the fuller economics of a studio are covered in Agent Evaluation and The Economics of an AI Studio.

We route work; we do not hunt for one best model

Model routing is choosing a model or another configuration for a step in a workflow. The configuration can include the model, the product or application programming interface (API), the reasoning level, the instructions, the available tools, permissions, data sources, budget limits and the method of verification. Looking only at the model's name is like choosing a tool without checking whether it is connected to the environment you need.

For a route, first establish:

  1. The type of task and the desired result. Classifying a message, finding a fact in documents, proposing an architecture, writing code, changing many similar files or reviewing a diff are different kinds of work. State what will count as a correct result.
  2. The cost of an error. A wrong tag that can be corrected is different from a faulty data migration, a dangerous action or a conclusion that affects someone's rights. Serious consequences demand stronger constraints, evidence and human oversight, regardless of the model's advertised abilities.
  3. Verifiability. Is there a test, schema, source or deterministic rule that can accept or reject the answer? When the condition is formal and stable, a script or an ordinary program is often a better fit than a language model.
  4. The full configuration. Does the chosen product support the required input types, context length, structured output and tools? Does it have the necessary permissions and access to the data?
  5. Operational constraints. Response time, availability, queues, rate limits, budget, privacy rules and regional processing can rule out an option before quality is even compared.

Only after these filters should you compare the remaining options on quality, time and cost using your own tasks. A longer context, a higher reasoning level or a strong model does not make up for missing data, a wrong criterion or a missing permission.

Model, product and working environment are not the same thing

The model produces an answer, or a request to use a tool, within the limits of its capabilities. The product or integration determines exactly how it is called, which tools are connected, what environment state is preserved and what it is allowed to do. For example, a model name appearing in an API catalogue does not prove that the same model is available in every chat or plan. Likewise, "supports tool use" does not mean that a tool is enabled, has access, or can run a command without a host.

Check at least the following differences between candidates:

  • Input and context. Text, images, audio or files; the actual request limit; the ability to find the relevant detail in a long context. The advertised window size does not guarantee that the model will use every part of it equally well.
  • Output and format. Can you get a schema, a JSON (JavaScript Object Notation, a text format for structured data) object or a short classification that code can check? A structured format makes processing easier, but it does not make the values true.
  • Tools. Web search, files, a terminal, an editor, a database or an API are available only through a specific product and a configured host. A command emitted by the model does not prove it was actually run: check the log of the action that really happened and the tool's response.
  • State and permissions. Is the agent working in the right repository or account? Does it have read-only access, can it edit, run code or change real state? Permissions are set by the environment, not by the text of the answer.
  • Time and availability. Long reasoning, retries, loading context and queues can all increase latency. A published rate limit is not a guarantee of the throughput available to your particular account.
  • Data and policy. Reconcile the exact product, endpoint, tools, and retention and processing settings with your organisation's rules. A general statement about privacy, or a setting such as turning off retention, does not prove that the whole route meets the requirements.

Keep a short register of verified options: provider and model identifier (ID), product or API, date checked, context and formats, tools and permissions, data-processing terms, current limits, prices, task class and the result of your internal evaluation. A dated record like this helps you spot assumptions that have gone stale. It must not turn into a permanent leaderboard.

Simple and combined routing strategies

For a small or new system, the initial route can often be a single suitable, high-quality option. That gives you a baseline for comparison. Add more models only when there is a concrete saving, availability need or quality requirement that can be measured.

Strategy When it may help What needs to be controlled
Single route Low volume, high consequences of error, or no evidence yet that splitting helps Cost and time of the baseline route; the risk of a single point of unavailability
Explicit rule The task type is known, or metadata reliably indicates the tool or class of work Unknown types and exceptions must not silently end up on an unsuitable route
Classifier before the call The input has several recurring classes, and the route for each has been measured Classifier errors, its own latency and cost, an "I don't know" state, the order of fallback routes
Cheaper first step with verification and escalation The first result often passes an independent test, and failed cases can safely be retried with a stronger option Quality of the verification, repeated calls, duplicated tokens, hidden failures and the full cost after escalation
Strong planner and fast executors The work can be split into bounded, verifiable steps with different requirements Coordination overhead, shared resources, and the cost of consolidation and shared verification discussed in The Orchestrator Pattern, Human Teams, Expertise and Specialist Agents and Parallelism
Second reviewer A separate critical reading, or a search for missed conditions, is needed A different model or provider is not evidence of independent correctness if neither saw the necessary source or both share the same false assumption

A classifier can be confidently wrong. So define when it returns "unknown" and which safe path applies to an unfamiliar task. If a misrouted task costs more than simply using the high-quality default option, classification does not justify its complexity.

Escalation is triggered by observable conditions: required evidence is missing, a test has failed, a retry has not helped, sources contradict one another, the request goes beyond the permissions, the task has become riskier, or the problem statement itself has changed. A model's self-assessed confidence is not a calibrated probability of being correct until it has been compared with outcomes on a suitable sample. At a critical step you need a rule tied to evidence or a check, not just to the phrase "I'm confident".

Worked decisions for three parts of one process

Imagine a service that receives support requests, proposes corrections to its reference data and plans schema changes. The route is decided separately for each step:

  1. Classify a message. If a keyword or an approved format unambiguously determines the label, a rule runs first. For free text, the model returns a label from a closed list plus a "needs clarification" field. Code checks the schema, and a person periodically checks a sample along with every unknown or contradictory case. There is no need to call a sophisticated model for a record where a rule has already given an unambiguous answer.
  2. Find a source for the answer. If the answer depends on a current internal document, you need a route with access to the latest version. The check compares the key claim against the document. If the search returns no confirmation or the documents conflict, the route must hand the case to a specialist. A bigger context without retrieving the right version does not cure stale knowledge.
  3. Design a migration. The model can gather dependencies, compare reversible options and draw up a safe rollout plan, but first it needs the schema, the readers and writers of the old format, the recovery constraints and authoritative sources. If the reviewer cannot confirm a critical compatibility condition or the order of the migration, the next step is checking the documentation, analysing the code or a human decision. Running a few more models on the same incomplete context does not produce the missing evidence.

For repetitive file formatting, a stable script can be the main executor, with a model used only for ambiguous text. For code, you are choosing more than a model: you need a working copy, specific tools, restricted permissions, checks on the output and an action log. A reviewer without the code, a test or a source may find inconsistencies in the reasoning, but cannot confirm actual compatibility.

Evaluate on identical tasks by accepted result

A demo on one lucky prompt or a model's general ranking does not answer the question of how it will handle your team's actual work. Before changing a route, assemble a sample that reflects the real distribution of tasks and includes edge cases. For each class, define an acceptable result, the error rate you can tolerate and a criterion that catches a wrong answer.

Compare candidates on the same task and the same process. Fix the data revision, the instructions, the environment, the available tools and the permissions; record the model ID and version, the reasoning level, the speed mode and the date. Distribute the cases equally among candidates; if behaviour varies, repeat some of the runs. Changing the test conditions at the same time as the model makes it impossible to attribute a difference to a single factor.

The evaluation should cover several dimensions:

  • Quality of the accepted result: the share of tasks passing the same criterion; critical errors; errors by class; the rate of false acceptances and false rejections.
  • Time: total time to acceptance, the median and the slow tail, including waiting, retries and human review.
  • Full cost: calls to all models, retries, input and output tokens, tools, search and compute, the router itself, human review, fixes and integration.
  • Oversight work: how many answers a person read, what they had to correct, why they rejected a result and how much time they spent.
  • Process reliability: tool failures, dropped connections, rate limiting, switching to a fallback path and the ability to resume a task.

A useful economic measure is the cost of the whole process divided by the number of verified, accepted results. Report it together with the failure rate: an average cost per success will hide a process that cheaply rejects most tasks. For subscription or plan-based products, show usage and the availability limit separately; do not attribute to them a precise marginal cost per token that the metering system does not provide.

A small sample gives a preliminary estimate, not a permanent ranking. Break down conclusions by type and risk; record the number of runs and the spread. If the difference between candidates is small compared with the run-to-run variation or the number of human reviews, there is not enough data for a reliable choice. Repeat the evaluation after any change to the model, instructions, tools, data or criteria. As the official evaluation guides from OpenAI and Anthropic note, quality requires criteria defined in advance, representative examples and graders suited to the task; an automated grader also needs to be checked (OpenAI: evaluation best practices, Anthropic: define success criteria and build evaluations).

Teaching example: a cheap model can raise total costs

An imaginary team has 100 similar tasks. For this example it has set internal notional costs: a short call to an inexpensive configuration costs $0.02, and to a stronger one $0.10; a five-minute human review costs $2.50 per task, at a notional labour cost of $30 an hour. These figures are not a provider's price list. Assume also that all the results shown went through one and the same verification process and were ultimately accepted. The table does not include the cost of building the router or the evaluator.

Plan for 100 accepted tasks Model calls Human review Total Cost per accepted task
Stronger configuration straight away: 100 × $0.10; five exceptions × $2.50 $10 $12.50 $22.50 $0.225
Inexpensive first: 100 × $0.02; 20 escalations × $0.10; four exceptions × $2.50 $4 $10 $14 $0.14
Inexpensive first, but many escalations: 100 × $0.02; 60 escalations × $0.10; 20 exceptions × $2.50 $8 $50 $58 $0.58

In this notional scenario, a cheap first step lowers the full cost only when most tasks pass verification and few of them reach a person. With a high escalation rate, the cheaper call price no longer offsets the human labour. For a real choice, the team must measure the same acceptance criteria on both routes and account for rejected tasks; the example demonstrates the arithmetic, it does not prove that one route has higher quality.

A limited engineering case: same system, different task, different result

In September 2026 Fiddler AI described three series of coding tests run on a single agent loop. In the first series, two models worked through five Python debugging tasks: three repetitions, three passes, 45 runs per model. In this experiment Haiku 4.5 had half the per-token price of Sonnet 5, yet the run costs calculated by the authors came to $4 for Haiku versus $0.45 for Sonnet. In one run, Haiku generated a 21,492-token response describing 60 shell commands, even though the loop allowed one command per turn and actually executed only the first. The model continued its response as if the commands it had predicted had already run; those tokens cost money, and the long response enlarged the context of the following turns. A task counted as a verified success if it passed hidden tests; across the three passes over those five tasks, the ratio between the two models' cost per confirmed success ranged from 8.4 to 14.1. The authors noted that this gap is sensitive to individual long responses, so it should not be read as a universal figure.

In another series, with nine specially written tasks, the visible tests accepted every run, but hidden tests revealed gaps. The results also differed on the HumanEval+ benchmark. This illustrates that a token price and one set of tasks predict neither the full cost nor the ranking on other work. The dollar amounts are the accounting of runs in this experiment, not an organisation's full cost: the study did not measure human integration or review. It was also produced by a vendor of a monitoring tool, the first series had only five tasks, and it used a single agent loop. Read it as an argument for testing your own process, not as a model ranking or as proof that cheaper models always lose (Fiddler AI: a study of coding-agent runs, 11 September 2026).

Risk, escalation and recovery

A fallback route is part of the initial design. It should define what the system does if the model is unavailable, a limit has been exhausted, the output does not match the schema, a tool has failed or quality falls below the set threshold. Retry only after classifying the cause: one more identical call will not fix an outdated document, a missing permission, a broken parser or a contradictory requirement.

For risky tasks, limit the consequences regardless of which model you choose: read-only mode first; tool permissions on the principle of least privilege; a test database or a simulation for changes; human review before any action that is hard to roll back. A strong model output can help a specialist, but it does not give the system permission to change the production environment.

Record the full chain: the task and the route, the version, the actual inputs and outputs, tool calls and their results, the automated criterion, human edits, escalation, and acceptance or rejection. Without this, a token counter on its own does not explain why a run was expensive or correct. Access to such a log and its retention period must also comply with your data-handling rules.

Summary

Route each type of work to a suitable path, taking into account the result, the risk, verification, context, tools, permissions, time, budget and data policy. Start with a simple, measurable route; add classification, a cascade or a second reviewer only when there is representative evidence that it helps. Compare the cost to an accepted result together with quality, latency, retries and human labour. Date your catalogue and repeat the evaluation whenever models, products or conditions change.

Sources and further reading