Introduction

When a model provider times out, sending the request to another model looks like an easy reliability win. The second model may return valid JSON and still break the application: it might omit provenance, handle ambiguous input differently, or send sensitive data across a boundary the user never approved.

Routing is therefore not just choosing a model name. It is deciding which execution paths satisfy the application's contract.

The useful starting point is a simple, measurable routing policy whose behavior does not depend on one provider's feature set.

Define the Contract Before the Router

Start with the behavior the application owes its user. For a support-ticket extractor, the contract might be:

from dataclasses import dataclass
from typing import Literal


@dataclass(frozen=True)
class RoutingContract:
    required_capabilities: frozenset[str]
    allowed_regions: frozenset[str]
    max_latency_ms: int
    minimum_quality: float
    max_cost_per_request: float
    fallback_on_refusal: bool = False


TICKET_CONTRACT = RoutingContract(
    required_capabilities=frozenset({"json_schema", "tool_calling"}),
    allowed_regions=frozenset({"australia", "us"}),
    max_latency_ms=8_000,
    minimum_quality=0.92,
    max_cost_per_request=0.03,
)

The exact thresholds depend on the task. What matters is that capability, policy, quality, latency, and cost are explicit. A fallback is eligible only if it satisfies the same non-negotiable constraints.

Create a dated capability matrix from official provider documentation. Do not infer support because two APIs use similar field names.

Classify Failures Before Falling Back

Not every failure should take the same route.

• Transport failures: connection reset or timeout may permit a retry or fallback.
• Rate limits: respect provider guidance and the shared request deadline.
• Unsupported capability: route before sending the request.
• Invalid structured output: retry only if the remaining budget and policy allow it.
• Safety refusal: do not use another provider to bypass a refusal policy.
• Partial stream: discard or clearly terminate it before starting another response.

def choose_action(error: str, remaining_ms: int) -> str:
    if error in {"policy_refusal", "disallowed_region"}:
        return "stop"
    if error == "unsupported_capability":
        return "route"
    if error in {"timeout", "rate_limit", "invalid_output"} and remaining_ms > 2_000:
        return "fallback"
    return "fail_visible"

Provider SDKs may already retry requests. If the application adds three retries around an SDK that also retries three times, one user action can expand into many calls. Use one end-to-end deadline and record every attempt.

Start With a Deterministic Policy

A learned router can be useful, but it is not the necessary starting point. RouteLLM showed that preference data can train routers to balance stronger and weaker models. An application still needs a clear baseline to determine whether a learned policy helps.

def select_model(request, candidates):
    eligible = [
        model
        for model in candidates
        if request.capabilities <= model.capabilities
        and request.region in model.allowed_regions
        and model.estimated_cost <= request.max_cost
    ]

    if not eligible:
        raise ValueError("No model satisfies the request contract")

    return min(eligible, key=lambda model: model.estimated_latency_ms)

This policy is understandable and testable. A more sophisticated router must beat it on a held-out workload, not just on a generic benchmark.

The router can wrap OpenAI's Python client behind the same application-level result used for every provider. The repository pins openai==2.24.0 so the example remains reproducible. Keep the model ID in configuration rather than hard-coding a choice that will eventually become stale.

import os
from dataclasses import dataclass

from openai import OpenAI


@dataclass(frozen=True)
class ModelResult:
    provider: str
    model: str
    text: str
    input_tokens: int
    output_tokens: int


openai_client = OpenAI()


def call_openai(prompt: str, model: str) -> ModelResult:
    response = openai_client.responses.create(
        model=model,
        input=prompt,
        metadata={"routing_policy": "support-v1"},
    )

    return ModelResult(
        provider="openai",
        model=response.model,
        text=response.output_text,
        input_tokens=response.usage.input_tokens,
        output_tokens=response.usage.output_tokens,
    )


primary_model = os.environ["OPENAI_PRIMARY_MODEL"]
result = call_openai("Classify this support ticket", primary_model)

The adapter records the model that actually served the response and token usage needed for later cost analysis. Add the provider's request identifier and error category in production. Do not catch every exception and return an empty string; the routing layer needs to distinguish timeouts, rate limits, invalid output, and policy failures.

Measure Cost per Accepted Answer

Build a labeled evaluation set that represents the actual task. Include ordinary tickets, ambiguous requests, malformed input, abstention cases, and long examples. Keep development examples separate from held-out evaluation cases.

Compare at least three policies:

1. always use the primary model;
2. always use the lower-cost eligible model;
3. apply routing and failure fallback.

Report schema validity and semantic correctness separately. Valid JSON can contain the wrong customer, category, or urgency. A useful metric is:

cost per accepted answer =
    total cost of successful and failed attempts
    / answers passing the task rubric

Also report latency percentiles, sample counts, failure categories, and the number of fallback attempts. Do not claim savings unless quality remains above the declared floor.

Roll Out With Observable Decisions

Every routed request should record why a model was selected, without logging sensitive prompt content unnecessarily.

{
  "request_id": "req_1842",
  "policy_version": "support-v1",
  "selected_route": "fast-model",
  "reason": "capabilities_met_lower_estimated_latency",
  "attempts": 1,
  "contract_passed": true
}

Begin with shadow evaluation or a small traffic percentage. Define rollback conditions before launch: task accuracy below the floor, policy violations, excessive fallback, or higher cost per accepted answer.

A single provider is often the right architecture. Routing earns its complexity when workloads have meaningful capability differences, reliability requirements, or measurable cost-quality tradeoffs.

Conclusion

Reliable routing preserves an application contract across every eligible path. Define the contract, classify failures, establish a deterministic baseline, and measure complete outcomes—including failed attempts.

The router should make provider choice visible and testable. If it cannot explain why a request moved or show that the move preserved quality, it is not yet a reliability feature.

References

• RouteLLM: Learning to Route LLMs with Preference Data, revised 23 February 2025
• The next evolution of the Agents SDK, OpenAI, 15 April 2026
• Demystifying evals for AI agents, Anthropic, 9 January 2026