Model routing sends different requests to different models instead of using one model for every task. A lightweight model might classify a simple message, while a stronger model handles complex reasoning or difficult generation. Routing rules can be based on fixed task types, measured confidence, cost budgets or fallback behavior.
An example setup sends short FAQ questions to a small, fast model and long contract questions to a stronger one, with a fallback if the first model reports low confidence or the provider is down. Log which route each request took, its cost and its outcome, so you can see whether routing saves money without lowering quality. Use LLM evaluation and AI agent observability to check.
Test the router as well as each model. A misrouted request can reduce quality or erase expected savings, and fallbacks should not silently exceed budget or privacy requirements. Related: Small Language Model, Inference Latency.