TokelyTHE SHORT ANSWER
LLM routing chooses which model or provider should handle a request. The most useful strategy minimizes cost only among routes that satisfy your quality, latency and data-handling requirements. A cheap route that cannot produce an acceptable answer is an expensive detour.
What should a router decide first?
Start with eligibility. Does the route support your input format, context length, output schema and tools? Is it permitted to process the customer’s data? Only then rank eligible routes by price or performance.
Separate model selection from supplier selection. Choosing a smaller model changes capability. Choosing another supplier for the same model changes the delivery path. These decisions need different evaluation evidence.
When does cost-based routing work?
Predictable tasks are the easiest starting point. Classification, field extraction and short rewriting can often be assigned by task type before a model runs. Use a stronger candidate for difficult cases only if the weaker candidate fails a measurable validation step.
Consider an illustrative workload of 100,000 requests. If a premium route costs $0.01 per request, using it for everything costs $1,000. Sending 80% directly to a $0.002 route and 20% directly to the premium route costs $360, a 64% reduction before routing overhead.
A cascade has different economics. Running the cheap model on every request, then escalating 20%, costs $400. Do not report the direct-routing saving for a system that actually pays for both attempts.
How do you route for latency?
Use time to first token for conversational responsiveness and total completion time for finished deliverables. Measure both at representative prompt lengths and concurrency. A provider can start quickly but take longer to finish a large response.
For a strict deadline, prefer routes whose recent tail latency fits the deadline. A low average is insufficient if one in twenty requests stalls. Include queue time in the measurement because the user experiences it too.
Can the model choose its own confidence threshold?
Self-reported confidence alone is a weak routing signal. Prefer schema validation, task-specific checks, retrieval coverage and measured performance on labeled examples. For ambiguous outputs, escalate or send the task for review according to its importance.
Keep a small sample of ordinary traffic under evaluation. A router can appear cheaper while gradually sending unsuitable requests to its least expensive candidate.
What routing does Tokely expose?
Tokely documents Balanced, Lowest cost, Fastest, Most reliable and Fixed supplier-routing modes. Adaptive selection depends on eligible routes and sufficient pricing or performance information. Lowest cost uses a normalized token mix, so it is not a promise of the minimum bill for every prompt. See the routing contract.
Start with one task family and two qualified candidates from Tokely’s models. Compare cost per accepted result before expanding the policy to the rest of your application.