July 29, 2026 · Tokenless · 5 min read
Building Tokenless: mixing models for cheap frontier performance
Our fan-out router watches candidate models think and commits to the one that's clearly on track — matching frontier performance at a fraction of the cost.
Most AI inference is wasted.
All day, agents are running and doing mundane tasks, such as searching through files or writing commit messages. To make sure the rare complex requests get handled well, agents get hooked up to super powerful models for every single call they make. This is a huge waste of both money and energy, especially as open source models close the gap with frontier models at a fraction of the cost.
Here at Tokenless, we’re solving this problem: we automatically send each request to the right model, and the results speak for themselves; we reach SOTA on τ³-Banking by using a mix of GPT-5.6, Claude Opus 5, Gemini 3.1 Flash-Lite, and Deepseek V4 Pro.
τ³-Bankingsolve rate and cost
As it turns out, model routing is a surprisingly hard problem. Try it yourself: choose which model succeeds on the below prompts for the cheapest price:
Author affiliations on nature.comBrowserBench
Locate articles related to quantum computing on nature.com and list the affiliations of the first three authors found.
Methodology
The highest-leverage technique we found was a fan-out router. This means we go through different models that we think will solve your task and watch them work. That is, we kick off response generation for multiple candidate models, and assign a “confidence score” to each model that updates live as each model streams its output. As soon as a model is clearly on track, we commit to its output, cutting off the other candidates, which avoids paying for unneeded outputs.
This approach solves several problems that prompt-based or heuristic-based routing struggle with:
Research-backed routing decisions: there is a large amount of “model astrology” when it comes to manual routing decisions. How many times have you been told to use Fable to plan and Sol to execute? These heuristics are usually based on some preconceived notion of how to delegate work between models, and operate at a very coarse level. We trained our own foundation models to accurately predict model performance on a per-turn basis instead.
Situation-aware routing: prompt-based routers usually miss the broader picture. For example “explain how string parsing works” might be a trivial task in one repo, while being quite complex in another. By letting the agents run and think, we make our judgements based on what agents have found, instead of solely on what they have been told.
“Double-dipping” on frontier reasoning: figuring out the difficulty of a problem is really hard (see Rice’s Theorem). Prompt-based routing often needs a very smart model to weigh in on which model to route to. Instead, if you watch the models work, you can use the computation of the frontier models themselves to inform your decision.
Cheap and fast routing: the “router” used to route agent turns is usually a cheap LLM or larger model that intercepts your prompts and wraps it in some system prompt to output an estimate. The problem with this is that LLMs are slow, costly, and notoriously bad at estimating confidence (see here and here). Our lightweight routing model adds overhead on the order of ≤ 10ms and costs thousandths of a cent.
Training the “Routing” Layer
We first trained a foundation model to calculate the confidence score of each generation using a corpus of partial generations. This model was made aware of the LLM and reasoning effort that is being used to continue from the current state.
Next, we trained a routing rule that takes these confidence scores into account to choose the best model to route to. We tried several approaches: heuristics, algorithms with provable guarantees, even training a neural network using reinforcement learning. What we found actually worked was an evolutionary algorithm instantiated as follows.
Given a large dataset of tasks, a batch size of B, a number of candidates per generation C, a number of elites per generation E, environment change rate R% and number of generations G:
- Take some subset of B tasks as the environment
- Evaluate C random routing rules on the environment; the top E are elites by fitness (a function of cost, latency, and accuracy)
- Mutate the E elites to create a new set of C total candidates
- Replace R% of the environment with new tasks
- Repeat G times, occasionally injecting new random routing rules for diversity
Step 4 meant that the environment was changing alongside the routing candidates. This tended to make sure the candidates that survived were robustly fit across many different data, while the batch size B stayed small and made sure iteration happens quickly and cheaply.
What about the cache?
One of the most common questions we get asked is: doesn’t model routing lose its cost savings immediately by losing the benefits of prompt caching? In particular, the cache state is kept on the provider’s infrastructure, and is incompatible between different models.
Our answer: no. The state of each provider’s cache is one of several inputs that inform whether or not to switch models. Since we route each request, we can maintain a local shadow of the cache state of each provider, which means our router always knows how much of a cost switching models incurs, and only switches when it’s actually worth it.
Something that a lot of routers miss is understanding that caching is provider- and possibly even time-dependent. The same model from different providers can have caching enabled or disabled. For instance, if we’re hitting OpenRouter, a cache-enabled provider might be overloaded and your requests might be sent to a cache-disabled one. We instrument the responses coming back from the providers so we have the most accurate cache status available to make routing decisions.
Another open research direction is making the KV cache partially transferrable between different open-weights models that are similar enough sizes. There is some evidence this is possible, see here.
Experimental Results
We tested our flagship router, Tokenless PRO mode, on three popular agentic benchmarks, both in terms of cost and performance. Currently, we use a curated combination of GPT-5.6, Claude Opus 5, Gemini 3.1 Flash-Lite, and Deepseek V4 Pro, and we’re actively working on improving performance and cost in the coming weeks by adding new models and reasoning efforts.
solve rate
avg $/task
Our Take
Lightweight open source models are rapidly closing the gap with frontier models, and while each frontier model release gives us a new answer to “what can models do?,” open-source models answer an equally important question: “what can models do cheaply?” The multi-model future is here, and we think that the benefits of switching models should be available for anyone to use and build on.
We welcome you to try out Tokenless for yourself; sign up, build something or route some work through it, and tell us how much it saved you!