If you serve open-weight models on your own GPUs, you pay for GPU time, and that time depends on the GPU type, the request load and the response length. CRE-Router (Cluster, Route, Escalate) is a router, which decides which model in a pool answers each query, and it prices each model by the time it takes on the hardware that serves it. On three benchmarks, it comes close to the accuracy of the strong model it escalates to, at a lower cost per token.
Model routing sends each coming query to the one model in a pool that best balances cost and accuracy. Cascading lets a cheaper model answer first and passes the query to a stronger one when its answer is not good enough [1]. Most methods price a model from an API price list, a token count or a recorded run time, not from the hardware that serves it.
Stage 1 assigns each query to a cluster and routes it to the model chosen for that cluster under the cost budget. Stage 2 lets a quality estimator accept the answer or escalate the query to the strong model.
How CRE-Router Works
CRE-Router combines routing and cascading in two stages.
Stage 1 groups the training queries into clusters, using k-means on their embeddings. For each cluster, it scores every model by its error on that cluster and its cost, measured either as the time per output token (TPOT) or the end-to-end latency per request (E2EL). The operator sets a budget on that cost, and CRE-Router selects the most accurate routing within it. At inference, a new query is assigned to its nearest cluster and sent to that cluster’s model, so a query from a hard cluster goes straight to a strong model and does not pay for a weak answer first.
Stage 2 applies where a cluster is routed to a cheaper model. A small classifier, the quality estimator (a fine-tuned ModernBERT-base), reads each answer and either accepts it or escalates the query to the strong model. It catches the errors that routing cannot predict from the query alone.
Both stages are fitted from task-correctness labels, which standard benchmark evaluation already provides, so no extra annotation is needed.
Results
We evaluated CRE-Router on two telecom benchmarks, TeleMath and TeleQnA, and on AIME, with pools of two to nine open-weight models served with vLLM on one and two A100 GPUs [2]. CRE-Router is more accurate than the cheaper models in its pool and close in accuracy to the strong model it escalates to, at a lower cost per token. Given the same cost per token, it is more than 12 points more accurate than FrugalGPT [3] on TeleMath, and significantly more accurate on TeleQnA.
Accuracy against cost on TeleMath on one A100, per output token (left) and per request (right, log scale). Higher and further left is better; the line joins the systems that no other system beats on both. CRE-T20 is CRE-Router fitted under a TPOT budget of 20 ms. CRE-Router lies on the frontier under both cost terms.
Three Findings About Cost
Cost term. The cost term decides which models serve the workload. TPOT does not see how long an answer is, so a TPOT budget admits models that reason at length. On TeleMath, a TPOT budget sends two of the four clusters to Qwen3-30B with reasoning, while an E2EL budget sends them to Gemma4-26B without reasoning, at about the same cost per token and a fraction of the wait per request.
Every path a TeleMath request can take through CRE-Router, with the share of requests that take it and their mean E2EL. Under the TPOT budget, over 70% of requests go to reasoning models and wait more than three minutes on average; under the E2EL budget, no path averages more than 40 seconds.
Per-cluster cost. A model can be cheap on one kind of query and expensive on another. Pricing each model per cluster, rather than with one cost for the whole workload, shortens the time per request by 20 to 22% on TeleMath, at comparable accuracy.
Hardware. A second GPU speeds models up unevenly, so at the same budget, one and two A100s select different routings on all three benchmarks. Refitting needs new cost measurements, but no new labels.
Routing pays when no single model is both accurate enough and cheap enough for the budget. If one model already is, serve that model.
CRE-Router at GitHub
The code, configurations, data and model outputs are on GitHub. The training statistics measured in the paper ship with the repository, so you can fit a routing without a GPU.
Fit a routing from the stored statistics:
git clone https://github.com/ymoslem/CRE-Router.git
cd CRE-Router
pip install -e .
cre fit --stats configs/telemath_stats_1xA100_Sep2026.json --budget 25
It prints the routing that a TPOT budget of 25 ms selects on TeleMath, and the quality estimators Stage 2 needs.
To use the package without cloning the repository, pip install "cre-router[full]" installs everything needed for serving, including vLLM, so it needs a GPU machine. The README covers the full pipeline, from clustering and evaluation to serving with vLLM.
References
- Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey (Moslem & Kelleher, TMLR 2026)
- Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving (Moslem et al., arXiv 2026)
- FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (Chen et al., TMLR 2024)
Acknowledgements
This work was funded by Huawei Ireland through a research collaboration with the ADAPT Centre at Trinity College Dublin.