Terminal-Bench 2.1: Routing at the Cost-Quality Frontier

Why Terminal-Bench is a useful transfer test

Terminal-Bench 2.1 asks an agent to operate a terminal, inspect an unfamiliar environment, make a correct change, and pass an automated verifier. The benchmark contains 89 tasks spanning software engineering, systems work, data processing, and scientific computing. It tests whether a routing policy developed on repository-level coding work transfers to a different agent environment.

Mentlio Route solved 82.92% of tasks in expectation at $0.933 per task. Always Fable solved 83.82% at $1.242 per task. The routed policy therefore retained 98.93% of Always Fable's solved-task rate and reduced average model cost by 24.90%.

How the benchmark was measured

The comparison uses the same 89 Terminal-Bench tasks and the same model results for every policy. Sonnet 5, Opus 4.8, and Fable 5 use published Claude Code results; Haiku 4.5 uses a verified Terminus 2 evaluation.

The replay changes only the model choice. For each task, it applies the selected model's result and average cost. This holds the underlying model performance constant and isolates the effect of routing.

How Mentlio routes the work

Before each model call, Mentlio estimates prompt difficulty and how certain it is about the task type. Clearer, lower-risk work can move to a less expensive model; harder or less certain work stays on a stronger one.

Across the 89 tasks, Mentlio selected Haiku once, Fable 34 times, and Opus 54 times.

Results

Mentlio solved 82.92% of tasks at $0.933 per task. Always Fable solved 83.82% at $1.242 per task. The routed policy finished 0.90 points behind Fable and reduced average cost by 24.90%. It also finished 4.04 points above Always Opus while costing $0.121 more per task.

Validity, controls, and limitations

GPT-5.5 is shown for context, not as part of the matched comparison. Its public result used a different agent and evaluation protocol. The 24.90% cost reduction compares Mentlio with Always Fable on the shared replay.

This remains a replay benchmark rather than a production trial. Model outputs vary between samples, and real traffic will not match a fixed benchmark. A production rollout still requires shadow evaluation on the customer's own workload and a quality floor agreed in advance.

Mentlio's impact

Mentlio makes the model choice before an agent call and keeps that decision measurable without exporting source code or raw prompts.

In this transfer test, that control retained 98.93% of Always Fable's solved-task rate and removed 24.90% of its average model cost. The same approach supports shadow evaluation, explicit quality floors, and escalation to a stronger tier when needed.