Within six points of Opus 5, at 1/40th the cost.
We trained a 27B open base model on τ³-bench banking, the standard benchmark for customer-experience agents and tool calling. It scores 0.483 against Claude Opus 5's 0.543, and costs $0.55 per solved task against Opus 5's $23.22.
Cost
1/40
of Opus 5’s cost per solved task
Quality
+93%
over the 27B base model
Accuracy
89%
of Opus 5’s pass^1
- Benchmark
- τ³-bench banking
- Split
- 68 train / 29 held out
- Trials
- 4 per task
- Rollouts
- 116 per arm
- Trained
- 27B open base
The result
Training took the 27B from 0.250 to 0.483, and cost per solved task moved one cent, from $0.54 to $0.55. The run costs more, and it solves proportionally more, so the two cancel.
Against the frontier that is four fifths of the distance closed, for 47.8× less total spend: $30.60 against $1,462.64.
Quality per dollar tells you less than it looks like it does. On that measure the untrained base ties the trained model, 42.7× against 42.5×, because it is cheaper in the same proportion that it is worse. Prime Agent tops the column at 507.8× while solving a third of the set. Training is not justified by the ratio, it is justified by reaching a level the base model cannot reach at any price.
| Arm | pass^1 | pass^4 | Cost | $/point |
|---|---|---|---|---|
| Claude Opus 5 frontier ceiling | 0.543 | 0.345 | $1,462.64 | 1.0× |
| Protégé 27B trained | 0.483 | 0.241 | $30.60 | 42.5× |
| Prime Agent alternative harness | 0.328 | 0.143 | $1.74 | 507.8× |
| 27B base untrained | 0.250 | 0.138 | $15.76 | 42.7× |
Protégé 27B is the trained model. The 27B base is the same model untrained. Prime Agent is a different agent harness over the base weights. Opus 5 ran at max_steps=200, Protégé 27B at 100, Prime Agent and the base at 50. Solves are pass^1 × 116 rollouts, rounded to whole tasks: 63, 56, 38 and 29. $/point is quality per dollar, pass^1 ÷ cost, normalised to Opus 5 at 1.0×. Opus 5's cost comes from the trace's own cost field rather than tokens priced by hand, and this run produced no standard deviations, so none are printed.
Where the money goes
The same 116 rollouts per arm, counted in tokens and calls rather than dollars.
Prime Agent is 16.7× leaner than Opus 5 on context per solve and lands 15.5 points lower. Optimizing for the cheapest rollout finds a real minimum, on the wrong axis.
Tool calls per run barely move: 30.9 to 38.5 across every arm. The customer request decides how many bank API calls are needed, not the model answering it. Better models do not use fewer tools, they use them more successfully.
Per solved task the trained 27B is near parity with the frontier: 4.7% more LLM calls, 17% more tool calls, 3.7% fewer input tokens. It is not doing more work per win, it simply wins less often.
| Arm | Context / solve | Tools / solve | Capped |
|---|---|---|---|
| Claude Opus 5 | 3,915,761 | 68.1 | 0% |
| Protégé 27B | 3,772,765 | 79.7 | 6% |
| Prime Agent | 234,453 | 99.8 | 31% |
| 27B base | 3,702,705 | 123.6 | 55% |
Context per solve is input tokens ÷ solves. Tools per run, 37.0, 38.5, 32.7 and 30.9, is tool calls ÷ 116 rollouts. Capped is the share of rollouts that stopped before finishing; Opus 5 ran with a larger step allowance than the other arms and never reached it.
How it was trained
Production traces from Qwen-3.8-Max, Kimi K3 and Claude Opus 5 distil into the 27B through a black-box on-policy distillation approach built for long-horizon tasks, where one episode runs to dozens of tool calls and the reward only arrives at the end.
Black-box because it needs no weights, no logits and no gradients from any teacher, only their trajectories. That is what makes a frontier API usable as a teacher for a model you end up owning.
Evaluation is closed-loop against the same tool surface the agent meets in production, on the 29 tasks held out of training. Nothing on this page is scored on a trajectory the model was trained on.
Interested?
We are in private preview, working with a small group of design partners at a time: capture the traces, write the eval, and ship the first replacement model alongside the people who already know the work.
The fit is a production workflow where cost or latency is starting to hurt, and the people who know what good looks like are not on an ML team.
Book a callSend two months of inference spend and what currently counts as a correct answer. We do not need your data to start.
Prefer email? contact@protege.sh