同一编程任务：9个模型都成功完成，但最贵的比最便宜的贵出32倍
这个被称为RRLab Bench的项目是RRLab成立后的第一个项目。做这件事的原因其实很朴素：过去半年我用了不少模型。有一个感受越来越强烈：很多模型发布的时候都说自己在某某排行榜上拿了多少分。但实际用起来，总觉得不是那么回事请。那么那些排行榜到底是在说什么呢？我们用户最终应该相信谁？
RRLab Bench is the first project since RRLab's founding. The motivation was simple: over the past half year, I've used many models, and one feeling grew stronger — models launch claiming high scores on various leaderboards, but actual experience rarely matches. What are those leaderboards really saying? Who should we users trust?
实际排名只是体现这个模型知道哪些方面的知识，能完成怎样的事情。但这个可能和你使用下来的感受不一定有关系。排名高的用着不一定快、排名低的用着不一定省…
Rankings only reflect what knowledge a model has and what tasks it can complete. That may have nothing to do with your actual experience. High-ranked doesn't mean fast; low-ranked doesn't mean cheap...
于是我想了下：能否给模型在同等思考深度的情况下，在完成同一个任务时看看各自的差异有哪些？！我模拟了三个在真实开发里会遇到的场景：
So I thought: what if we give models the same task, at the same thinking depth, and see the differences? I simulated three real-world dev scenarios:
- 改一个API的方法签名，三个调用点会跟着挂
- 修一个日期解析的bug，依赖旧行为的模块会崩
- 加一个输入校验，边界数据会不通过
- Change an API method signature — three call sites break
- Fix a date-parsing bug — modules depending on old behavior crash
- Add input validation — edge-case data fails
每个模型跑这三个场景，每个场景重复3次。不是让它回答选择题，是让它真的在沙箱里改代码，跑测试，出结果。
Each model runs all three scenarios, 3 repetitions each. Not multiple-choice — they actually modify code in a sandbox, run tests, produce results.
第一轮我们审计了9个模型配置。所有模型都用最高的思考深度。
Round one: we audited 9 model configurations, all at maximum thinking depth.
数据
The data:
先说不意外的部分：9个模型全部完成了任务，FRR全是0%。换句话说，所有9个模型都完成了任务，谁都没把别的东西搞崩。这表明：在"会不会做"这件事上，这9个模型没有区别。
The unsurprising part: all 9 models completed the tasks, FRR was 0% across the board. In other words, every model got it done, nobody broke anything else. This shows: in terms of "can they do it," these 9 models are indistinguishable.
但换个角度区分度就出来了：
But from another angle, differentiation emerges:
| 模型 | 回合 | 耗时 | Token | 费用/次 | 综合评分 |
|---|---|---|---|---|---|
| Grok 4.5 | 6 | 23s | 19,000 | ¥0.31 | 0.772 |
| DS Flash | 14 | 39s | 71,000 | ¥0.07 | 0.615 |
| DS Pro | 12 | 49s | 53,000 | ¥0.17 | 0.447 |
| MiniMax M3 | 12 | 72s | 59,000 | ¥0.15 | 0.410 |
| GLM 5.2 | 10 | 84s | 34,000 | ¥0.30 | 0.387 |
| Opus 4.8 | 7 | 81s | 35,000 | ¥1.98 | 0.377 |
| Kimi K3 | 10 | 124s | 45,000 | ¥1.17 | 0.280 |
| Opus 5 | 21 | 226s | 228,000 | ¥9.94 | 0.106 |
Model Rounds Time Tokens Cost/run Score Grok 4.5 6 23s 19K ¥0.31 0.772 DS Flash 14 39s 71K ¥0.07 0.615 DS Pro 12 49s 53K ¥0.17 0.447 MiniMax M3 12 72s 59K ¥0.15 0.410 GLM 5.2 10 84s 34K ¥0.30 0.387 Opus 4.8 7 81s 35K ¥1.98 0.377 Kimi K3 10 124s 45K ¥1.17 0.280 Opus 5 21 226s 228K ¥9.94 0.106
综合评分是费用 30%、耗时 30%、回合数 20%、Token 20% 加权算出来的。不是"谁最聪明"，是"谁最便宜地用同样的质量把事办了"。
Composite score: cost 30%, time 30%, rounds 20%, tokens 20%. Not "who's smartest" — but "who gets it done cheapest at equal quality."
三个反直觉的地方
Three counterintuitive findings:
第一：最快的不一定最贵。
Grok 4.5 一骑绝尘——6 个回合、23 秒、19K Token，一次任务 ¥0.31。9 个模型里三项第一。而最贵的 Opus 5 MAX 花了 ¥9.94，做了同一件事，结果一样。
First: fastest isn't necessarily most expensive. Grok 4.5 dominated — 6 rounds, 23 seconds, 19K tokens, ¥0.31 per task. Three first-place metrics among 9 models. The most expensive, Opus 5 MAX, cost ¥9.94 for the exact same task with the same result.
这两个数字之间差了 32 倍。
That's a 32× difference.
第二：深度思考在 Agent 场景里是个负资产。
Second: deep thinking is a liability in agent scenarios.
Opus 5 在Thinking Level为MAX的情况下给出了21轮回复、226秒时间和超22万的Token输出。这难不成是"人傻钱多"的表现？想想是不是因为Opus 5的思考深度太高了？我调整了Opus 5的Thinking Level为：MEDIUM 档，结果如下：12 回合，77 秒，花费 ¥3.19。MAX 档：21 回合，226 秒，花费 ¥9.94——费用翻了 3 倍，结果没变。
Opus 5 at MAX thinking level: 21 rounds, 226 seconds, over 220K tokens. Is this just "more money than sense"? What if the thinking depth was too high? I lowered Opus 5 to MEDIUM: 12 rounds, 77 seconds, ¥3.19. At MAX: 21 rounds, 226 seconds, ¥9.94 — 3× the cost, same result.
那么Opus 5之前Thinking为MAX的时候，多出来的回合去哪了？全耗在过度自我验证上。反复检查、反复推敲、反复确认一个自己已经做对了的事。在考试里这是加分项，在Agent里每一轮都是用户的时间和钱。
Where did those extra rounds go at MAX? All wasted on excessive self-verification. Repeatedly checking, re-checking, re-confirming something already done right. In exams that's a bonus. In agents, every round is the user's time and money.
第三：排行榜和实际使用是两码事。
Third: leaderboards and real-world usage are two different things.
Kimi K3在好几个标准基准里都是名列前茅的。但在这个测试里，它比DS V4 Flash贵16倍，慢3倍，Token用得更多，回合数还不占优。
Kimi K3 ranks high on many standard benchmarks. But in this test, it's 16× more expensive than DS V4 Flash, 3× slower, uses more tokens, and doesn't win on rounds.
不是 Kimi K3不好——它FRR也是0%，它最终也完成了任务。而是"排行榜高"和"用起来划算"之间隔了一整条河。排行榜测的是模型在标准考试里能拿多少分，RRLabBench测的是你请它干一天活，月底的账单长什么样。
Not that K3 is bad — its FRR was also 0%, it completed the task. But there's a river between "ranks high on leaderboards" and "cost-effective to use." Leaderboards measure exam scores. RRLabBench measures: you hire it for a day's work — what does the month-end bill look like?
从一开始我们就没打算做一个新的"第一名第二名"榜单。
From the start, we never intended to build another "#1 vs #2" ranking.
一方面是觉得没什么意思——排名本身就是一种简化，它把一堆不同维度的数据压成一个数字，反而把最有用的信息弄丢了。我想知道的是，在我需要完成的任务类中哪个模型最合适！
Partly because it's pointless — ranking is inherently reductive, crushing multi-dimensional data into a single number, losing the most useful information. What I want to know: for my type of task, which model is the best fit?
另一方面，RRLabBench 已经开源了：https://github.com/rrlab-tech/rrlab-bench。你可以自己跑自己的模型，跑出来的数据自己看。我们也接受社区提交数据——跑完把结果通过 GitHub Issue 或 PR 发过来，经验证后归档到社区数据集里。每季度汇总一份匿名化的效率对比报告，不做排名、不做评价，只做数据可视化。
Also: RRLabBench is open source at github.com/rrlab-tech/rrlab-bench. Run your own models, see your own data. We accept community submissions — send results via GitHub Issue or PR, verified and archived to the community dataset. Quarterly anonymized efficiency comparison reports: no rankings, no judgment, just data visualization.
我们提供方法论，你决定用哪个模型。
We provide the methodology. You decide which model to use.
下一步
Next steps:
RRLab 后面要做的事情，是从"效率"往"可靠性"走。我们定义了几个新的指标：
RRLab's next phase moves from "efficiency" toward "reliability." We've defined several new metrics:
- DRI（破坏性风险指数）：模型完成任务时，改了不该改的文件的概率
- FDR（首次诊断成功率）：遇到错误时是诊断根因，还是换个方案再试
- 多轮一致性：长对话里能不能保持同一套规则和风格
- DRI (Destructive Risk Index): probability a model modifies files it shouldn't while completing a task
- FDR (First Diagnosis Rate): when hitting an error, does it diagnose the root cause or try a different approach
- Multi-round Consistency: can it maintain the same rules and style across long conversations
这些是现有评测框架完全不碰的东西。也是真正决定"能不能把模型放进生产线"的问题。
These are things existing evaluation frameworks never touch. And they're what truly determines "can this model go into production."
这些还在开发中，后面有了数据会再写。
These are still in development — I'll write more when we have data.
这几天我在考虑另外一个事情，如果谁有兴趣可以加我一起商讨。"现有模型的注意力是否真的能覆盖 1M 的上下文？"
I've been pondering another question lately — if anyone's interested, let's discuss: "Can existing models' attention truly cover 1M context windows?"
RRLab-Bench 代码：https://github.com/rrlab-tech/rrlab-bench
联系我：rodneyrui@gmail.com
RRLab-Bench: github.com/rrlab-tech/rrlab-bench
Contact: rodneyrui@gmail.com