The score you’re quoted isn’t the score you get

Every reasoning model now has a dial on it.

It goes by different names depending on the lab. Reasoning effort, thinking budget, extended thinking, a token allowance for the part of the answer you never see.1 The idea is the same everywhere: let the model spend more compute on a hard problem and it does better on that problem, at a price.

Two things follow from that dial, and between them they make most published model scores unusable as a buying signal. The first is that a score is meaningless without the setting it was produced at. The second is that labs benchmark at the top of the dial and ship you something lower by default.

Nobody is hiding this. It is in the documentation. It is just never in the comparison.

The gap, plainly

A lab publishes a headline score. That score was produced at maximum effort, because of course it was; that is the configuration where the model looks best and the announcement exists to announce a capability.

Then you call the API without specifying an effort, because most people do not, and you get the default. OpenAI’s default sits at medium. Grok 4.3 defaults to low.2 Some models default to no reasoning at all unless you turn it on.

You are now running a different thing than the one that was measured. Your results are worse than the chart and you have no way of knowing by how much, because the chart does not have a row for your configuration.

And the price moves in the opposite direction to the score. Reasoning tokens bill as output tokens on every vendor we have checked. So the configuration that produces the published number also produces the largest bill, and the relationship between the two is not something any lab publishes as a curve.

One score and one price per model is structurally wrong. It is the most common way a comparison table misleads, and almost every leaderboard does it.

Why this is hard to measure honestly

I would like to tell you we have a clean multiplier: raise the effort one notch, get this much more accuracy for this much more money. We have not, and I do not think anyone has. Three reasons, and they are all worth knowing if you are trying to do this yourself.

• No vendor publishes what the tiers actually mean. "Medium" is not a token count you can look up. It is a policy inside the model, and the policy changes between versions without the label changing. Two models both set to high are not doing a comparable amount of work.
• The gain is not uniform across tasks. Turning the dial up helps most on problems with a long chain of dependent steps, and helps very little on recall or on straightforward drafting. A single average across a benchmark suite hides that completely.
• The cost is variable, not fixed. You are billed for the reasoning tokens the model actually emits, and that varies per prompt. A price per tier is a range, not a figure, and presenting it as a figure would be the same category of error this piece is complaining about.

So what we show is a ladder rather than a coefficient: for each model, each effort tier the provider exposes, the price at that tier, and a marker for two specific things. Which tier the vendor benchmarked at. Which tier you get if you do not ask for one. Where those two markers sit on different rungs, that is the gap, and you can see it without us having to reduce it to a number we cannot defend.

The subscription version, which is worse

If you are on a subscription rather than an API key, you cannot see the dial at all. You do not know which effort tier your plan gives you. You do not know whether it changes under load, or by time of day, or once you have used some amount of an allowance nobody has told you the size of.

The dominant pattern is that quotas are compute-weighted and deliberately unpublished. Anthropic describes Claude Pro’s allowance as a multiple of the free tier per five-hour session. Google describes a tier the same way, against a baseline. xAI describes a shared weekly pool. Several others use a phrase like "significantly higher" and leave it there.3

None of those is a number. All of them are, as far as I can tell, honest descriptions of systems that genuinely do move with demand. The vendors are not being evasive so much as declining to promise something they would then have to hold to. But the effect on you is the same either way: you cannot work out what you are getting, so you cannot work out whether you are getting value.

What to actually do about it

• Set the effort explicitly, every time. Do not inherit a default you did not choose. If you are going to be surprised by a bill or by a result, be surprised by one you selected.
• Match the tier to the shape of the task, not to its importance. Long dependent reasoning, turn it up. Drafting, extraction, classification, summarising, turn it down; you are usually paying for thinking that changes nothing. The instinct to reach for maximum effort because the task matters is expensive and mostly ineffective.
• Measure on your own work. Take twenty real tasks from your actual backlog, run them at two tiers, and compare. That is a couple of hours, and it is measuring the thing you are going to do rather than the thing someone else measured.
• Read every published score as a ceiling. Not a typical result. Not what you will get. The best configuration, on a set the model may have seen some of, run by people who wanted a good number.

The dial is not a problem in itself; it is a genuinely useful control and I would rather have it than not. The problem is that the industry standardised on advertising the top of it while shipping the middle, and then compared everyone on the top. Once you know that, the charts get easier to read. They just get less impressive.