Bar chart cover: low effort 1x versus max effort up to 42x cost on easy tasks


Two pieces of research landed in the last two weeks with the same uncomfortable message for anyone who leaves a coding agent on its highest reasoning setting: most of the money you spend at max effort buys very little. Vals AI published a study on November 3 that ran seven frontier models at their lowest, middle and leaderboard effort settings. A week earlier, on October 25, six researchers posted a paper on arXiv titled What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents. Both have been passed around developer feeds and AI newsletters all week, and the 42x figure from the paper is the line people quote.

This post goes through what the two studies measured, where they agree, and what they do not tell you. Then it shows how to act on it: how effort is set in Claude Code, the Claude API and Codex, and a small script that starts every task at low effort and moves up only when your own tests fail. That last idea comes straight from the arXiv paper, and it is cheap to try on your own repo this weekend.

What Vals AI measured

Vals ran each model at three settings: its lowest, a middle one, and the setting it uses on its public leaderboard, which is usually the highest. The score is the Vals Index, an average over eight benchmarks: Finance Agent 2.1, EMB 1.1, Terminal-Bench 4, Vibe Code Bench, Code Migration 1.1, Legal Research, HLAB 1.1 and Tax Agent Bench. Cost is at list price.

Here is how much of its leaderboard score each model kept at its lowest setting, and what that cost relative to the leaderboard run:

ModelLowest settingScore keptCost vs leaderboardIndex change (points)
Gemini 4 Argonlow99%84%-0.5
DeepSeek V4.1 Flashlow89%50%-6.7
GPT-6 Astralow88%18%-7.5
Grok 4.7low86%16%-8.6
GLM-5.3low81%53%-11.8
Claude Opus 5.5low80%16%-13.7
Muse Spark 1.3minimal77%29%-12.6

The headline from Vals is that four of the seven models kept at least 80% of their score for half the cost or less. Look at the Opus 5.5 row, because it is the model many Claude Code users run: at low effort it kept 80% of its max-effort score for 16% of the cost. That is a big saving, and it is also a 13.7-point drop on the index, the largest in the table. Both facts matter, and which one wins depends on the work.

Three more findings from the post are worth knowing before you change anything:

  • In 22 of 56 model and benchmark pairs, a lower setting matched the leaderboard score at lower cost. In those cases the extra spend bought nothing measurable.
  • No cheaper setting scored significantly higher than the leaderboard setting, and 33 scored significantly lower. Lower effort is cheaper, and on average it is also a bit worse.
  • No model matched its leaderboard index score at its lowest setting. If you want the top score on hard work, you still pay for it.

The benchmarks also differ a lot in how much they reward reasoning. Going from lowest to highest effort raised the median score by 19.2 points on Legal Research, 15.6 on Code Migration 1.1, 12.1 on Terminal-Bench 4 and 10.6 on Vibe Code Bench, but only 3.3 on HLAB and EMB. The three coding benchmarks sit near the top of that list. For coding agents, effort pays off more than it does on many other tasks. It just does not pay off on every task.

Vals also ran a hindsight test: if you picked the best setting per task after seeing the results, the median gain over the leaderboard setting was 6.0 points at a median 64% of the cost. For Opus 5.5 that meant 60.6 to 66.6 at half the cost. Vals calls this an upper bound, because a real system has to choose before it knows the answer. That gap between "best in hindsight" and "fixed at max" is exactly what the arXiv paper tries to close.

What the arXiv paper adds

The paper by Kaiser Sun, Yongchan Kwon, Zain Hasan, Shang Zhu, Kaitlyn Zhou and Federico Bianchi tested proprietary and open-weight models on four benchmarks, from single-call reasoning to long agentic tasks. Its abstract makes four claims:

  • Higher effort raises overall accuracy, but it often fails on tasks that a lower effort solved.
  • On easy tasks that every level can solve, the highest effort costs up to 42 times as much as the lowest.
  • Higher effort changes how agents behave. They verify their own work more and search more, so runs get longer without always ending better.
  • Starting at the lowest effort and escalating only after a verified failure matches or beats a fixed top setting on every model and benchmark they tested, and it costs less for most proprietary models.

The authors call reasoning effort a "credence good": you pick the level before you know whether the task needed it, and even a correct answer does not tell you whether the extra thinking was necessary. If you have ever watched an agent at max effort re-read the same five files, run the test suite three times and then write a long summary of a one-line fix, the third finding will sound familiar.

This post draws on the paper's abstract and the coverage of it, not its full results tables, so it quotes no per-model numbers from the paper. The paper names its benchmarks and models in the body; check them before you lean on it for a specific model.

What the studies do not tell you

Before you turn every setting down, keep the limits in view. Vals ran each setting on about half the tasks per benchmark, with mostly one run per setting, so each score is noisier than a full leaderboard run. Most leaderboard runs came from earlier months than the lower-effort runs, so model updates could move the comparison. A "no significant difference" result does not prove two settings are equal; the test can miss small real gaps.

Neither study ran on your repository. Benchmarks are made of self-contained tasks with clear pass conditions. Real iOS work has fuzzy specs, a signing setup that breaks for reasons unrelated to code, and tests that may not exist yet. Low effort can look fine on a benchmark and still skip the file that only fails on a device build.

Finally, "cost" here means API list price per token. If you use Claude Code on a Pro or Max subscription, lower effort does not lower a bill. It stretches your usage limits further, which matters if you hit them every week, but it is a different trade.

How effort is set in the tools you use

Claude Code

Claude Code reads effort in this order, and the first one that applies wins: an explicit choice (the CLAUDE_CODE_EFFORT_LEVEL environment variable, the --effort flag, or /effort in the session), then your settings (a per-model level saved under modelSettings, or the effortLevel key), then the model's default.

# in a session
/effort status      # show the level in effect
/effort low         # set it directly
/effort             # open the slider
/effort auto        # clear your saved level for this model

# for one run
claude --effort high

# in ~/.claude/settings.json (low, medium, high or xhigh; max is not accepted here)
{
  "effortLevel": "medium"
}

The defaults are worth checking, because many people assume they run at high. In Claude Code's model docs, Opus 5.5, Sonnet 5.5 and Haiku 5.5 default to medium. Opus 5, Sonnet 5 and Opus 4.8 default to high, Fable 5.1 and Fable 5 to high, and Opus 4.7 to xhigh. If you set a level a model does not support, Claude Code uses the highest supported level below it; xhigh runs as high on Opus 4.6, for example. Choosing max in the slider applies to the current session only, unless you set it through the environment variable.

For a single hard turn you do not need to change the session at all. Put the word ultrathink in the prompt and Claude Code asks for deeper reasoning on that turn only. Other phrases such as "think hard" are treated as normal text.

Claude's own guidance points the same way as the research. Anthropic's post on choosing a model and effort level in Claude Code says to use the model's default effort for most tasks, raise it when Claude cuts corners (skips a file, does not run tests, abandons a refactor), and check whether the prompt or project context is the real problem before reaching for more effort.

The Claude API

In the Messages API, effort sits under output_config. The allowed values are low, medium, high, xhigh and max, though not every model supports every level. Sending the model's default is the same as leaving the field out.

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=4096,
    messages=[{"role": "user", "content": "Review this diff for retain cycles: ..."}],
    output_config={"effort": "low"},
)

Three details from Anthropic's effort documentation affect cost directly. Effort applies to all output tokens, including tool calls and thinking, and it is a behavioral signal, not a hard token budget; max_tokens is the hard limit. Lower effort usually means fewer and shorter tool calls. And changing the top-level effort between requests invalidates the prompt cache, so if you escalate inside one long conversation, the first request at the new level pays full price for the cached prefix again. Anthropic documents a beta for per-message effort that keeps the cache on supported models; read the page before you build on it.

Codex

In Codex, the key is model_reasoning_effort in config.toml. OpenAI's config reference lists values such as low, medium, high, xhigh, max and ultra, and says the available levels depend on the model and client.

# ~/.codex/config.toml
model_reasoning_effort = "medium"

The escalation ladder: start low, prove it, then move up

The paper's strategy is simple enough to script: run the task at the lowest effort, check the result with something you trust, and only rerun at a higher level if the check fails. The important word is "verified". An escalation rule based on the model saying it is unsure would bring back the guessing problem. Your test suite, a build, or a linter gives a yes or no that does not depend on the model's opinion of its own work.

The script below does that with Claude Code in headless mode. It refuses to start on a dirty git tree, runs claude -p at each level in turn, runs your check command, and resets the tree before the next level if the check fails. Every attempt goes into a CSV in your home folder, with the cost estimate Claude Code reports in its JSON output (total_cost_usd).

#!/usr/bin/env python3
"""Run one Claude Code task at the lowest effort first.
Escalate only when your own check fails. Log every attempt."""
import argparse, csv, json, subprocess, sys, time
from pathlib import Path

def sh(cmd, **kw):
    return subprocess.run(cmd, shell=isinstance(cmd, str),
                          capture_output=True, text=True, **kw)

def tree_is_clean():
    return sh(["git", "status", "--porcelain"]).stdout.strip() == ""

def reset_tree():
    sh(["git", "reset", "--hard", "-q"])
    sh(["git", "clean", "-fdq"])

def main():
    p = argparse.ArgumentParser()
    p.add_argument("--task", required=True)
    p.add_argument("--verify", required=True,
                   help='e.g. "swift test" or an xcodebuild test command')
    p.add_argument("--levels", default="low,medium,high,xhigh")
    p.add_argument("--log", default=str(Path.home() / "effort_log.csv"))
    a = p.parse_args()

    if not tree_is_clean():
        sys.exit("Commit or stash your changes first; the ladder resets the tree.")

    log = Path(a.log)
    new = not log.exists()
    with log.open("a", newline="") as f:
        w = csv.writer(f)
        if new:
            w.writerow(["time", "task", "effort", "cost_usd", "seconds", "verify"])
        for level in a.levels.split(","):
            t0 = time.time()
            run = sh(["claude", "-p", a.task, "--effort", level,
                      "--permission-mode", "acceptEdits",
                      "--output-format", "json"])
            try:
                cost = json.loads(run.stdout).get("total_cost_usd")
            except json.JSONDecodeError:
                cost = None
            ok = sh(a.verify).returncode == 0
            secs = round(time.time() - t0, 1)
            w.writerow([time.strftime("%Y-%m-%d %H:%M"), a.task[:60],
                        level, cost, secs, "pass" if ok else "fail"])
            f.flush()
            print(f"{level:>7}: verify {'PASS' if ok else 'fail'}, "
                  f"cost {cost}, {secs}s")
            if ok:
                print(f"Stopped at {level}. Review the diff before you commit.")
                return 0
            reset_tree()
    print("Every level failed. The task or the check needs a human look.")
    return 1

if __name__ == "__main__":
    sys.exit(main())

Run it from the root of a Swift package or app repo:

python3 effort_ladder.py \
  --task "Fix the failing DateFormatter tests in Sources/Formatting without changing the tests" \
  --verify "swift test"

# for an app target
python3 effort_ladder.py \
  --task "Make the login screen's error label support Dynamic Type" \
  --verify "xcodebuild test -scheme MyApp -destination 'platform=iOS Simulator,name=iPhone 17' -quiet"

A few notes on how it behaves, because the details decide whether it is safe:

  • It needs a clean tree. A failed level is wiped with git reset --hard and git clean -fd, which deletes untracked files that are not ignored. That is why the script stops if git status shows anything, and why the log lives in your home folder instead of the repo: during testing, a version that kept the log in the repo had it deleted by the reset.
  • Claude only edits files. With --permission-mode acceptEdits, Claude Code writes files and runs common filesystem commands without prompting, but other shell commands still need an allow rule. The script runs your check itself, so Claude cannot decide its own result by running a different command.
  • The check has to mean something. If your tests pass on an empty implementation, the ladder will happily stop at low. Write the failing test first, or add the test to the task and keep it out of reach by saying the tests must not change.
  • Costs are estimates. Claude Code's docs say total_cost_usd is a client-side estimate that can differ from your bill. On a subscription it is still useful as a relative number between levels.

The script was tested with a stand-in claude command that fails at low and passes at medium, plus a run where every level fails. It stopped at the right level, wrote the log, and left the tree clean after the failed run. It has not been run against a live Claude account, so treat your first real run as a test and read the diff.

Measure your own repo for a week

The studies give you a reason to try lower settings. Your own log tells you whether it works for your code. After a week of routing real tasks through the ladder, open ~/effort_log.csv and look at three things.

First, where tasks stop. If most of them pass at low or medium, your default is set too high and you are paying for checks you do not need. If most climb to high or xhigh, either the tasks are hard or the prompts are thin, and Anthropic's own advice is to fix the context before adding effort.

Second, which kinds of tasks climb. In the Vals data, code migration and terminal work gained the most from effort. You may see something similar: renames, copy changes and small view fixes stopping at low, and concurrency bugs, migrations to a new API and build setting problems climbing. That split is a better guide than any single default, and you can save it as a habit: /effort low for chores, the default for features, and ultrathink on the one turn that needs it.

Third, the cost of failed rungs. A failed low run is not free. If a task type fails at low almost every time, skip that rung for it by passing --levels medium,high,xhigh. The paper reports that escalation came out cheaper for most proprietary models, not all, and your log will show whether that holds for you.

Here is a short script that summarizes the log by the level each task stopped at:

import csv
from collections import Counter
from pathlib import Path

rows = list(csv.DictReader(open(Path.home() / "effort_log.csv")))
stops = Counter(r["effort"] for r in rows if r["verify"] == "pass")
spent = sum(float(r["cost_usd"]) for r in rows if r["cost_usd"] not in ("", "None"))
print("Tasks finished at each level:", dict(stops))
print(f"Estimated spend across all attempts: ${spent:.2f}")

When max effort is still the right call

None of this means max effort is useless. Vals found no model matched its top index score at its lowest setting, and the coding benchmarks were among those that gained the most from more reasoning. Keep the higher settings for work where a wrong answer costs more than the tokens:

  • Changes you cannot test well, such as a migration of persisted data or a change to Keychain access.
  • Security reviews and anything touching authentication, where a confident shallow answer is worse than a slow one.
  • Long refactors across many files, where lower effort is more likely to stop early.
  • Bugs you have already failed to fix at a lower level, which is the ladder's last rung anyway.

Anthropic's docs also note that the higher levels use far more output, so if you run xhigh or max through the API, give max_tokens plenty of room or the response gets cut off mid-thought.

A starting setup for this week

  1. Run /effort status in Claude Code and write down what you are actually using. If you never set it, it is the model's default.
  2. If you set max or xhigh months ago and forgot, go back to the default with /effort auto.
  3. Use /effort low for chores: renames, comments, copy changes, small UI tweaks with a test or a quick visual check.
  4. Keep ultrathink for the single turn that needs it instead of raising the whole session.
  5. Route a handful of real tasks through the ladder script and read the log on Friday.

If you want a ready set of working documents for this, the Opus 5.5 Coding Workflow & Cost Kit is JRDEVHUB's companion kit for running Claude Code with checks. It has a 13-page PDF guide on planning, testing, code review and cost accounting, five editable Markdown templates plus a CLAUDE.md starter, an Excel calculator that estimates Claude API token charges from rates you enter, and a small Swift example with a flawed and a corrected package that share the same 12 tests. The calculator estimates API charges only; it does not predict Pro or Max limits.

The studies are about two weeks old and the models will change again, but the method holds up either way: pick the cheapest setting your tests accept, and let the log tell you when to spend more.

Sources