A team of agents just beat your single agent by twelve points. You are about to tell your team swarms are worth the spend. Stop. The number that decides that sentence is not the score. It is the token count, and it is usually the thing nobody recorded.
This week a developer on dev.to, p0rt, had eight agents each review a slice of the multi-agent research, and six of the eight carried the same figure back: 17.2x. Five of the six read it as a plain error multiplier, because the preprint they had open still said the independent agents "amplify errors 17.2x." In the published version the number is a trace-level overhead factor, not a failure rate. His catch was not a second opinion; it was a checker that reopened the paper. That is the right instinct, and this post turns the other half of the problem into a check you can paste: was the comparison even on the same budget?
The study that keeps winning the argument
The paper behind that 17.2x is Kim et al., published in Nature Machine Intelligence in July 2026 under the title "Capable language models can outgrow the benefits of collaboration." It is the largest matched-budget comparison I have found, and the one to hold every swarm pitch against.
They held the prompt, the tool access, and the per-system compute budget constant, and changed only the coordination structure and the model. 260 configurations across six benchmarks, five architectures (a single agent plus independent, centralized, decentralized, and hybrid teams), and three model families.
- The mean multi-agent improvement over the single agent is 0.0%, with a 95% confidence interval from -58.7% to +77.2%. The average is a coin flip, and the variance is the whole story.
- The best case is +80.8%: a centralized team on a financial-analysis task that splits into independent research branches. The worst is -70.0%: independent agents on sequential planning, where every team design lost.
- The 17.2x is a trace-level factor, "how much extra computational work arises from interagent coordination failures," highest for the independent design. The task-level error ratio the paper works out from its own success rates is about 1.2, not 17.
- Token efficiency is where the structure shows. By one account of the paper, the single agent gets 67.7 successes per 1,000 reasoning tokens while the communicating designs drop to 13.6 through 23.9. The team spends more to do the same.
I read the journal page myself; the first three bullets are its own words, the fourth is relayed from the p0rt post and a Chinese digest. I did not read the full text of the six suites, so I am reporting what the paper states, not what I reproduced.
Why your crew demo is lying to you, on purpose
A swarm demo almost never matches the budget. The single agent gets one context and one budget; the team gets four contexts, a coordinator, and the messages between them. The score goes up a little, and the demo is about "how the team worked together."
The study's own numbers say what that gain is worth. At a matched reasoning-token budget, the team's edge is, on average, nothing. Add more tokens and the team usually wins, because more tokens is the only lever that always works. The paper points at the cause: where the big numbers appeared, token usage alone explained most of the variance. The team won by spending, not by structure.
So the question that separates a real finding from a bill is one sentence: which did you benchmark, the structure, or the budget? The study keeps asking it; your pipeline keeps answering "the structure" without checking.
That is a check, not a vibe. You already have the two numbers that answer it. You ran the baseline and the team; both token counts are in your logs. What you are missing is a gate that refuses to report the score delta until the token delta is ruled out.
The gate: prove the runs were on the same budget
The shape is the same as every other gate in this space. Turn "was this a fair fight" into "how far over the baseline token budget did the team run," and make the pipeline answer before the score means anything.
The script below does exactly that on your run records. Nothing installed, no model, no API key, standard library only.
#!/usr/bin/env python3
import json
import sys
REQUIRED = ("name", "score", "tokens")
def die2(msg):
print("CANNOT COMPARE: " + msg)
print("A structure comparison needs a single-agent baseline run named 'baseline'.")
sys.exit(2)
def fmt_tokens(n):
try:
return "{:,.0f}".format(float(n))
except (TypeError, ValueError):
return str(n)
def fmt_score(x):
try:
return "%.3f" % float(x)
except (TypeError, ValueError):
return str(x)
def main():
if len(sys.argv) != 2:
die2("usage: budget_gate.py runs.json")
try:
with open(sys.argv[1], encoding="utf-8") as fh:
data = json.load(fh)
except (OSError, json.JSONDecodeError) as exc:
die2("could not read runs.json (%s)" % exc)
runs = data.get("runs") or []
if not runs:
die2("no 'runs' array found")
for r in runs:
if not isinstance(r, dict) or any(k not in r for k in REQUIRED):
die2("each run needs name, score, and tokens")
base = [r for r in runs if r["name"] == "baseline"]
if len(base) != 1:
die2("exactly one run must be named 'baseline' (the single agent)")
base = base[0]
try:
tolerance = float(data.get("tolerance", 0.10))
except (TypeError, ValueError):
die2("'tolerance' must be a number (default 0.10)")
tolerance = max(0.0, tolerance)
limit = base["tokens"] * (1.0 + tolerance)
teams = [r for r in runs if r["name"] != "baseline"]
if not teams:
die2("no team runs to compare against the baseline")
print("baseline tokens: %s" % fmt_tokens(base["tokens"]))
print("tolerance: +%d%% -> team tokens must be <= %s"
% (int(round(tolerance * 100)), fmt_tokens(limit)))
print("-" * 64)
budget_wins = 0
for r in teams:
if r["tokens"] > limit:
budget_wins += 1
verdict = "BUDGET WIN, not structure"
note = ("spent %s tokens. Do not credit the team; you gave it a bigger budget."
% fmt_tokens(r["tokens"]))
else:
verdict = "matched budget: structure comparison is fair"
note = ("within tolerance. The score delta may be attributed to structure, cautiously."
)
print("%-12s score %s (base %s) -> %s"
% (r["name"], fmt_score(r["score"]), fmt_score(base["score"]), verdict))
print(" %s" % note)
for label, key in (("wall", "wall_s"), ("cost", "cost")):
if key in r and key in base:
print(" %-4s %s (base %s)" % (label, r[key], base[key]))
print()
if budget_wins:
print("RESULT: at least one team run was a budget win.")
print("Re-run the single agent at the team's token budget first.")
sys.exit(1)
print("RESULT: every team run stayed within the token tolerance.")
print("Report the wall-time and cost deltas alongside the score.")
sys.exit(0)
if __name__ == "__main__":
main()
runs.json holds an optional tolerance (default 0.10, so a team may spend up to 10% more tokens and still count as matched) and a runs array. Exactly one run is named baseline, the single agent; the rest are team runs.
Exit code 0: every team run stayed within the token tolerance, so the structure is comparable. Exit code 1: a team run spent more than the tolerance above the baseline, so the delta is a budget delta and must not be credited to the team. Exit code 2: an input problem, no baseline or a missing field, and the comparison is refused rather than guessed at.
A real run I just executed, on a worked example with round numbers, not a production trace. Two team runs, one within budget and one five times over:
baseline tokens: 1,000
tolerance: +10% -> team tokens must be <= 1,100
----------------------------------------------------------------
team-4 score 0.580 (base 0.500) -> matched budget: structure comparison is fair
within tolerance. The score delta may be attributed to structure, cautiously.
wall 95 (base 42)
cost 0.07 (base 0.03)
team-9 score 0.520 (base 0.500) -> matched budget: structure comparison is fair
within tolerance. The score delta may be attributed to structure, cautiously.
wall 210 (base 42)
cost 0.11 (base 0.03)
RESULT: every team run stayed within the token tolerance.
Report the wall-time and cost deltas alongside the score.
Same baseline, one team run at five times the budget:
team-4 score 0.620 (base 0.500) -> BUDGET WIN, not structure
spent 5,200 tokens. Do not credit the team; you gave it a bigger budget.
wall 210 (base 42)
cost 0.41 (base 0.03)
RESULT: at least one team run was a budget win.
Re-run the single agent at the team's token budget first.
That is the whole point. The team score is higher, and the run is still refused. The higher score is the price of five times the tokens. The gate does not need a model to say that.
What "worth the swarm" actually requires
The study's biggest wins, and the ones others reported in the same week, cluster around two conditions, worth keeping as the only two yes questions.
- Is there an automatic check on the result? A test suite, a verifier, an oracle. The big coding-swarm numbers come from runs where a grader that is not an agent has the last word. Where nothing can say no, you are mostly paying for one model agreeing with itself several times.
- Does the work split into parts that do not depend on each other? The +80.8% case is financial research that fans out into independent sources. The -70.0% case is sequential planning where each step changes the state the next reads. Same team, opposite sign, because of the task shape.
Two yeses, and a larger swarm is worth a pilot at a matched budget. Without two yeses, the evidence stops at a handful of agents; the paper's grid only measures teams of three or four. If the first answer is no, build the check before you build the swarm. That is the order.
What this post might get wrong
- I reproduced nothing. The 0.0%, the +80.8% and -70.0%, and the 17.2x trace factor are from the journal page I read; the 67.7 vs 13.6 to 23.9 token-efficiency split is relayed, not pulled off the page. I did not rerun the 260 configurations, so I am reporting the paper, not verifying it.
- The p0rt numbers (six reports carrying the 17.2x figure, five of the six misreading it, 46 of 118 claims changed on recheck) are from one post, which the author calls "an anecdote with an audit trail." I use it as the hook, not as a statistic.
- The matched-budget framing answers one question only: does a structure beat one agent at the same spend. It does not cover a job that will not fit one context, or a split that exists to keep tools and permissions apart. A team that is worth it for those reasons will look like a budget win to this gate.
The gate is not a verdict that swarms are bad. It is a refusal to let a score carry a claim the tokens do not support. Put it where your run results land, and let it exit 1 before your standup gets a sentence it has to un-say.
Sources
- Kim et al., 2026, "Capable language models can outgrow the benefits of collaboration," Nature Machine Intelligence, read directly at the journal page. 260 configurations, six benchmarks, matched compute: mean team effect 0.0% (95% CI -58.7% to +77.2%), +80.8% centralized on financial analysis, -70.0% independent on sequential planning, 17.2x as a trace-level factor.
- p0rt, 2026-11-12, "Five of My Eight Agents Left Out the Same Caveat," dev.to. Eight agents reviewing multi-agent research; six reports carried the 17.2x figure and five of the six read it as an error multiplier; a per-claim source recheck changed 46 of 118 claims. The author flags it as a single run with no control arm.
- A Chinese-language digest of the same Nature Machine Intelligence paper (mp.weixin.qq.com, read 2026-11-15) and a me.news piece on the same study cross-checked the 260-configuration, +80.8%/-70.0%, and 45% baseline-threshold figures; treated as relayed, not as a primary source.
This post was written with AI assistance. The author is responsible for its content.
Top comments (0)