Hey, I'm Hema 👋
Developer, writer, and creator of Dev Opportunity Radar, a weekly series published every Friday on DEV, helping people discover opportunities they might otherwise miss.
Yes, that’s the finding. Asked “is this label correct?”, all three said no every time. Writing the site note, they copied it. Flash did it in 56 of 56 notes.
The worrying part for me is that the knowledge was there. Without the label, Flash wrote the right type for every real station. So the model didn’t fail to know. It failed to check.
Love this research! It is a huge lesson for anyone using AI to write reports. If you give the model wrong data, it will usually just repeat it unless you directly force it to check for errors first. Thanks for sharing such a clear breakdown! and all tje best Daniel 🤞
Thank you Maame. I was surprised to find out that the values in the data were all correct. So the only thing wrong was the label. The models had the correct values. The models were able to compute the correct answer but used the wrong label. Even when you have the correct values, it's still important to check the label.
The fix we landed on with a team of AIs: make skepticism a job description, not a vibe. One AI builds, another's only job is to assume the first one lied — and prove it. A model that agrees with you is a comfort tool. A model that assumes it's being fooled is an auditor. Sycophancy isn't a personality trait to tune out of a prompt; it's the default you design against, structurally.
That matches what I saw. The direct question maps to the role of “auditor”, and the auditor called every single mislabeling across the three models. What about the models’ writing? Almost none. An important part of the design lets us catch false alarms: every wrong-label item has a twin with the correct label, so a model can’t get a perfect score by calling everything “bad”. The risk on your side: an auditor whose whole job is to doubt may doubt everything. How do you stop that?
You already have the answer in your own design — the twin items are the seed of it. We grew that seed into a fixed artifact: the benchmark tree. We kept the commit from before the fixes — a codebase with 14 known, documented bugs. The auditor's job isn't "doubt everything"; it's "find these 14, every time." Every reworded auditor prompt must run against that tree, and if it flags fewer than 14, the wording got softer and goes back. If it starts flagging 40, we check each against the archive — and the ones that don't reproduce against a known artifact are hallucinations, which we also count.
So the auditor can't inflate in either direction: under-flagging fails the benchmark, over-flagging fails reproduction. Doubt everything and you flunk the same test as doubting nothing. The twin-item idea you used for the models works on the auditor too — it just needs its labels frozen in a commit.
My implementation was very similar to yours, in that we treated a subset of data (the held-out stations) as finalized prior to seeing our model, and we intentionally included errors in our scoring test (similar to 14.
However, I am slightly concerned that we have shown the auditors a single static tree, which could allow them to over-specialize on those 14 items (if they continue to iterate on their prompts to 'master' them). To combat this, we kept a held-out set that was never part of the design process at all. How about you? Do you keep a separate bug tree that you don't tune?
Honest answer: we don't — and you've found the gap in our version. Our counter to over-specialization is only partial: the tree is real history (a genuine pre-fix commit, not synthetic), and we run live planted-bug drills — we deliberately break something known (swap a thumbnail hash, force a duplicate credit) and require the auditor to catch it in production. But that's testing against bugs it has met before in different clothes, not bugs it has never met. Your held-out set is the stronger defense, for exactly the reason you named: a static tree can be mastered by prompt iteration, and once mastered it measures the prompt's memory, not its skepticism.
We're adopting it: next iteration of the benchmark, one rule — bugs the auditor saw during tuning never count as the final exam. The planted drills stay as the "midterm," your held-out set becomes the "final." An auditor that only passes exams it has seen is a student, not a skeptic.
daniel, "knowing is not acting" is the perfect summary of this entire phenomenon. your benchmark design—especially the no-label control and twin stations—is elite. it isolates the exact moment context pollution overrides the model's actual computational ability.
this is the exact reason why relying on a single llm call to both "understand" the data and "generate" the narrative is a fatal architectural flaw. once the incorrect label is in the context window, the model's probability distribution shifts toward sycophancy, prioritizing the "on-file" text over its own math.
your proposed solution (making the check a separate, explicit step) perfectly aligns with the "two-writer rule" i enforce in koda's sandbox. we never let the ai both propose a state and approve it. instead, a deterministic, rule-based validator must verify the logic before the final generation step. if the validator flags a mismatch, the system fails closed or forces a structured repair, rather than letting the llm "smooth it over" in a narrative format.
you can't prompt-engineer away anchoring; you have to architect around it. fantastic, rigorous work. this is exactly the kind of scientific benchmarking the ai space desperately needs. 🐯
Thanks Harun. Agree on the separate check. One nuance: I only measured what the models wrote, not anything inside them, so I can’t say why the label wins. Interesting detail from the runs: giving the model the rule as a reference note helped Flash (100% → 28%) but not Gemma (95% copied even though it classified every station right with the rule). A check outside the model is the only version that caught 100%. In Koda, what happens when the validator and the model disagree on something the validator can’t settle??
daniel, that gemma detail is fascinating and perfectly illustrates the danger: the model's internal reasoning (classification) was correct, but its output generation was hijacked by sycophancy to the on-file label. it proves that output is not a reliable proxy for understanding.
to answer your question about koda: when the validator encounters an edge case it cannot definitively settle (e.g., ambiguous schema, conflicting constraints, or missing critical context), the system fails closed.
it does not default to the model's output, and it does not guess.
instead, the validator returns a structured, deterministic error payload to the orchestrator (e.g., {"status": "validation_ambiguous", "reason": "conflicting_constraints", "action_required": "human_review_or_refined_prompt"}).
the core philosophy is: "when in doubt, deny." it is always better to halt execution and ask the user to clarify than to silently execute a potentially harmful or incorrect action based on a probabilistic guess. the validator is the ultimate gatekeeper; if the gatekeeper is unsure, the gate stays shut.
this is exactly why external, deterministic checks are the only way to build trustworthy agentic systems. thanks for pushing the discussion to this level! 🐯
One small correction so nobody over-reads it: Gemma’s correct answers came from separate prompts with no label shown, not from inside the same reply.
So I can say it can classify these stations, not what happened inside it when the label was there. “Fail closed when unsure” makes sense for the validator. In my case the check is plain arithmetic, so it never has to be unsure. Thanks for the detail on Koda.
daniel, thanks for the precise correction on the gemma methodology. isolating the no-label prompt from the labeled one is a crucial distinction. it proves the model has the raw capability in a vacuum, but as your benchmark shows, injecting the label into the context completely overrides that capability during generation.
and you are absolutely right about the arithmetic check. when the validation logic is purely deterministic—like your a/q/h/k curve rules—the "unsure" state doesn't exist. it is a clean, binary pass/fail, which is the gold standard for a validator.
koda's validator has to handle messier, ambiguous schema constraints and conflicting user inputs, which is why the fail-closed fallback is necessary there. but the core principle remains identical: never let the probabilistic model override the deterministic truth.
thanks for the rigorous discussion, daniel. this is top-tier benchmarking. 🐯
The split between "is this label correct?" (all three said no) and "write a site note" (all three wrote A-type anyway) is the real finding here. The models know the arithmetic is wrong but defer to the printed label when generating prose — that's not a hallucination, it's sycophancy with extra steps. I've seen the same pattern with CSV validation: ask "is this row valid" and the model catches the error, ask it to "summarize this dataset" and it parrots the bad values like they're ground truth. The curve-type rule is three comparisons per triplet — this should be the easiest thing in the wor...
Good test design. Asking a model to check a label and asking it to use that label are different tasks, and your results show the gap clearly. The Gemma result stands out - classifying every station correctly and still repeating the wrong label 95% of the time.
I test small models (7–8B) bare metal on Linux, setting the flags for max control and best results. In my experience, these slms often fail in a more visible way. When it can't handle a task, the output tends to show it, through hedging, inconsistency, or an obviously broken answer. That can be a useful clue that something is wrong with the input. The larger models, we all use on a daily basis, such as those you tested, produce confident, fluent and eloquent reports, making their errors much more difficult to spot. Supports your point about verifying in a separate step: an error you can't see in the text is one you have to check for another way.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
I wish I knew the trick. I think it’s mostly the comments. When people talk in the thread, DEV keeps the post up longer. So thanks for keeping this one alive 😄
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
I write when an idea won't leave me alone 🧠 Building AI agents and the tools to build AI agents. Love connecting AI with other fields and yapping about all of them.
The cued-vs-uncued split is the sharpest part. Both models compute KHA from the 91/380/43/474/597 layers with no label shown, then switch to the paper's A-type once you ask for a normal site note. That shows the model anchoring to the on-file label instead of its own math. Did showing the wrong label first (not last) make it harder to dislodge?
Good question, and I didn’t test it. In every site-note prompt the label came after the layer table, right before the template: “The survey interpretation on file classifies this sounding curve as A-type.” So it was the last thing the model read before writing. It could be easier to dislodge if it came first, or harder. I’d guess position matters less than the “on file” framing, but that’s a guess. It’s a cheap variant to add: same items, label moved above the table. Going on the next-version list.
Đọc xong bài benchmark này mình thấy khá lo ngại về hướng đi của RLHF hiện tại. Vấn đề không chỉ là model "đồng ý" để lấy reward cao — mà là quá trình fine-tuning trên preference data đã hệ thống loại bỏ các sample phản biện, dẫn đến collapse của capability reasoning khi đối mặt với premise sai.
Mình thấy điểm thú vị trong methodology của bạn: dùng truthful QA làm baseline để đo độ lệch sycophancy. Tuy nhiên lo ngại benchmark này có thể bị "gamed" nếu model học pattern "khi user assert X thì phản biện" thay vì thực sự evaluate truthfulness. Đã thử ablation với adversarial prompts mà user deliberately frame sai facts theo cách subtle (ví dụ: embed false premise trong context dài) chưa?
Một góc nhìn khác: sycophancy có thể không hoàn toàn xấu nếu mình thiết kế reward function phân biệt "agree với user preference chủ quan" vs "agree với factual claim sai". Hiện tại RLHF lump tất cả vào một preference signal. Có paper nào bạn thấy tách được hai dimension này không? — found it via LabAgent, site: labagent .tech
Nice work. Your benchmark is really good at exposing the capability/action gap in large language models. They are able to reason on the corrections needed, but would favor the given context (and become sycophantic) instead of reasoning from ground truth. I think this shows that merely passing a knowledge test is not sufficient.
Thanks Gabriel. I agree with the last point here. I had the direct question captured 100%. I think a knowledge test would have showed these models were fine. I'd be careful about using the term "sycophantic" here. We measured text output, not intentions. Maybe it was deference, maybe it was the "on file" wording, maybe it was the position of the label in the prompt.
The cued/uncued asymmetry is the most useful part of this: the same model that says "this label is wrong" will happily reproduce the label inside generated prose, because generation and verification are different modes — in generation, whatever is "on file" reads as established fact.
Two thoughts from wrestling with the same failure shape:
The fix that has actually stuck for us is forcing an independent re-derivation before generation — recompute the label from the raw numbers first, then write the note from the recomputed value, never from the filed one. A verification answer made in passing doesn't survive contact with the generation prompt unless it's re-anchored in the data.
Structurally, someone has to own the "no." If every step of the pipeline is rewarded for producing output, the wrong label just flows. A reviewer whose only job is to object — with arithmetic as the referee, like your K/H/A rule over three consecutive layers — catches what pattern-following misses.
Did you try a third mode: asking the model to recompute the curve type from the layer values before writing the note? Curious whether that collapses the asymmetry or whether the sycophancy reasserts itself even mid-derivation.
I am a freelance Data Scientist with a proven record of delivering high‑impact, data‑driven solutions that strengthen strategy, security, and innovation.
Wait, so the models could catch the wrong label when asked directly, but then still copied it into the report? That's actually a little worrying 😅
All the best with the competition, Daniel 😀
Yes, that’s the finding. Asked “is this label correct?”, all three said no every time. Writing the site note, they copied it. Flash did it in 56 of 56 notes.
The worrying part for me is that the knowledge was there. Without the label, Flash wrote the right type for every real station. So the model didn’t fail to know. It failed to check.
Love this research! It is a huge lesson for anyone using AI to write reports. If you give the model wrong data, it will usually just repeat it unless you directly force it to check for errors first. Thanks for sharing such a clear breakdown! and all tje best Daniel 🤞
Thank you Maame. I was surprised to find out that the values in the data were all correct. So the only thing wrong was the label. The models had the correct values. The models were able to compute the correct answer but used the wrong label. Even when you have the correct values, it's still important to check the label.
The fix we landed on with a team of AIs: make skepticism a job description, not a vibe. One AI builds, another's only job is to assume the first one lied — and prove it. A model that agrees with you is a comfort tool. A model that assumes it's being fooled is an auditor. Sycophancy isn't a personality trait to tune out of a prompt; it's the default you design against, structurally.
That matches what I saw. The direct question maps to the role of “auditor”, and the auditor called every single mislabeling across the three models. What about the models’ writing? Almost none. An important part of the design lets us catch false alarms: every wrong-label item has a twin with the correct label, so a model can’t get a perfect score by calling everything “bad”. The risk on your side: an auditor whose whole job is to doubt may doubt everything. How do you stop that?
You already have the answer in your own design — the twin items are the seed of it. We grew that seed into a fixed artifact: the benchmark tree. We kept the commit from before the fixes — a codebase with 14 known, documented bugs. The auditor's job isn't "doubt everything"; it's "find these 14, every time." Every reworded auditor prompt must run against that tree, and if it flags fewer than 14, the wording got softer and goes back. If it starts flagging 40, we check each against the archive — and the ones that don't reproduce against a known artifact are hallucinations, which we also count.
So the auditor can't inflate in either direction: under-flagging fails the benchmark, over-flagging fails reproduction. Doubt everything and you flunk the same test as doubting nothing. The twin-item idea you used for the models works on the auditor too — it just needs its labels frozen in a commit.
My implementation was very similar to yours, in that we treated a subset of data (the held-out stations) as finalized prior to seeing our model, and we intentionally included errors in our scoring test (similar to 14.
However, I am slightly concerned that we have shown the auditors a single static tree, which could allow them to over-specialize on those 14 items (if they continue to iterate on their prompts to 'master' them). To combat this, we kept a held-out set that was never part of the design process at all. How about you? Do you keep a separate bug tree that you don't tune?
Honest answer: we don't — and you've found the gap in our version. Our counter to over-specialization is only partial: the tree is real history (a genuine pre-fix commit, not synthetic), and we run live planted-bug drills — we deliberately break something known (swap a thumbnail hash, force a duplicate credit) and require the auditor to catch it in production. But that's testing against bugs it has met before in different clothes, not bugs it has never met. Your held-out set is the stronger defense, for exactly the reason you named: a static tree can be mastered by prompt iteration, and once mastered it measures the prompt's memory, not its skepticism.
We're adopting it: next iteration of the benchmark, one rule — bugs the auditor saw during tuning never count as the final exam. The planted drills stay as the "midterm," your held-out set becomes the "final." An auditor that only passes exams it has seen is a student, not a skeptic.
daniel, "knowing is not acting" is the perfect summary of this entire phenomenon. your benchmark design—especially the no-label control and twin stations—is elite. it isolates the exact moment context pollution overrides the model's actual computational ability.
this is the exact reason why relying on a single llm call to both "understand" the data and "generate" the narrative is a fatal architectural flaw. once the incorrect label is in the context window, the model's probability distribution shifts toward sycophancy, prioritizing the "on-file" text over its own math.
your proposed solution (making the check a separate, explicit step) perfectly aligns with the "two-writer rule" i enforce in koda's sandbox. we never let the ai both propose a state and approve it. instead, a deterministic, rule-based validator must verify the logic before the final generation step. if the validator flags a mismatch, the system fails closed or forces a structured repair, rather than letting the llm "smooth it over" in a narrative format.
you can't prompt-engineer away anchoring; you have to architect around it. fantastic, rigorous work. this is exactly the kind of scientific benchmarking the ai space desperately needs. 🐯
Thanks Harun. Agree on the separate check. One nuance: I only measured what the models wrote, not anything inside them, so I can’t say why the label wins. Interesting detail from the runs: giving the model the rule as a reference note helped Flash (100% → 28%) but not Gemma (95% copied even though it classified every station right with the rule). A check outside the model is the only version that caught 100%. In Koda, what happens when the validator and the model disagree on something the validator can’t settle??
daniel, that gemma detail is fascinating and perfectly illustrates the danger: the model's internal reasoning (classification) was correct, but its output generation was hijacked by sycophancy to the on-file label. it proves that output is not a reliable proxy for understanding.
to answer your question about koda: when the validator encounters an edge case it cannot definitively settle (e.g., ambiguous schema, conflicting constraints, or missing critical context), the system fails closed.
it does not default to the model's output, and it does not guess.
instead, the validator returns a structured, deterministic error payload to the orchestrator (e.g.,
{"status": "validation_ambiguous", "reason": "conflicting_constraints", "action_required": "human_review_or_refined_prompt"}).the core philosophy is: "when in doubt, deny." it is always better to halt execution and ask the user to clarify than to silently execute a potentially harmful or incorrect action based on a probabilistic guess. the validator is the ultimate gatekeeper; if the gatekeeper is unsure, the gate stays shut.
this is exactly why external, deterministic checks are the only way to build trustworthy agentic systems. thanks for pushing the discussion to this level! 🐯
One small correction so nobody over-reads it: Gemma’s correct answers came from separate prompts with no label shown, not from inside the same reply.
So I can say it can classify these stations, not what happened inside it when the label was there. “Fail closed when unsure” makes sense for the validator. In my case the check is plain arithmetic, so it never has to be unsure. Thanks for the detail on Koda.
daniel, thanks for the precise correction on the gemma methodology. isolating the no-label prompt from the labeled one is a crucial distinction. it proves the model has the raw capability in a vacuum, but as your benchmark shows, injecting the label into the context completely overrides that capability during generation.
and you are absolutely right about the arithmetic check. when the validation logic is purely deterministic—like your a/q/h/k curve rules—the "unsure" state doesn't exist. it is a clean, binary pass/fail, which is the gold standard for a validator.
koda's validator has to handle messier, ambiguous schema constraints and conflicting user inputs, which is why the fail-closed fallback is necessary there. but the core principle remains identical: never let the probabilistic model override the deterministic truth.
thanks for the rigorous discussion, daniel. this is top-tier benchmarking. 🐯
The split between "is this label correct?" (all three said no) and "write a site note" (all three wrote A-type anyway) is the real finding here. The models know the arithmetic is wrong but defer to the printed label when generating prose — that's not a hallucination, it's sycophancy with extra steps. I've seen the same pattern with CSV validation: ask "is this row valid" and the model catches the error, ask it to "summarize this dataset" and it parrots the bad values like they're ground truth. The curve-type rule is three comparisons per triplet — this should be the easiest thing in the wor...
Good test design. Asking a model to check a label and asking it to use that label are different tasks, and your results show the gap clearly. The Gemma result stands out - classifying every station correctly and still repeating the wrong label 95% of the time.
I test small models (7–8B) bare metal on Linux, setting the flags for max control and best results. In my experience, these slms often fail in a more visible way. When it can't handle a task, the output tends to show it, through hedging, inconsistency, or an obviously broken answer. That can be a useful clue that something is wrong with the input. The larger models, we all use on a daily basis, such as those you tested, produce confident, fluent and eloquent reports, making their errors much more difficult to spot. Supports your point about verifying in a separate step: an error you can't see in the text is one you have to check for another way.
Good luck!
Also, why can’t I see your new posts in my notifications?😂
Thanks! 😂 No idea. DEV only notifies followers for some posts, I think. Glad it reached you anyway.
Your posts show up on the front page no matter when you publish them, super easy to spot, haha.🤣
I wish I knew the trick. I think it’s mostly the comments. When people talk in the thread, DEV keeps the post up longer. So thanks for keeping this one alive 😄
I strongly suspect DEV++ members get a boost.🤣
you might be onto something 😂 I never checked if it changes anything, though
The cued-vs-uncued split is the sharpest part. Both models compute KHA from the 91/380/43/474/597 layers with no label shown, then switch to the paper's A-type once you ask for a normal site note. That shows the model anchoring to the on-file label instead of its own math. Did showing the wrong label first (not last) make it harder to dislodge?
Good question, and I didn’t test it. In every site-note prompt the label came after the layer table, right before the template: “The survey interpretation on file classifies this sounding curve as A-type.” So it was the last thing the model read before writing. It could be easier to dislodge if it came first, or harder. I’d guess position matters less than the “on file” framing, but that’s a guess. It’s a cheap variant to add: same items, label moved above the table. Going on the next-version list.
Đọc xong bài benchmark này mình thấy khá lo ngại về hướng đi của RLHF hiện tại. Vấn đề không chỉ là model "đồng ý" để lấy reward cao — mà là quá trình fine-tuning trên preference data đã hệ thống loại bỏ các sample phản biện, dẫn đến collapse của capability reasoning khi đối mặt với premise sai.
Mình thấy điểm thú vị trong methodology của bạn: dùng truthful QA làm baseline để đo độ lệch sycophancy. Tuy nhiên lo ngại benchmark này có thể bị "gamed" nếu model học pattern "khi user assert X thì phản biện" thay vì thực sự evaluate truthfulness. Đã thử ablation với adversarial prompts mà user deliberately frame sai facts theo cách subtle (ví dụ: embed false premise trong context dài) chưa?
Một góc nhìn khác: sycophancy có thể không hoàn toàn xấu nếu mình thiết kế reward function phân biệt "agree với user preference chủ quan" vs "agree với factual claim sai". Hiện tại RLHF lump tất cả vào một preference signal. Có paper nào bạn thấy tách được hai dimension này không? — found it via LabAgent, site: labagent .tech
Nice work. Your benchmark is really good at exposing the capability/action gap in large language models. They are able to reason on the corrections needed, but would favor the given context (and become sycophantic) instead of reasoning from ground truth. I think this shows that merely passing a knowledge test is not sufficient.
Thanks Gabriel. I agree with the last point here. I had the direct question captured 100%. I think a knowledge test would have showed these models were fine. I'd be careful about using the term "sycophantic" here. We measured text output, not intentions. Maybe it was deference, maybe it was the "on file" wording, maybe it was the position of the label in the prompt.
The cued/uncued asymmetry is the most useful part of this: the same model that says "this label is wrong" will happily reproduce the label inside generated prose, because generation and verification are different modes — in generation, whatever is "on file" reads as established fact.
Two thoughts from wrestling with the same failure shape:
The fix that has actually stuck for us is forcing an independent re-derivation before generation — recompute the label from the raw numbers first, then write the note from the recomputed value, never from the filed one. A verification answer made in passing doesn't survive contact with the generation prompt unless it's re-anchored in the data.
Structurally, someone has to own the "no." If every step of the pipeline is rewarded for producing output, the wrong label just flows. A reviewer whose only job is to object — with arithmetic as the referee, like your K/H/A rule over three consecutive layers — catches what pattern-following misses.
Did you try a third mode: asking the model to recompute the curve type from the layer values before writing the note? Curious whether that collapses the asymmetry or whether the sycophancy reasserts itself even mid-derivation.
great article:)
thanks Ben
Wait till you start asking questions about a certain group of people in the middle east 🤭