DEV Community

Cover image for Super-Intelligent Yes-Men: Are We Training AI to Ignore the Truth?

Super-Intelligent Yes-Men: Are We Training AI to Ignore the Truth?

Daniel Nwaneri on October 09, 2026

This is a submission for the Kaggle Benchmarking Challenge What I Benchmarked A published paper on Choba, a survey point (station) in P...
Collapse
 
hemapriya_kanagala profile image
Hemapriya Kanagala • • Edited

Wait, so the models could catch the wrong label when asked directly, but then still copied it into the report? That's actually a little worrying 😅

All the best with the competition, Daniel 😀

Collapse
 
dannwaneri profile image
Daniel Nwaneri •

Yes, that’s the finding. Asked “is this label correct?”, all three said no every time. Writing the site note, they copied it. Flash did it in 56 of 56 notes.

The worrying part for me is that the knowledge was there. Without the label, Flash wrote the right type for every real station. So the model didn’t fail to know. It failed to check.

Collapse
 
maame-codes profile image
Maame Afua A. P. Fordjour •

Love this research! It is a huge lesson for anyone using AI to write reports. If you give the model wrong data, it will usually just repeat it unless you directly force it to check for errors first. Thanks for sharing such a clear breakdown! and all tje best Daniel 🤞

Collapse
 
dannwaneri profile image
Daniel Nwaneri •

Thank you Maame. I was surprised to find out that the values in the data were all correct. So the only thing wrong was the label. The models had the correct values. The models were able to compute the correct answer but used the wrong label. Even when you have the correct values, it's still important to check the label.

Collapse
 
bpecha14 profile image
Bogdan •

The fix we landed on with a team of AIs: make skepticism a job description, not a vibe. One AI builds, another's only job is to assume the first one lied — and prove it. A model that agrees with you is a comfort tool. A model that assumes it's being fooled is an auditor. Sycophancy isn't a personality trait to tune out of a prompt; it's the default you design against, structurally.

Collapse
 
dannwaneri profile image
Daniel Nwaneri •

That matches what I saw. The direct question maps to the role of “auditor”, and the auditor called every single mislabeling across the three models. What about the models’ writing? Almost none. An important part of the design lets us catch false alarms: every wrong-label item has a twin with the correct label, so a model can’t get a perfect score by calling everything “bad”. The risk on your side: an auditor whose whole job is to doubt may doubt everything. How do you stop that?

Collapse
 
bpecha14 profile image
Bogdan •

You already have the answer in your own design — the twin items are the seed of it. We grew that seed into a fixed artifact: the benchmark tree. We kept the commit from before the fixes — a codebase with 14 known, documented bugs. The auditor's job isn't "doubt everything"; it's "find these 14, every time." Every reworded auditor prompt must run against that tree, and if it flags fewer than 14, the wording got softer and goes back. If it starts flagging 40, we check each against the archive — and the ones that don't reproduce against a known artifact are hallucinations, which we also count.

So the auditor can't inflate in either direction: under-flagging fails the benchmark, over-flagging fails reproduction. Doubt everything and you flunk the same test as doubting nothing. The twin-item idea you used for the models works on the auditor too — it just needs its labels frozen in a commit.

Thread Thread
 
dannwaneri profile image
Daniel Nwaneri •

My implementation was very similar to yours, in that we treated a subset of data (the held-out stations) as finalized prior to seeing our model, and we intentionally included errors in our scoring test (similar to 14.

However, I am slightly concerned that we have shown the auditors a single static tree, which could allow them to over-specialize on those 14 items (if they continue to iterate on their prompts to 'master' them). To combat this, we kept a held-out set that was never part of the design process at all. How about you? Do you keep a separate bug tree that you don't tune?

Thread Thread
 
bpecha14 profile image
Bogdan •

Honest answer: we don't — and you've found the gap in our version. Our counter to over-specialization is only partial: the tree is real history (a genuine pre-fix commit, not synthetic), and we run live planted-bug drills — we deliberately break something known (swap a thumbnail hash, force a duplicate credit) and require the auditor to catch it in production. But that's testing against bugs it has met before in different clothes, not bugs it has never met. Your held-out set is the stronger defense, for exactly the reason you named: a static tree can be mastered by prompt iteration, and once mastered it measures the prompt's memory, not its skepticism.

We're adopting it: next iteration of the benchmark, one rule — bugs the auditor saw during tuning never count as the final exam. The planted drills stay as the "midterm," your held-out set becomes the "final." An auditor that only passes exams it has seen is a student, not a skeptic.

Collapse
 
koda2026 profile image
Harun - solo dev •

daniel, "knowing is not acting" is the perfect summary of this entire phenomenon. your benchmark design—especially the no-label control and twin stations—is elite. it isolates the exact moment context pollution overrides the model's actual computational ability.

this is the exact reason why relying on a single llm call to both "understand" the data and "generate" the narrative is a fatal architectural flaw. once the incorrect label is in the context window, the model's probability distribution shifts toward sycophancy, prioritizing the "on-file" text over its own math.

your proposed solution (making the check a separate, explicit step) perfectly aligns with the "two-writer rule" i enforce in koda's sandbox. we never let the ai both propose a state and approve it. instead, a deterministic, rule-based validator must verify the logic before the final generation step. if the validator flags a mismatch, the system fails closed or forces a structured repair, rather than letting the llm "smooth it over" in a narrative format.

you can't prompt-engineer away anchoring; you have to architect around it. fantastic, rigorous work. this is exactly the kind of scientific benchmarking the ai space desperately needs. 🐯

Collapse
 
dannwaneri profile image
Daniel Nwaneri •

Thanks Harun. Agree on the separate check. One nuance: I only measured what the models wrote, not anything inside them, so I can’t say why the label wins. Interesting detail from the runs: giving the model the rule as a reference note helped Flash (100% → 28%) but not Gemma (95% copied even though it classified every station right with the rule). A check outside the model is the only version that caught 100%. In Koda, what happens when the validator and the model disagree on something the validator can’t settle??

Collapse
 
koda2026 profile image
Harun - solo dev •

daniel, that gemma detail is fascinating and perfectly illustrates the danger: the model's internal reasoning (classification) was correct, but its output generation was hijacked by sycophancy to the on-file label. it proves that output is not a reliable proxy for understanding.

to answer your question about koda: when the validator encounters an edge case it cannot definitively settle (e.g., ambiguous schema, conflicting constraints, or missing critical context), the system fails closed.

it does not default to the model's output, and it does not guess.

instead, the validator returns a structured, deterministic error payload to the orchestrator (e.g., {"status": "validation_ambiguous", "reason": "conflicting_constraints", "action_required": "human_review_or_refined_prompt"}).

the core philosophy is: "when in doubt, deny." it is always better to halt execution and ask the user to clarify than to silently execute a potentially harmful or incorrect action based on a probabilistic guess. the validator is the ultimate gatekeeper; if the gatekeeper is unsure, the gate stays shut.

this is exactly why external, deterministic checks are the only way to build trustworthy agentic systems. thanks for pushing the discussion to this level! 🐯

Thread Thread
 
dannwaneri profile image
Daniel Nwaneri •

One small correction so nobody over-reads it: Gemma’s correct answers came from separate prompts with no label shown, not from inside the same reply.

So I can say it can classify these stations, not what happened inside it when the label was there. “Fail closed when unsure” makes sense for the validator. In my case the check is plain arithmetic, so it never has to be unsure. Thanks for the detail on Koda.

Thread Thread
 
koda2026 profile image
Harun - solo dev •

daniel, thanks for the precise correction on the gemma methodology. isolating the no-label prompt from the labeled one is a crucial distinction. it proves the model has the raw capability in a vacuum, but as your benchmark shows, injecting the label into the context completely overrides that capability during generation.

and you are absolutely right about the arithmetic check. when the validation logic is purely deterministic—like your a/q/h/k curve rules—the "unsure" state doesn't exist. it is a clean, binary pass/fail, which is the gold standard for a validator.

koda's validator has to handle messier, ambiguous schema constraints and conflicting user inputs, which is why the fail-closed fallback is necessary there. but the core principle remains identical: never let the probabilistic model override the deterministic truth.

thanks for the rigorous discussion, daniel. this is top-tier benchmarking. 🐯

Collapse
 
onizuka profile image
Onizuka •

The split between "is this label correct?" (all three said no) and "write a site note" (all three wrote A-type anyway) is the real finding here. The models know the arithmetic is wrong but defer to the printed label when generating prose — that's not a hallucination, it's sycophancy with extra steps. I've seen the same pattern with CSV validation: ask "is this row valid" and the model catches the error, ask it to "summarize this dataset" and it parrots the bad values like they're ground truth. The curve-type rule is three comparisons per triplet — this should be the easiest thing in the wor...

Collapse
 
tekk73 profile image
Ben Santora • • Edited

Good test design. Asking a model to check a label and asking it to use that label are different tasks, and your results show the gap clearly. The Gemma result stands out - classifying every station correctly and still repeating the wrong label 95% of the time.

I test small models (7–8B) bare metal on Linux, setting the flags for max control and best results. In my experience, these slms often fail in a more visible way. When it can't handle a task, the output tends to show it, through hedging, inconsistency, or an obviously broken answer. That can be a useful clue that something is wrong with the input. The larger models, we all use on a daily basis, such as those you tested, produce confident, fluent and eloquent reports, making their errors much more difficult to spot. Supports your point about verifying in a separate step: an error you can't see in the text is one you have to check for another way.

Collapse
 
xulingfeng profile image
xulingfeng •

Good luck!
Also, why can’t I see your new posts in my notifications?😂

Collapse
 
dannwaneri profile image
Daniel Nwaneri •

Thanks! 😂 No idea. DEV only notifies followers for some posts, I think. Glad it reached you anyway.

Collapse
 
xulingfeng profile image
xulingfeng •

Your posts show up on the front page no matter when you publish them, super easy to spot, haha.🤣

Thread Thread
 
dannwaneri profile image
Daniel Nwaneri •

I wish I knew the trick. I think it’s mostly the comments. When people talk in the thread, DEV keeps the post up longer. So thanks for keeping this one alive 😄

Thread Thread
 
xulingfeng profile image
xulingfeng •

I strongly suspect DEV++ members get a boost.🤣

Thread Thread
 
dannwaneri profile image
Daniel Nwaneri •

you might be onto something 😂 I never checked if it changes anything, though

Collapse
 
kartik-nvjk profile image
Kartik N V J K •

The cued-vs-uncued split is the sharpest part. Both models compute KHA from the 91/380/43/474/597 layers with no label shown, then switch to the paper's A-type once you ask for a normal site note. That shows the model anchoring to the on-file label instead of its own math. Did showing the wrong label first (not last) make it harder to dislodge?

Collapse
 
dannwaneri profile image
Daniel Nwaneri •

Good question, and I didn’t test it. In every site-note prompt the label came after the layer table, right before the template: “The survey interpretation on file classifies this sounding curve as A-type.” So it was the last thing the model read before writing. It could be easier to dislodge if it came first, or harder. I’d guess position matters less than the “on file” framing, but that’s a guess. It’s a cheap variant to add: same items, label moved above the table. Going on the next-version list.

Collapse
 
baumgaerben profile image
baumgaerben •

Đọc xong bài benchmark này mình thấy khá lo ngại về hướng đi của RLHF hiện tại. Vấn đề không chỉ là model "đồng ý" để lấy reward cao — mà là quá trình fine-tuning trên preference data đã hệ thống loại bỏ các sample phản biện, dẫn đến collapse của capability reasoning khi đối mặt với premise sai.

Mình thấy điểm thú vị trong methodology của bạn: dùng truthful QA làm baseline để đo độ lệch sycophancy. Tuy nhiên lo ngại benchmark này có thể bị "gamed" nếu model học pattern "khi user assert X thì phản biện" thay vì thực sự evaluate truthfulness. Đã thử ablation với adversarial prompts mà user deliberately frame sai facts theo cách subtle (ví dụ: embed false premise trong context dài) chưa?

Một góc nhìn khác: sycophancy có thể không hoàn toàn xấu nếu mình thiết kế reward function phân biệt "agree với user preference chủ quan" vs "agree với factual claim sai". Hiện tại RLHF lump tất cả vào một preference signal. Có paper nào bạn thấy tách được hai dimension này không? — found it via LabAgent, site: labagent .tech

Collapse
 
gabrielsegura9 profile image
Gabriel Segura •

Nice work. Your benchmark is really good at exposing the capability/action gap in large language models. They are able to reason on the corrections needed, but would favor the given context (and become sycophantic) instead of reasoning from ground truth. I think this shows that merely passing a knowledge test is not sufficient.

Collapse
 
dannwaneri profile image
Daniel Nwaneri •

Thanks Gabriel. I agree with the last point here. I had the direct question captured 100%. I think a knowledge test would have showed these models were fine. I'd be careful about using the term "sycophantic" here. We measured text output, not intentions. Maybe it was deference, maybe it was the "on file" wording, maybe it was the position of the label in the prompt.

Collapse
 
empireironcorps profile image
Empire Iron Corps •

The cued/uncued asymmetry is the most useful part of this: the same model that says "this label is wrong" will happily reproduce the label inside generated prose, because generation and verification are different modes — in generation, whatever is "on file" reads as established fact.

Two thoughts from wrestling with the same failure shape:

  1. The fix that has actually stuck for us is forcing an independent re-derivation before generation — recompute the label from the raw numbers first, then write the note from the recomputed value, never from the filed one. A verification answer made in passing doesn't survive contact with the generation prompt unless it's re-anchored in the data.

  2. Structurally, someone has to own the "no." If every step of the pipeline is rewarded for producing output, the wrong label just flows. A reviewer whose only job is to object — with arithmetic as the referee, like your K/H/A rule over three consecutive layers — catches what pattern-following misses.

Did you try a third mode: asking the model to recompute the curve type from the layer values before writing the note? Curious whether that collapses the asymmetry or whether the sycophancy reasserts itself even mid-derivation.

Collapse
 
benjamin_nguyen_8ca6ff360 profile image
Benjamin Nguyen •

great article:)

Collapse
 
dannwaneri profile image
Daniel Nwaneri •

thanks Ben

Collapse
 
rinrinrinrin profile image
Rin Katsuragi •

Wait till you start asking questions about a certain group of people in the middle east 🤭