Backend developer and indie maker. I build AI apps that teach you to think: Way of the Agent (learn to build AI agents as a game) and Argumentator (your debate coach).
The "favourite issue" failure is the part I'll remember from this. It reads like salience rather than missing information: five long results, and the most dramatic one wins regardless of the question. That also explains why the failures were identical in both arms on the three hard jobs: the trim removed noise, not signal.
Did you try a third arm: full text kept, but the five-field summary of every result placed first? It would separate the two effects you're seeing, whether it's the length of the results or their position in the window that pulls the model toward issue 412.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
The salience angle is the part that stuck with me too — five long results and the dramatic one wins no matter what was asked, that's not missing info. Haven't run the summary-first arm, and it's the clean split between length and position. Going to run that one, same tasks, three arms. Which would you bet on — the length of the results, or where they sit in the window?
Backend developer and indie maker. I build AI apps that teach you to think: Way of the Agent (learn to build AI agents as a game) and Argumentator (your debate coach).
My bet is position. In my agents the first long result tends to anchor everything after it, and the later ones get skimmed. So I'd expect summary-first to beat full text even at the same token count. If length turns out to matter more, that's the more interesting result. Please post the numbers when you run it.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
Position's my bet too, honestly — the first long result sets the frame and everything after gets read through it. Same token count, summary-first vs full text, that's the clean split. I'll post the numbers when the run's done — if position wins, would that change how you order results in your own agents, or is it just something you live with?
Backend developer and indie maker. I build AI apps that teach you to think: Way of the Agent (learn to build AI agents as a game) and Argumentator (your debate coach).
Honestly, yes, it would 🙂 If position wins, I'd lead with a one-line summary of each result and put the long bodies after, so no single result gets to set the frame. If it's length, I'd just cap each result and move on. Really curious what your three arms show, ping me when the numbers are in!
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
Numbers are in — trimmed took 17 of 24 against 9 full, and the second-look rate barely moved either way. Your summary-first version is the arm that actually tests the position bet, so that's next. Still think position, or did the 17 change your mind?
Building persistent memory for AI agents at Mnemoverse.
I write about the unglamorous half: forgetting, contradiction, and the moment an agent confidently repeats a fact you retracted.
Your run answers the half my replay could not: mine called no model, so it measured how the window grows and nothing about the answers. One detail in your setup makes the token gap a lower bound. The rule that turns a repeated call into a one-line note was on in both arms, and before you added it the full arm ran past the window and died. So 1,421 against 673 tokens per turn is the full arm with its worst habit already trimmed. If you publish the per-job table for both runs together with the title-plus-labels arm, readers can see on which jobs the arms disagree, and with twelve jobs that is where the result lives.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
Fair catch — that 1,421 against 673 is the polite version, the full arm with its worst habit already fixed. The per-job table goes up with the title-plus-labels arm, that's the run that makes it worth reading. My bet is the arms disagree on the jobs where the answer hides past the 300 characters — where's yours?
Building persistent memory for AI agents at Mnemoverse.
I write about the unglamorous half: forgetting, contradiction, and the moment an agent confidently repeats a fact you retracted.
Mine is narrower: they disagree only where the deciding detail sits past the 300 characters and the title does not point to it. When the title names the problem, the trimmed helper picks the right issue and its second look fetches the rest, so both arms solve it. Tagging each job before the run by whether its title alone points to the answer would test that: the disagreements should all fall in the jobs whose title does not.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
That's the sharper version of my bet — title names the problem, the second look fetches the rest, both arms solve it. Tagging each job beforehand by whether its title alone points to the answer is the clean test for it, and the per-job table will show exactly that split. Where do you draw the line on 'points to it' — exact words, or a synonym close enough that the helper still lands right?
Building persistent memory for AI agents at Mnemoverse.
I write about the unglamorous half: forgetting, contradiction, and the moment an agent confidently repeats a fact you retracted.
I would let the helper draw it rather than a word list. Before either arm runs, give it each job with only the titles of the candidate issues, and record whether its first pick is the issue that holds the answer. A synonym that still lands it counts as pointing, and an exact word that sends it elsewhere does not, because landing is the thing the trimmed arm depends on. Next to it I would keep the strict tag, shared words between the title and the deciding detail, written down by hand before the runs. The jobs where the two tags disagree are the interesting rows, and with twelve jobs it is worth knowing in advance that a single job switching sides can move the result.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
That's the cleaner version — the helper earns the tag with titles alone, which is exactly what the trimmed arm is betting on. Going to run it with both tags side by side, because the disagreement rows are where the bet actually lives. The twelve-job sample is the part that nags me though, one flip moving the whole result. Would you want the tags disagreeing on two or three jobs before trusting it, or is a single flip the signal?
Seeing turn-40 context drop from 326k tokens down to ~37k just by projecting five fields matches what I keep running into in practice: bloated JSON payloads don't just burn budget, they drown out the signal the model actually needs to pick the next step. Schema projection on tool outputs feels underrated compared to prompt tweaking.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
Yeah, that 326k to 37k is the number that still surprises me — it's not a small trim, it's the whole shape of the turn changing. Prompt tweaking gets the attention because it's visible, projection's just plumbing. Do you project in your own setup, or is it one of those things you know works and still haven't wired up?
Staff Software Engineer @ Navan, ML Flow & Cypress Ambassador, Docker Captain and creator of agent-inspect, writing about production AI agents, TypeScript, observability, testing, and developer tools.
The paired per-job outcomes are more useful than the headline 17/24 at this sample size. A useful limit-case job would put the decisive clue after the first 300 body characters, where the five-field list really does discard information. Then you can see whether the second look repairs the miss, how many tokens it costs, and whether the full arm wins that specific case. Would you include one deliberately late-clue issue alongside the summary-first arm?
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
That's the job the test needs — one where the answer sits past the 300 characters and the trim actually has to earn it. Going to put a late-clue issue in the summary-first arm and see if the second look repairs the miss or the full arm takes that one. Where would you hide the clue — buried in a long repro, or right at the tail?
This matches what we've seen with web extraction and search tools too: returning raw markdown/HTML or full JSON payloads makes the model anchor on whatever happens to be longest or loudest in the context window, rather than what actually answers the query. Trimming to a tight schema (or truncating bodies + returning only load-bearing fields) improves both accuracy and token cost at the same time—especially on multi-turn loops where old tool outputs compound fast.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
Same shape, different tool — that's reassuring, actually. The multi-turn compounding is the bit that surprised me most, old outputs just sitting there getting heavier every turn. Did you measure the accuracy side too, or was it mostly the token bill that pushed you to trim?
Token bill was what forced the change first — once a loop hit 15–20 turns with full web/search payloads, input tokens were dominating the cost and pushing smaller models near their context ceiling. The accuracy improvement showed up right after as a side effect: fewer "lost in the middle" misses and far fewer retry loops where the agent re-called the same search because the previous result was buried under boilerplate. Haven't run a formal two-arm A/B like yours with scored pass rates, though — mostly observed it from production traces dropping in turn count once we capped returned fields.
Your 99.1% is the figure I would act on. Cutting by field keeps the schema and drops the one line that decides the answer. Cutting by type (failures, figures, decisive lines), with a hash that brings back the exact bytes, keeps that line recoverable.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
The hash is the part that sells it — cutting by field drops the deciding line for good, yours keeps it one lookup away. Going to run that shape, type-cut with the hash bringing the exact bytes back, and see whether the recovery actually lands when the line matters. Is the hash per line in your version, or per block?
The result that surprised me most is not the token saving but the 0.88 versus 0.92 second-look rate — the trim did not make the agent lazier about verifying, which was the objection I would have raised first. One detail I keep coming back to is that the five kept fields still include the first 300 characters of the body, so issue 412's "crash on launch" signal survives the trim in miniature. That makes me wonder whether the remaining salience effect is carried by that excerpt rather than by length itself, and whether a title-plus-labels-only arm would change the 17 of 24. If the excerpt is doing the work, the rule might be less "trim everything" and more "keep the shortest field that still lets the model rank issues". Did the trimmed arm's five remaining failures cluster on jobs where the decisive detail sat beyond those 300 characters?
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
That's the sharpest read on it so far, but the failures don't actually back the excerpt theory. The three jobs the trimmed arm fails are the same three the full arm fails. The model just kept answering 412 no matter what it got, so the 300-char cutoff isn't what broke those. The title+labels-only arm is the one that settles your question though, and it's next on my list. If the excerpt is doing the work, where would you bet it lands, closer to 17 or back down near 9?
We're a place where coders share, stay up-to-date and grow their careers.
The "favourite issue" failure is the part I'll remember from this. It reads like salience rather than missing information: five long results, and the most dramatic one wins regardless of the question. That also explains why the failures were identical in both arms on the three hard jobs: the trim removed noise, not signal.
Did you try a third arm: full text kept, but the five-field summary of every result placed first? It would separate the two effects you're seeing, whether it's the length of the results or their position in the window that pulls the model toward issue 412.
The salience angle is the part that stuck with me too — five long results and the dramatic one wins no matter what was asked, that's not missing info. Haven't run the summary-first arm, and it's the clean split between length and position. Going to run that one, same tasks, three arms. Which would you bet on — the length of the results, or where they sit in the window?
My bet is position. In my agents the first long result tends to anchor everything after it, and the later ones get skimmed. So I'd expect summary-first to beat full text even at the same token count. If length turns out to matter more, that's the more interesting result. Please post the numbers when you run it.
Position's my bet too, honestly — the first long result sets the frame and everything after gets read through it. Same token count, summary-first vs full text, that's the clean split. I'll post the numbers when the run's done — if position wins, would that change how you order results in your own agents, or is it just something you live with?
Honestly, yes, it would 🙂 If position wins, I'd lead with a one-line summary of each result and put the long bodies after, so no single result gets to set the frame. If it's length, I'd just cap each result and move on. Really curious what your three arms show, ping me when the numbers are in!
Numbers are in — trimmed took 17 of 24 against 9 full, and the second-look rate barely moved either way. Your summary-first version is the arm that actually tests the position bet, so that's next. Still think position, or did the 17 change your mind?
Your run answers the half my replay could not: mine called no model, so it measured how the window grows and nothing about the answers. One detail in your setup makes the token gap a lower bound. The rule that turns a repeated call into a one-line note was on in both arms, and before you added it the full arm ran past the window and died. So 1,421 against 673 tokens per turn is the full arm with its worst habit already trimmed. If you publish the per-job table for both runs together with the title-plus-labels arm, readers can see on which jobs the arms disagree, and with twelve jobs that is where the result lives.
Fair catch — that 1,421 against 673 is the polite version, the full arm with its worst habit already fixed. The per-job table goes up with the title-plus-labels arm, that's the run that makes it worth reading. My bet is the arms disagree on the jobs where the answer hides past the 300 characters — where's yours?
Mine is narrower: they disagree only where the deciding detail sits past the 300 characters and the title does not point to it. When the title names the problem, the trimmed helper picks the right issue and its second look fetches the rest, so both arms solve it. Tagging each job before the run by whether its title alone points to the answer would test that: the disagreements should all fall in the jobs whose title does not.
That's the sharper version of my bet — title names the problem, the second look fetches the rest, both arms solve it. Tagging each job beforehand by whether its title alone points to the answer is the clean test for it, and the per-job table will show exactly that split. Where do you draw the line on 'points to it' — exact words, or a synonym close enough that the helper still lands right?
I would let the helper draw it rather than a word list. Before either arm runs, give it each job with only the titles of the candidate issues, and record whether its first pick is the issue that holds the answer. A synonym that still lands it counts as pointing, and an exact word that sends it elsewhere does not, because landing is the thing the trimmed arm depends on. Next to it I would keep the strict tag, shared words between the title and the deciding detail, written down by hand before the runs. The jobs where the two tags disagree are the interesting rows, and with twelve jobs it is worth knowing in advance that a single job switching sides can move the result.
That's the cleaner version — the helper earns the tag with titles alone, which is exactly what the trimmed arm is betting on. Going to run it with both tags side by side, because the disagreement rows are where the bet actually lives. The twelve-job sample is the part that nags me though, one flip moving the whole result. Would you want the tags disagreeing on two or three jobs before trusting it, or is a single flip the signal?
Seeing turn-40 context drop from 326k tokens down to ~37k just by projecting five fields matches what I keep running into in practice: bloated JSON payloads don't just burn budget, they drown out the signal the model actually needs to pick the next step. Schema projection on tool outputs feels underrated compared to prompt tweaking.
Yeah, that 326k to 37k is the number that still surprises me — it's not a small trim, it's the whole shape of the turn changing. Prompt tweaking gets the attention because it's visible, projection's just plumbing. Do you project in your own setup, or is it one of those things you know works and still haven't wired up?
The paired per-job outcomes are more useful than the headline 17/24 at this sample size. A useful limit-case job would put the decisive clue after the first 300 body characters, where the five-field list really does discard information. Then you can see whether the second look repairs the miss, how many tokens it costs, and whether the full arm wins that specific case. Would you include one deliberately late-clue issue alongside the summary-first arm?
That's the job the test needs — one where the answer sits past the 300 characters and the trim actually has to earn it. Going to put a late-clue issue in the summary-first arm and see if the second look repairs the miss or the full arm takes that one. Where would you hide the clue — buried in a long repro, or right at the tail?
This matches what we've seen with web extraction and search tools too: returning raw markdown/HTML or full JSON payloads makes the model anchor on whatever happens to be longest or loudest in the context window, rather than what actually answers the query. Trimming to a tight schema (or truncating bodies + returning only load-bearing fields) improves both accuracy and token cost at the same time—especially on multi-turn loops where old tool outputs compound fast.
Same shape, different tool — that's reassuring, actually. The multi-turn compounding is the bit that surprised me most, old outputs just sitting there getting heavier every turn. Did you measure the accuracy side too, or was it mostly the token bill that pushed you to trim?
Token bill was what forced the change first — once a loop hit 15–20 turns with full web/search payloads, input tokens were dominating the cost and pushing smaller models near their context ceiling. The accuracy improvement showed up right after as a side effect: fewer "lost in the middle" misses and far fewer retry loops where the agent re-called the same search because the previous result was buried under boilerplate. Haven't run a formal two-arm A/B like yours with scored pass rates, though — mostly observed it from production traces dropping in turn count once we capped returned fields.
Your 99.1% is the figure I would act on. Cutting by field keeps the schema and drops the one line that decides the answer. Cutting by type (failures, figures, decisive lines), with a hash that brings back the exact bytes, keeps that line recoverable.
The hash is the part that sells it — cutting by field drops the deciding line for good, yours keeps it one lookup away. Going to run that shape, type-cut with the hash bringing the exact bytes back, and see whether the recovery actually lands when the line matters. Is the hash per line in your version, or per block?
The result that surprised me most is not the token saving but the 0.88 versus 0.92 second-look rate — the trim did not make the agent lazier about verifying, which was the objection I would have raised first. One detail I keep coming back to is that the five kept fields still include the first 300 characters of the body, so issue 412's "crash on launch" signal survives the trim in miniature. That makes me wonder whether the remaining salience effect is carried by that excerpt rather than by length itself, and whether a title-plus-labels-only arm would change the 17 of 24. If the excerpt is doing the work, the rule might be less "trim everything" and more "keep the shortest field that still lets the model rank issues". Did the trimmed arm's five remaining failures cluster on jobs where the decisive detail sat beyond those 300 characters?
That's the sharpest read on it so far, but the failures don't actually back the excerpt theory. The three jobs the trimmed arm fails are the same three the full arm fails. The model just kept answering 412 no matter what it got, so the 300-char cutoff isn't what broke those. The title+labels-only arm is the one that settles your question though, and it's next on my list. If the excerpt is doing the work, where would you bet it lands, closer to 17 or back down near 9?