DEV Community

Cover image for Don't Write Markdown Files for Your Coding Agent. Write Scripts.
Daniel Nwaneri
Daniel Nwaneri Subscriber

Posted on AI-assisted

Don't Write Markdown Files for Your Coding Agent. Write Scripts.

Comments warn agents modifying their own scripts

If you want agents to do what you want, you need to learn how to box them into deterministic actions. Unfortunately, this is not what current coding agents are good at. They are designed to be exploratory and open ended. They take a vague prompt, produce a plausible patch, run tests, loop a few times, and so on. This is great for prototyping and greenfield work. But when you need something reliable, when you want the same command to have the same effect every time, the same refactoring to happen to twenty files, the same deploy process to execute in the same order, the vague prompt and the "please read the docs" prompt become a liability.

So you end up with dozens of md files telling your agent to keep reading. That's the wrong place to stop.

Markdown documents aren't constraints. They're recommendations. Agents can read the CONTRIBUTING.md, AGENTS.md, CLAUDE.md, rules file, style guide, runbook, whatever, summarize them, quote from them, say "I've read the docs, and I'll do my best to follow them." They add more and more surface area that needs to be parsed and understood: "Use the existing logging conventions in the repository." Which? Where? "Use small functions." How small? "Run the tests before pushing." Which tests? Which command? Which environment? The agent has to process all of this, balance conflicting directives, and exercise judgment. It's in the judgment where the agents can stray.

In contrast, deterministic actions don't need judgment. "Read the docs and follow the coding style" becomes "Run the ./scripts/lint.sh, ./scripts/format.sh, ./scripts/test.sh scripts." The agent runs them and gets a pass/fail outcome. If one fails, the agent sees the exact error and has nothing to interpret or guess at. "I thought the rule meant…" becomes ./scripts/lint.sh. "Do the refactoring" becomes "Run the ast-grep or jscodeshift or sed tool to apply the transformation." The agent doesn't need to decide which files to update. The script does that, by design. You're boxing the agent into a sequence of deterministic actions.

This is a change in design paradigm from "teach the agent what you prefer" to "translate your preferences into deterministic actions." You can't run a markdown document. You can run a script, a checklist, a Makefile target, a GitHub Action. If you have a Makefile with make setup, make lint, make test, make build targets, the agent doesn't need to understand your coding philosophy. It just needs to run make lint and respond to the results.

The point isn't to dumb down the agent. The point is to make the environment unambiguous. An agent can still be smart when it executes a set of deterministic actions. It can read the failures, make corrections, and re-run the tests. It just can't step outside of the box. It can be clever within the constraints, but the box exists as a series of commands that either pass or fail.

This is why coding agents don't do this well. They want to be independent and explore the repo, read the files, build a mental model, and then make things happen. That works until it doesn't. The more md files you have, the more the agents think they have discretion. "The docs say to use snake_case, but this one existing file is camelCase. Maybe the docs are old." The agent is trying to be helpful, but it's not being deterministic.

Most AI models write replies that are hard to follow. So I added ASD-STE100, a plain-English standard written for aircraft maintenance manuals, to my coding agent's markdown rules. One rule says "Keep sentences to 20 words or fewer." As a script, that becomes MAX_SENTENCE = 20. When the agent ran the script over 10 of its own replies from one session, it found 9 failures. The script found them only after the replies were written, because nothing forced an earlier run. So the script goes in a hook.

I saw CI do this on a fix I sent to Forem. I don't write Ruby, and I had no Ruby installed, so CI was my test machine. My coding assistant wrote the fix and the tests. The first run failed, and the issue was in the test, not the code. That result did not depend on the assistant's opinion of its own work. But CI also passed an unused line, and only the reviewer caught it. A check only boxes in what it tests.

Full story: I Don't Write Ruby. A Fix I Sent Is Now in Forem.

If you want agents to do what you want, don't stop at md files. Write scripts.

Make every "please remember to" into a command. Every "we prefer" into a check. Every "before you push" into a pre-push hook. An agent can skip a local hook, so back it with a required CI job and a merge gate the agent has no rights to change. If a check fails, the agent can make corrections and run it again. That's how you box the agent into a deterministic sequence of actions. It's not a recommendation. It's a contract. And the agent can't ship work that fails the check.

Top comments (48)

Collapse
 
tokenlat profile image
TokenLat •

"It's not a recommendation. It's a contract" is the line that matters, and the comments have already done the hard work on root of trust — the settings file that declares the hook, the script hash, and a check that never ran printing the same line as one that found nothing. I don't want to retread that.

The gap I'd name sits one step later. A script makes a constraint executable, but a single exit code still compresses several failure classes into one boolean. Your sentence-length example is the clearest case: the check went from nine failures in ten to a pass, the replies got shorter, and you noted you hadn't looked at what went missing. Length is now enforced; whether the answer is still complete is a different question the same green cannot answer. That isn't an argument against scripts — prose can't even tell you which rule was consulted — but a gate needs to say which class it evaluated, not merely that it ran.

So I'd give the gate a small positive / boundary / follow-up set instead of one threshold. Positive: the ordinary violation. Boundary: exactly twenty words, a code block, a quoted line, an abbreviation — does the counter agree with the rule's intent? Follow-up: after the hook sends the draft back and the agent rewrites it, is the second receipt attributable to the same candidate, or just another green with no link to the first failure? Then coverage becomes recordable — script revision, input identifier, which class ran, each result — and "deterministic by construction" has a scope: the checks that exist and were evaluated on this input, not a claim about the whole task.

When your hook logs a pass, does it record which case set produced it and the revision of the script, or only the exit code?

Collapse
 
fm profile image
Fayaz • • Edited

Exactly! Deterministic + unambiguous agent control is the right way forward.
That is proper Software Engineering Automation (SWEA) as opposed to vibe coding.

Vibe only coders may hate me for saying this, but I'll say it anyway: No, English is not the next programming language, purpose built Deterministic + Unambiguous Agent Harness is. And as I said before: There's No AI Agent, Nor Any Harness 🤣🥰


Not sure about how good or bad ASD-STE100 performs though. Will check it out.

Collapse
 
dannwaneri profile image
Daniel Nwaneri •

measure it if you try it. As a markdown rule alone, my agent still wrote 9 sentences over the limit in 10 replies.

Collapse
 
mp3sss31 profile image
mp3sss •

The md = recommendation, script = constraint framing is exactly right. We hit the same split in Electron preload design: a line in the architecture doc saying "renderer must not call shell.openExternal on arbitrary URLs" is just prose the renderer can ignore, but a contextBridge method that validates the URL against an allowlist before calling shell.openExternal is a constraint the renderer cannot bypass. The corollary we learned the hard way: put enforcement in the smallest deterministic unit (a typed function, a schema, a binary check), and use markdown only for the why — never for the what. If the agent can phrase its way around it in natural language, it was never a rule.

Collapse
 
skillselion profile image
Skillselion •

An agent can skip a local hook, so back it with a required CI job and a merge gate the agent has no rights to change.

You asked upthread who guards the settings file that holds your Stop hook. The Claude Code docs answer most of it. The file watcher normally picks up direct edits to hooks, so an edit lands in the running session. One line, "disableAllHooks": true, turns every hook off, and a project-level value overrides a user-level one. .claude is a protected path: writes there are prompted in default and acceptEdits mode, and allowed in bypassPermissions.

The closest built-in guard is permissions.blockWritesOutsideWorkingDirectories. While it is on, Claude Code refuses Write, Edit and NotebookEdit writes to settings files in every permission mode (v2.1.314 or later). It does not limit shell commands, so a Bash edit of the file still lands unless the sandbox blocks it. A ConfigChange hook can also block a settings change from applying to the running session, but silently (debug log only), and the file stays edited for the next run. So your CI gate stays the backstop.

Collapse
 
panthpatel profile image
Panth Patel •

We went one step further and took those steps out of the agent. In our coding-agent pipeline the agent can't run git push at all: a guard on every tool call blocks it, and a plain script commits, runs the build check and pushes once a human moves the card. The agent's only exit, endWork, is refused until its plan is finished, its stack is healthy and the merge is clean. Same lesson as your hook: the check runs before the agent can say it's done.

Collapse
 
prpatel05 profile image
Pratik Patel •

Agree on boxing the agent into scripts. The failure mode I keep seeing is the agent "improving" the script mid-run — it edits lint.sh because the failure looked like a false positive, then green-lights itself. We treat ./scripts as root-owned and refuse any write to that directory from the agent session; if the checker is wrong, a human opens a PR. Pass/fail only works if the checker cannot rewrite the checker.

Collapse
 
dannwaneri profile image
Daniel Nwaneri •

Pratik, does your write block also apply to the configuration that calls the scripts? I have my hook in a settings file, and changing that file could disable the check.

Collapse
 
dafili profile image
Renhua Wei •

The settings file is the actual root of trust, so it has to sit in the same class as the script. If the agent can rewrite the file that declares the hook, it does not need to touch the script at all -- it just removes the invocation and the check goes quiet, which looks exactly like a passing check.

Where ours landed: the session config is readable but not writable while the agent is running. Any change to it is a file write outside the project subtree, and that is rejected by the tool layer before review -- not because a reviewer would object, but because the path is not in the writable root at all. So the hook declaration, the script path and the pinned hash all live outside what the agent can write, and the run fails loudly rather than silently when one of them stops matching.

Collapse
 
gapmine profile image
Balance •

The line that stuck with me is that markdown is a recommendation and a script is a constraint. I hit the same wall with the agent saying "fixed" when nothing had run since the last edit. A CLAUDE.md rule never stopped it, because by then the model already believes it's done. What worked was a Stop hook that checks the session when the turn ends and sends it back once if nothing ran after the last code edit. It's a small repo if anyone wants to see how it works: github.com/Balance0014/claude-code... . The surprise for me was how many false blocks my first version had, so I'd say replay your own history before you trust any check like this.

Collapse
 
reidmarlow profile image
Reid Marlow •

The markdown-versus-script trap is real: prose instructions invite negotiation.

When an invariant lives as a markdown paragraph ("keep functions under 30 lines and preserve existing error formatting"), an agent with a crowded context window treats it as a soft recommendation. If a test fails later, it starts explaining why its violation was justified by the surrounding code.

The clean boundary is keeping markdown strictly for high-level intent and delegating every mechanical rule to a script that exits non-zero. Replacing a prose rule about import hygiene with ruff check --select I or a custom ast-grep rule leaves no room for interpretation. When a command fails with an exit code and a diff, the loop has an exact receipt to fix instead of a preference to balance. If an invariant cannot fail a command, an agent will eventually negotiate around it.

Collapse
 
alexshev profile image
Alex Shev •

Scripts are often a better operational contract than prose because they can validate inputs and produce evidence as part of the run.

The useful split is to keep the why and the safety boundary in a short instruction file, while putting repeatable state changes and checks in versioned scripts with explicit exit conditions.

Collapse
 
nark3d profile image
Adam Lewis •

Know what happens to the scripts when the thing they check changes, because a markdown rule can be argued with and a script just fails. A lint cap or a sentence-length check is a number someone fixed on a particular day, and once it's a required gate that number keeps its authority long after the reason for it has gone. Who owns the script, then, and how does anyone go about changing it once the rule turns out to be the wrong one? Otherwise the box gets tighter and the code gets worse and the gate stays green.

Collapse
 
dannwaneri profile image
Daniel Nwaneri •

Adam, I haven't tested but the ones I have tested all passed but are 15% to 41% shorter. I haven't looked at what is missing from them.