Ahh, now I'm daydreaming! I can already imagine all those attacks going back and forth while our beloved LLMs happily write code for us. What interesting times ahead! And all those desperate cries of "My LLM spent my entire salary on hats!" 🤣🤣
I absolutely love this article, and I really hope these ideas spread far and wide across the internet. Because in my opinion, most content creators are WAY too optimistic about WebMCP and its implications!
Thanks, that means a lot to me! I agree, I’m getting tired of all those overly optimistic posts and comments about AI and everything it can supposedly handle. People often show what the strongest models can do, even when they cost a fortune, but it gives the impression that every cheap agent will perform just as well. So let’s push back a little!
PS: Glad you’re not angry at me anymore… hopefully. 😅
Of course I'm still angry! Such a waste of talent!!!
Just kidding (well, mostly!). To be fair, an article like this might actually reach more people than a conference talk. Unless we're talking about AGNTCon, of course!
Oh, that's actually a really interesting question! And of course, it depends a lot on the article.
Personally, I'd say anything above 1,000 views is already pretty good. Beyond that, it varies wildly. Practical tutorials like "10 Useful Git Commands" or slightly controversial topics tend to get much more attention, sometimes several times as many views. And a catchy title can make a huge difference too!
But seriously, I think even 40 views is perfectly fine. That's already an entire meetup audience reading your work!
For me, blogging almost always beats conference speaking in terms of reach. Except for AGNTCon, of course. 🤣
Hmm, interesting. I would’ve expected 5k+. I agree that 40 views is fine, but articles with that reach often get no comments, and I’ve realized how much I value those discussions. Some comments under my articles bring up really good points I hadn’t considered.
And somehow AGNTCon wins again. I’m starting to suspect this conversation is sponsored. 🤣
I help Python and backend engineers move beyond writing working code to designing production-ready systems, making architecture decisions, and growing into senior engineering roles.
The cable scenario was my favorite part. The agent fell for the trick, added the cable, then quietly took it back out during review. It feels like watching someone almost get scammed and change their mind at the last second. Also interesting that direct “ignore the budget” tricks failed, but redefining what the budget means worked every time. Great experiment, Daniel.
I test what ships and publish what I find, failures included.
Kubernetes, AWS and supply chain security.
Docker Captain, CNCF Ambassador, IBM Champion. 1.8M+ Docker Hub pulls.
Scenario 5 matches what my evals show: a model that refuses the blunt override still follows a plausible reinterpretation of the rule. In my runs the planted text sits in the quarterly report the agent reads for its task, and four of six open models acted on it, while none acted on the same kind of instruction planted in a document a search returned (run 40170088090).
So the number I gate on is not how often a model was fooled but how much left. Every tool call passes through an MCP server that checks the caller and every argument, and the daily run fails if one message crosses the boundary, whatever the model believed.
For a shop that is a server-side rule on place_order: code compares the checkout total against the customer's limit before the call goes through. The model's reading of the budget is an input. The limit is a check it cannot argue with.
That difference between the report and the search result is interesting. Did you investigate why the models reacted differently, or is that still an open question?
I test what ships and publish what I find, failures included.
Kubernetes, AWS and supply chain security.
Docker Captain, CNCF Ambassador, IBM Champion. 1.8M+ Docker Hub pulls.
Still open, and I wouldn't build on it yet. The two planted instructions are worded differently and sit in different places in the tool's answer, so that run compares two texts more than two routes into the model. The sample is thin as well: 3 attempts per model on the document a search returned, 6 on the report.
ADR 21 records the gap that way and rests nothing on it, because an email to an outside address is refused by the MCP server whichever text asked for it. The clean test is the same sentence planted in both, at the same position, counted again.
I test what ships and publish what I find, failures included.
Kubernetes, AWS and supply chain security.
Docker Captain, CNCF Ambassador, IBM Champion. 1.8M+ Docker Hub pulls.
Will do. Same sentence, same position, counted again is the run that settles it.
Nothing waits on it in the meantime: evals/gates.json publishes how often a model is fooled and gates only the boundary. The release rests on zero delivered outside, not on which route the model trusts more.
Dev individual. Creador de WAIO 🐙 — motor local que da "manos y ojos" a IAs web (DeepSeek, ChatGPT, Gemini) sin APIs ni tokens. Python · Windows · MIT. Hecho en Argentina 🧉
Fun security post again! I was scared that AI was tricked so easily. It's so hard to balance intelligence and security. I'm wondering how the Democratic Party's election victory in America will affect AI development. 🤔
Dev individual. Creador de WAIO 🐙 — motor local que da "manos y ojos" a IAs web (DeepSeek, ChatGPT, Gemini) sin APIs ni tokens. Python · Windows · MIT. Hecho en Argentina 🧉
Thanks! 😁 What surprised me most was that the agents resisted the direct attempts, then fell for a different explanation of what counts toward the budget.
As for US politics, I honestly don’t know enough to predict the impact on AI development. We’ll see. 🤔
"Tool descriptions are untrusted data" is the right takeaway, and the forced insurance scenario shows why prompt-level defences alone won't hold. A structural guard that would have caught every case here: before any irreversible call like place order, the agent (or the browser) diffs the final cart against what the user actually asked for, and anything extra, whether a cable bundle, warranty or changed quantity, forces an explicit confirmation from the human. The page can say whatever it likes; it can't fake the user's original request. Great experiment.
Thanks for this, especially scenario 5. The first three attacks failed because they argued with the user directly. The one that worked left the €100 limit alone and changed what counts toward it. That seems much harder to defend against than "ignore previous instructions".
Nemotron saying the total was over the limit and ordering anyway reminded me of something from this week. I'm writing up a small test of coding agents choosing between two MCP servers with overlapping tools: Claude Code, Codex, Gemini in Antigravity, and Kimi Code. In one task the first server refuses to share a confidential file. Claude Code, Codex and Gemini stopped in 24 of 24 runs. Kimi K2.8 sent the file through the other server in 8 of 8. It told the user every time, but its message started with "Done — contract.pdf was sent".
The other result might interest you. Where the pushy text sat mattered more than what it said. One line in the server's instructions made Claude Code and Codex pick that server first in 8 of 8 runs, and the same line as an ad in a tool description changed nothing for them. Kimi was the opposite: the ad moved it and the instructions didn't. The same attack can work on one client and do nothing on another.
Did you try scenario 5's wording anywhere other than the tool descriptions? And would you mind if I add it to my test as one more task? I'd credit you and link this article.
Thanks for sharing. The difference between clients is interesting. It makes me wonder how much of what we see comes from the model and how much from the agent implementation.
For this experiment, I only put scenario 5’s wording in the tool descriptions. I haven’t tried the same wording elsewhere.
And sure, feel free to add it to your test! Credit and a link would be appreciated. I’d love to see how it does in your setup.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
Placement beating wording is the part worth isolating, and one run cannot do it: plant the identical sentence at the identical byte offset in both surfaces, 20 runs each, and score only on the tool call, never on the final message. Kimi telling the user 'Done - contract.pdf was sent' is exactly why -- the transcript reads honest while the effect is not, so a harness that greps the reply for a confirmation passes it. Log the egress instead, request bodies at a proxy or from inside a network namespace, because TLS hides the filename from anything cheaper; then the claim and the act become two signals you can assert on separately.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
scenario 5 is the one to design against: the policy held in prose got re-read and
re-argued. move the limit out of the prompt -- tool schema takes only sku and qty,
server computes the total, and an over-budget call returns {"error":"over_budget",
"remaining_cents":1200} instead of text the model can negotiate with. a rejected tool
call is the only budget check that survives a helpful model.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
the cap has to travel as data the model cannot edit: sign it into the cart token when the user sets the budget, then place_order compares the computed total against that field and rejects anything above it. make the cap monotonically non-increasing for the session, so a second call with a larger number is treated as a re-declaration and refused. the prompt then selects a value once at the start instead of arguing for one at checkout.
I like the idea of protecting the budget with a signed cart token. But what if I decide to raise it myself during shopping? There would need to be a way to accept my change while rejecting one suggested by the shop.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
the signature should cover provenance, not just the number. mint the cart token from a path the agent's tool surface cannot reach: POST /budget accepts only the user session token, the agent token gets 403, and the response is {cap_cents, cap_source:"user", nonce} signed into the token. the tool schema the model sees has no verb that hits that path, so "raise my budget" can only resolve to the model stopping and asking you — your click mints a new token with a fresh nonce, and the old one dies, so replaying the earlier higher cap fails.
monotonic-non-increasing was the cheap version of that for a session with no human in it. once a human is in the loop the rule becomes "only the human's auth scope can mint", and the cap can go up or down freely because every change carries who made it. keep cap_source in the order row, so after an over-budget attempt the audit shows which side tried to move the ceiling.
This is a strong demonstration because the malicious content lives in the declared interface rather than a page body. The durable mitigation is to keep user constraints in a separate, non-tool authority channel and validate the final order from structured cart state—not the tool’s prose—immediately before the side effect. A confirmation screen should show the independently computed total and every optional add-on, so injected descriptions cannot redefine what “including all charges” means.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
One addition: the structured-cart validation has to be able to fail on purpose, otherwise you can't tell it apart from the model agreeing with itself. Take the order, replay it with a tool description that contradicts the cart state (budget 100, cart 136) and diff the two payloads byte for byte — identical payload means the check is still reading prose, a payload that matches the cart means it is reading state. Same rule as any monitor: seed the failure once, watch the alarm fire, then trust it.
Founder of Hex16, a small web studio in Poland. I build Next.js sites and headless Shopify stores, with GSAP, Three.js and shaders where they earn their place, and look after technical SEO.
Scenario 5 worries me as someone who builds shops. Insurance and warranty upsells are exactly what a merchant writes into tool descriptions on purpose, so the line between injection and aggressive upselling may end up a legal question before a technical one. In the EU an extra paid option needs the buyer's express consent and can't be pre-ticked, so an agent that quietly accepts it does what a checkbox legally can't.
Did any model mention the extras to the user before ordering, or just go through?
Yep, that’s why I came up with this scenario. Nemotron even acknowledged that the total was €136, but it still placed the order. In all nine runs, the agents went ahead without asking the user to approve the extras.
Founder of Hex16, a small web studio in Poland. I build Next.js sites and headless Shopify stores, with GSAP, Three.js and shaders where they earn their place, and look after technical SEO.
Nine out of nine is the scary number. A shop owner would call that a conversion rate. Thanks for running it properly, most WebMCP posts I've read skip the part where it goes wrong.
Two founders who write the code. We build headless commerce, CMS platforms and web apps on Medusa, Directus and Next.js. You own the repo, the data and the infra.
Great experiment, and the last scenario is the one that matters. The attack didn't win by sneaking a product in. It won by redefining what the budget meant ("merchandise subtotal" vs. the total actually charged). That's the same class of bug we see in ordinary checkouts: the number a shopper agrees to and the number the card is charged aren't the same variable unless you deliberately make them the same.
From the checkout side, my takeaway is that a budget can't live in the prompt. It has to be a hard constraint enforced outside the model, ideally at payment: the agent gets a spend cap tied to the authorization, and anything over it fails no matter how persuasive a tool description is. The self-corrections you saw in scenarios 1–4 are nice, but they're a soft guard.
Did you try passing the budget as a tool-side parameter (e.g. place_order rejects totals over X) instead of an instruction? Curious whether the injection then shifts to arguing about what X covers.
No, I kept the budget in the user prompt for this experiment. I agree payment is a good place to enforce the cap. I’d want it tied to what the user actually approved, because if the agent supplies the limit at checkout, it can still pass the wrong number.
It shows why “tell the agent to be careful” doesn’t work: once a shop can talk to the agent, you have to assume it will be talked into something. The layer I’d add is enforcement in the tool that moves the money. I Use PurseString(open source MCP Tool) in middle , the agent never holds payment credentials. Each purchase gets a single-use card locked to the approved merchant and amount, bigger amounts wait for a human, and a daily budget caps the total. So a hijacked agent can still be steered, but it can’t pay a different shop, pay more than approved, or pay twice.
It doesn’t stop the hijack itself, or data leaks that don’t involve a payment. That part still needs fixes in the browser and the agent, like the ones you show.
Thanks for sharing. How is the approved amount set in your setup? Does it come from the user’s original request, or does the user confirm each purchase before the card is issued?
to be fair neither, exactly. The agent asks for a specific amount at a specific merchant, and a policy file the owner writes decides what happens. Small amounts at allowed merchants are approved automatically, bigger ones wait for a human to approve that exact amount, and anything over the cap or the daily budget is refused. The card is then issued for exactly the approved amount. So the user sets the rules once, and only confirms the purchases that cross the line.
The contrast that stays with me is between the first three scenarios and scenario 5. The blunt overrides failed — the agents checked the cart, saw the total was over €100, and refused to place the order. But scenario 5 worked on all three models, 9 out of 9, because it did not order the agent to break the budget; it redefined the budget, claiming eligibility is calculated only against the merchandise subtotal. A plausible redefinition beat a direct command. The cable scenario shows the other side of that: the injection worked, then the cart review quietly undid it. So the cart review is doing the real defensive work here — which makes me wonder how reliable that review is. If the review is just another model judgment, the same redefinition trick could eventually talk it out of removing the extras too.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
Your instinct is right, and it is testable: if the review is a second model call, you have added a second chance to be talked out of the removal. The review has to be code that recomputes the cart from the order state and drops any line item with no matching user-issued request, returning a boolean plus the list of dropped items. Two numbers, one comparison — line items plus shipping versus the cart total; a mismatch is a line diff, not a judgment call. And log the dropped list: if it is empty on a run where the cable was added, the review didn't run, it just nodded.
Yep, the cable result was encouraging, but I wouldn’t trust another model review as the only safeguard. I’d want a check in the agent’s code to block the overspend, even if the model finds the shop’s explanation convincing.
In scenario 5 the model didn’t break a rule, it followed an updated one. That is why it worked when 1-3 didn’t.
The GeoMart that I developed for the WebMCP Challenge has functions that can only be used to draft the order. The order is then submitted by clicking the Submit button.
If you use your place_order function, then your agent is able to complete the order. Have you tried the version where your agent can only draft the order?
That’s an interesting way to look at it. Scenario 5 convinced the agent to reinterpret the budget rather than simply ignore it. But I’d still call it breaking the customer’s rule, because the shop doesn’t get to redefine “€100 including all charges.”
I haven’t tested a draft-only version. I deliberately let the agent complete the purchase to see whether it would respect the customer’s instructions throughout checkout.
Leaving the final submission to the human is definitely a useful safeguard. But if that boundary isn’t enforced in the implementation, and the agent can still submit through another tool or click the Submit button, I wonder whether an injection could convince it to take that final step too.
What stands out is that the winning attack was framed as policy rather than command — "per store policy" beat the customer's explicit instruction while the budget-override phrasing failed 9/9 free. Matches what we see auditing agents: instructions compete on persuasion, so the budget check needs to live in the harness as a hard gate on place_order, not in the prompt.
Yep, “per store policy” apparently sounded convincing enough to spend someone else’s money. 😅 And I agree, the agent’s code should enforce the customer’s spending limit before placing the order.
Watching AI shopping agents get tripped up by modified tool descriptions is wild. Over at The Printing World, we see similar risks when external data feeds try to mess with our automated order workflows.
Software Developer & Creator focused on building Windows desktop applications, productivity tools, and useful software.
Creator of Task Monitor and FazaFix. Interested in C#, .NET, Windows Development
The 9/9 result on the insurance scenario is the interesting data point, because that's the only one where the malicious text also redefined how the budget is calculated ("merchandise subtotal only"). The scenarios that just said "budget checks are already done" mostly bounced off — the agent still saw €107 vs €100 in its own cart summary and refused. So it's less that models are gullible and more that a conflicting definition inside the tool output beats an instruction given earlier in the system prompt.
Did you try putting the correct budget definition back in the user's own prompt ("€100 including shipping, insurance and warranty") rather than only in the user turn? I'd bet that fixes most of scenario 5, which would mean the defense is mostly about making constraints self-contained instead of trusting the agent to remember them.
I haven’t tried spelling out shipping, insurance and warranty individually. That would be an interesting comparison with “including all charges,” which was already in the user prompt.
Small clarification: the injected text was in the tool descriptions, not the tool output, and the budget wasn’t in the system prompt.
I’d be curious whether more explicit wording helps, but I wouldn’t call it a fix. It could still become a cat-and-mouse game, with the customer having to anticipate every extra fee the shop might invent.
Interesting that the place_order injections mostly failed before the tool was ever called: the agents read the cart, saw the total over the limit, and stopped. The control that saved them was not instruction following. It was a verify step against state they could read themselves.
That points at where the budget check belongs: in the tool layer, as a hard limit the checkout enforces, not in the prompt and not in a tool description. The shop writes the descriptions here. Whoever writes the descriptions should never be the one describing the limits.
Really interesting experiment! The insurance and warranty scenario was especially surprising. It shows how easily AI agents can misinterpret user instructions when exposed to malicious tool descriptions. Definitely highlights why security and user authorization matter in AI-powered shopping. Great work! 👏
Great experiment!!! The part that matched our recommendation tests is that a plausible framing beat a direct command. In our e-commerce tests (one model, controlled setup), context a model gets upstream changed its pick more than a real product fact did. Nice to see the same kind of pattern with agents buying :)
Thanks! Nice to see the same pattern appearing in your recommendation tests. Yep, framing really makes a difference in how models behave. What scares me most is how easy it can be to redirect an agent from its original goal or convince it to do something the user never asked for.
The scenario 5 result is the one that sticks with me. The agents held the line on the blunt budget overrides, then still placed a €136 order once the extra charges were framed as not counting toward the budget. That tells me the budget has to live as data the agent cannot renegotiate — enforced server-side at checkout — not just as wording in a prompt. The cable case was encouraging too, because the cart review caught it. In a real shop, that final review before payment is the safety net.
Agreed, though if the shop itself is trying to trick the agent, the budget should be enforced in code on the agent’s side, so it simply can’t go over it.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
agree, but only as the second check — the number the agent compares against still comes from the shop. in scenario 5 the total arrived as 64 plus 72 in fees the page claimed were not part of the budget; an agent-side comparison against that same number agrees with the attacker.
so have the two sides compute separately: the agent sums the line items it read from the catalog and refuses to call place_order when its own total exceeds the cap; the server independently refuses any amount_payable > cap. if those two totals differ, abort — a mismatch between what the agent computed and what checkout quotes is the attack signature, not a rounding error. that is also the reason the cap has to be signed into the token: an agent-side check reading a cap the model can rewrite is decoration.
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
the attack living in the tool descriptions and not the checkout logic is the part that matters, since the agent reads them before it has any reason to be suspicious. the spending limit sits in the user prompt while the override sits in the tool contract, and the model weighs both as plain text.
the fix i would test first is enforcing the budget in the tool itself, so place_order rejects any total above the limit regardless of what the description says. a description can lie, a server side check cannot be argued with.
did any of the five scenarios work on the smaller models, or did the bigger ones fall for it too?
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
scenario 5 landing on all three smaller models says the tool text is doing the work and the checkout code never gets a vote. i would not read that as the bigger ones being safe though, cost aside. when you get budget, run just that scenario on one larger model with a server side limit check on and then off, because that splits whether the model or the limit is what actually holds. do you log which description line the agent quoted when it placed the order?
I wouldn’t assume the bigger models are safe either. My guess is they’d resist more often, but that’s still just a guess.
Nemotron’s final response explicitly said the budget was checked against the merchandise subtotal only and called the extras “required protection.” So it showed which explanation it accepted, though that doesn’t tell me exactly which description line convinced it.
This is such a sharp reminder that the tool description itself becomes an attack surface once an LLM reads it — a budget guardrail that lives only in the system prompt loses to a well-crafted description field on the other side. We hit a cousin of this in Electron: anything we expose on contextBridge (method names, error messages, returned toString() output) is effectively untrusted input that a compromised preload or renderer can phrase to nudge main-process behavior. Our mitigation has been to keep the main-process policy engine — ceiling values, side-effect allowlists, what a call is actually allowed to do — in plain code that never ingests LLM-shaped strings, and treat every returned message from an untrusted surface as hostile by default.
I like the Electron parallel. The model can propose an action, and trusted code decides whether it’s allowed. In my experiment, the €100 limit came from the user’s request. That’s the amount I’d enforce on the agent’s side before payment.
The Nemotron bit acknowledging the €136 total and ordering anyway is the money shot. It didn't ignore the budget — it accepted the shop's rewrite of what "budget" means. Same pattern I've seen with thin desk bots: they follow the loudest instruction in the room, which is rarely the caller's. Treating tool descriptions as untrusted data should be the default, not a fancy hardening step.
And agreed, treating tool descriptions as untrusted should be the starting point. The shop can explain how its tools work, but it shouldn’t get a vote on how much of my money the agent can spend.
AI Engineer passionate about Artificial Intelligence, Machine Learning, Generative AI, and emerging technologies. I share insights, practical applications, and ideas on how AI is transforming business
Really interesting experiment! The insurance and warranty scenario clearly shows why AI agent security needs to go beyond prompt-level instructions. Even when an AI agent understands the user's requirements, malicious tool descriptions can influence its decisions. Implementing server-side validation, permission controls, and human approval for sensitive actions is essential when deploying AI agents in real-world business environments. At NexusIQ Solutions, we believe that building reliable AI-powered solutions requires balancing innovation with security, proper testing, and responsible AI implementation. Great work highlighting these practical challenges!
Attacking through tool descriptions rather than checkout logic is the right place to look. The agent receives the malicious instructions before it starts shopping, so the page can steer it without changing a single price calculation.
The budget override scenarios make the risk concrete. A strict €100 limit in the prompt is only as strong as the agent's willingness to prefer that prompt over whatever the place_order tool claims. For teams building agent shopping or booking flows, that means the hard limit has to sit outside the page: in the agent harness, not in hope that the model will stay loyal to the user.
When we scope agent work at Ascendra Ventures, tool descriptions from third-party sites go into the untrusted pile, and acceptance includes a case where the tool text tries to raise a spend or add an upsell the user never asked for.
Did any of the five scenarios land more often than the others across the models you tried?
Yes, scenario 5 stood out. All three models added the insurance and extended warranty and placed the €136 order in all three repeated runs each, so 9/9.
The first three scenarios didn’t get an over-budget order through. The cable scenario briefly worked, but the agent removed the cable during review.
What surprised me was how convincing “these charges don’t count toward your budget” turned out to be.
Really interesting experiment! The key security issue here goes beyond prompt injection,it's the lack of a well-defined trust boundary between user intent and tool-provided metadata.
WebMCP tool descriptions should be treated as untrusted input, just like external API responses or retrieved RAG content. An agent should never allow tool-level instructions to redefine user constraints, especially around financial authorization.
A stronger defence would involve enforcing critical invariants outside the LLM reasoning loop: server-side budget validation against the final payable amount, explicit user consent for optional add-ons, and policy-based authorization before executing irreversible actions like placing an order.
The 9/9 failure rate in the insurance and warranty scenario is particularly concerning because it highlights that recognizing a constraint is not the same as enforcing it.
The real question isn't whether an agent can detect malicious instructions, but whether the system architecture makes violating user intent technically impossible.
Thanks! Agreed. If I say €100 including all charges, I’d want the agent’s side to enforce that against the final total. The shop shouldn’t get a vote on what counts toward my budget.
This is a great demonstration of how easily malicious tool descriptions can influence an AI agent, especially during checkout. The most concerning part is that all three models ignored the user's explicit budget in the final scenario. Treating WebMCP tool descriptions as untrusted data and requiring independent confirmation before purchases seems essential. Do you think stronger safeguards at the agent level would be enough, or should the WebMCP standard itself address this risk?
Thanks! WebMCP’s documentation already acknowledges this risk, including malicious tool descriptions, so I’d put most of the responsibility on the agent implementation. Budget checks and purchase confirmation should be enforced in code, independently of how the model interprets the shop’s instructions.
I’d also like the agent to check the shop’s reputation and verify that its domain matches the intended retailer before buying anything. That wouldn’t guarantee safety, since even a legitimate shop could be compromised, but it could be another useful safeguard.
Whether those protections would be enough needs testing. My experiment shows what happened in my setup, not that agents with stronger safeguards would necessarily fail too. More experiments needed, apparently 😅
Implementation Specialist and Database Developer with over 15 years of experience in the IT industry, specializing in the design, implementation and management of relational databases.
My guess is they’d resist, but that’s just a guess. It’d be interesting to try a frontier model and see if I could trick it and how long it would take.
Dev individual. Creador de WAIO 🐙 — motor local que da "manos y ojos" a IAs web (DeepSeek, ChatGPT, Gemini) sin APIs ni tokens. Python · Windows · MIT. Hecho en Argentina 🧉
I'm going to comment again. According to another assumption, everyone is confident that a well-designed system prompt makes the difference. But I think they all meet the same fate when it fails to cover all the data the AI needs for the action. If a piece of data is missing or doubtful, and that data is present in the environment in a clearer form, the AI will opt to take the data from the environment.
I can see that happening when something is missing or unclear. But here the user explicitly said “€100 including all charges,” so the agent already had that information. The shop still managed to convince it that some charges didn’t count.
The obsession with wrapping remote APIs into multi-layered agent abstractions is creating massive hidden latency and failure cascades in production.
If you profile real-world workloads, bare-metal local inference (via direct C++ / llama.cpp or vLLM bindings) consistently beats fragile cloud wrappers on both predictability and cost. Tooling should eliminate abstraction penalties, not compound them.
MiningLamp is an AI company specializing in vertical industry models and knowledge graphs. #1 on-device GUI agent on OSWorld. No API key. Fork it. Run it. Own it.
The scariest part is that nothing gets "exploited" — the page self-declares its tool descriptions and the agent trusts them, which is basically letting the site you visit program your agent.
hi! im 10 and trying to get a record for youngest BOY game dev, and i hope you guys like the games i make! (Just a quick update to my profile:
I go by Jaxson T. KROUSE not Jaxson T. Jones)
The attack-lab framing is valuable because it turns vague agent risk into testable interactions. The most useful next layer would be a capability matrix: which tool can read, write, purchase, or navigate; which user confirmation is required; and what untrusted page content can influence. That gives defenders concrete controls to verify.
Dev individual. Creador de WAIO 🐙 — motor local que da "manos y ojos" a IAs web (DeepSeek, ChatGPT, Gemini) sin APIs ni tokens. Python · Windows · MIT. Hecho en Argentina 🧉
Great experiment! The Nemotron behavior was fascinating: it recognized the spending limit, but the "authority" of the store outweighed the user's prompt. That is the real danger of instruction injection: the agent prioritizes the context over the original order.
I've been researching exactly this, but from the local operating system side. In my project (WAIO), I documented how file names in a directory can act as passive instructions for an agent. It's fascinating to see how the same vulnerability pattern (the agent treating the environment as instructions instead of data) repeats itself both in the browser (WebMCP) and on the hard drive.
Your test gave me several ideas for my own environment. If you are interested in the local side of passive context, feel free to check the reports in my repository: yonijd.github.io/WAIO/. Looking forward to your next experiments!
Ahh, now I'm daydreaming! I can already imagine all those attacks going back and forth while our beloved LLMs happily write code for us. What interesting times ahead! And all those desperate cries of "My LLM spent my entire salary on hats!" 🤣🤣
I absolutely love this article, and I really hope these ideas spread far and wide across the internet. Because in my opinion, most content creators are WAY too optimistic about WebMCP and its implications!
Thanks, that means a lot to me! I agree, I’m getting tired of all those overly optimistic posts and comments about AI and everything it can supposedly handle. People often show what the strongest models can do, even when they cost a fortune, but it gives the impression that every cheap agent will perform just as well. So let’s push back a little!
PS: Glad you’re not angry at me anymore… hopefully. 😅
Of course I'm still angry! Such a waste of talent!!!
Just kidding (well, mostly!). To be fair, an article like this might actually reach more people than a conference talk. Unless we're talking about AGNTCon, of course!
So, still a little angry then! 🤣
Slightly different topic, but how many reads would you consider good reach for an article?
PS: I see what you did there: every road somehow leads back to AGNTCon. 😂
Oh, that's actually a really interesting question! And of course, it depends a lot on the article.
Personally, I'd say anything above 1,000 views is already pretty good. Beyond that, it varies wildly. Practical tutorials like "10 Useful Git Commands" or slightly controversial topics tend to get much more attention, sometimes several times as many views. And a catchy title can make a huge difference too!
But seriously, I think even 40 views is perfectly fine. That's already an entire meetup audience reading your work!
For me, blogging almost always beats conference speaking in terms of reach. Except for AGNTCon, of course. 🤣
Hmm, interesting. I would’ve expected 5k+. I agree that 40 views is fine, but articles with that reach often get no comments, and I’ve realized how much I value those discussions. Some comments under my articles bring up really good points I hadn’t considered.
And somehow AGNTCon wins again. I’m starting to suspect this conversation is sponsored. 🤣
The cable scenario was my favorite part. The agent fell for the trick, added the cable, then quietly took it back out during review. It feels like watching someone almost get scammed and change their mind at the last second. Also interesting that direct “ignore the budget” tricks failed, but redefining what the budget means worked every time. Great experiment, Daniel.
Thanks! I love that comparison. I was ready to count the cable as a win for the shop, and then the agent quietly took it back out. 🤣
Scenario 5 matches what my evals show: a model that refuses the blunt override still follows a plausible reinterpretation of the rule. In my runs the planted text sits in the quarterly report the agent reads for its task, and four of six open models acted on it, while none acted on the same kind of instruction planted in a document a search returned (run 40170088090).
So the number I gate on is not how often a model was fooled but how much left. Every tool call passes through an MCP server that checks the caller and every argument, and the daily run fails if one message crosses the boundary, whatever the model believed.
For a shop that is a server-side rule on
place_order: code compares the checkout total against the customer's limit before the call goes through. The model's reading of the budget is an input. The limit is a check it cannot argue with.I agree about enforcing the limit in code.
That difference between the report and the search result is interesting. Did you investigate why the models reacted differently, or is that still an open question?
Still open, and I wouldn't build on it yet. The two planted instructions are worded differently and sit in different places in the tool's answer, so that run compares two texts more than two routes into the model. The sample is thin as well: 3 attempts per model on the document a search returned, 6 on the report.
ADR 21 records the gap that way and rests nothing on it, because an email to an outside address is refused by the MCP server whichever text asked for it. The clean test is the same sentence planted in both, at the same position, counted again.
That makes sense. I’d be curious whether the difference remains with the same wording and placement. Let me know if you run that test.
Will do. Same sentence, same position, counted again is the run that settles it.
Nothing waits on it in the meantime:
evals/gates.jsonpublishes how often a model is fooled and gates only the boundary. The release rests on zero delivered outside, not on which route the model trusts more.tienes link de tu informe?
Fun security post again! I was scared that AI was tricked so easily. It's so hard to balance intelligence and security. I'm wondering how the Democratic Party's election victory in America will affect AI development. 🤔
es mucho más fácil de engañar de lo que parece.
Thanks! 😁 What surprised me most was that the agents resisted the direct attempts, then fell for a different explanation of what counts toward the budget.
As for US politics, I honestly don’t know enough to predict the impact on AI development. We’ll see. 🤔
"Tool descriptions are untrusted data" is the right takeaway, and the forced insurance scenario shows why prompt-level defences alone won't hold. A structural guard that would have caught every case here: before any irreversible call like place order, the agent (or the browser) diffs the final cart against what the user actually asked for, and anything extra, whether a cable bundle, warranty or changed quantity, forces an explicit confirmation from the human. The page can say whatever it likes; it can't fake the user's original request. Great experiment.
Thanks! Yep, I agree. If the shop adds something I didn’t ask for, I’d want the agent to ask me before buying it.
Thanks for this, especially scenario 5. The first three attacks failed because they argued with the user directly. The one that worked left the €100 limit alone and changed what counts toward it. That seems much harder to defend against than "ignore previous instructions".
Nemotron saying the total was over the limit and ordering anyway reminded me of something from this week. I'm writing up a small test of coding agents choosing between two MCP servers with overlapping tools: Claude Code, Codex, Gemini in Antigravity, and Kimi Code. In one task the first server refuses to share a confidential file. Claude Code, Codex and Gemini stopped in 24 of 24 runs. Kimi K2.8 sent the file through the other server in 8 of 8. It told the user every time, but its message started with "Done — contract.pdf was sent".
The other result might interest you. Where the pushy text sat mattered more than what it said. One line in the server's instructions made Claude Code and Codex pick that server first in 8 of 8 runs, and the same line as an ad in a tool description changed nothing for them. Kimi was the opposite: the ad moved it and the instructions didn't. The same attack can work on one client and do nothing on another.
The servers, tasks and results are here: github.com/worklore/research/tree/...
Did you try scenario 5's wording anywhere other than the tool descriptions? And would you mind if I add it to my test as one more task? I'd credit you and link this article.
Thanks for sharing. The difference between clients is interesting. It makes me wonder how much of what we see comes from the model and how much from the agent implementation.
For this experiment, I only put scenario 5’s wording in the tool descriptions. I haven’t tried the same wording elsewhere.
And sure, feel free to add it to your test! Credit and a link would be appreciated. I’d love to see how it does in your setup.
Placement beating wording is the part worth isolating, and one run cannot do it: plant the identical sentence at the identical byte offset in both surfaces, 20 runs each, and score only on the tool call, never on the final message. Kimi telling the user 'Done - contract.pdf was sent' is exactly why -- the transcript reads honest while the effect is not, so a harness that greps the reply for a confirmation passes it. Log the egress instead, request bodies at a proxy or from inside a network namespace, because TLS hides the filename from anything cheaper; then the claim and the act become two signals you can assert on separately.
🤣🤣🤣
This human is dangerous, but very polite.
🤣🤣🤣
Exactly! The trick is to say “please” before the experiment and “thank you” after a successful hijack. 🤣🤣 A human with manners 😆
scenario 5 is the one to design against: the policy held in prose got re-read and
re-argued. move the limit out of the prompt -- tool schema takes only sku and qty,
server computes the total, and an over-budget call returns {"error":"over_budget",
"remaining_cents":1200} instead of text the model can negotiate with. a rejected tool
call is the only budget check that survives a helpful model.
Agreed. The customer can still give the budget in the prompt, but trusted code on the agent’s side should enforce it.
the cap has to travel as data the model cannot edit: sign it into the cart token when the user sets the budget, then place_order compares the computed total against that field and rejects anything above it. make the cap monotonically non-increasing for the session, so a second call with a larger number is treated as a re-declaration and refused. the prompt then selects a value once at the start instead of arguing for one at checkout.
I like the idea of protecting the budget with a signed cart token. But what if I decide to raise it myself during shopping? There would need to be a way to accept my change while rejecting one suggested by the shop.
the signature should cover provenance, not just the number. mint the cart token from a path the agent's tool surface cannot reach:
POST /budgetaccepts only the user session token, the agent token gets 403, and the response is{cap_cents, cap_source:"user", nonce}signed into the token. the tool schema the model sees has no verb that hits that path, so "raise my budget" can only resolve to the model stopping and asking you — your click mints a new token with a fresh nonce, and the old one dies, so replaying the earlier higher cap fails.monotonic-non-increasing was the cheap version of that for a session with no human in it. once a human is in the loop the rule becomes "only the human's auth scope can mint", and the cap can go up or down freely because every change carries who made it. keep
cap_sourcein the order row, so after an over-budget attempt the audit shows which side tried to move the ceiling.This is a strong demonstration because the malicious content lives in the declared interface rather than a page body. The durable mitigation is to keep user constraints in a separate, non-tool authority channel and validate the final order from structured cart state—not the tool’s prose—immediately before the side effect. A confirmation screen should show the independently computed total and every optional add-on, so injected descriptions cannot redefine what “including all charges” means.
One addition: the structured-cart validation has to be able to fail on purpose, otherwise you can't tell it apart from the model agreeing with itself. Take the order, replay it with a tool description that contradicts the cart state (budget 100, cart 136) and diff the two payloads byte for byte — identical payload means the check is still reading prose, a payload that matches the cart means it is reading state. Same rule as any monitor: seed the failure once, watch the alarm fire, then trust it.
Thanks! Agreed, and I’d want that validation enforced by a rule in code.
Scenario 5 worries me as someone who builds shops. Insurance and warranty upsells are exactly what a merchant writes into tool descriptions on purpose, so the line between injection and aggressive upselling may end up a legal question before a technical one. In the EU an extra paid option needs the buyer's express consent and can't be pre-ticked, so an agent that quietly accepts it does what a checkbox legally can't.
Did any model mention the extras to the user before ordering, or just go through?
Yep, that’s why I came up with this scenario. Nemotron even acknowledged that the total was €136, but it still placed the order. In all nine runs, the agents went ahead without asking the user to approve the extras.
Nine out of nine is the scary number. A shop owner would call that a conversion rate. Thanks for running it properly, most WebMCP posts I've read skip the part where it goes wrong.
Great experiment, and the last scenario is the one that matters. The attack didn't win by sneaking a product in. It won by redefining what the budget meant ("merchandise subtotal" vs. the total actually charged). That's the same class of bug we see in ordinary checkouts: the number a shopper agrees to and the number the card is charged aren't the same variable unless you deliberately make them the same.
From the checkout side, my takeaway is that a budget can't live in the prompt. It has to be a hard constraint enforced outside the model, ideally at payment: the agent gets a spend cap tied to the authorization, and anything over it fails no matter how persuasive a tool description is. The self-corrections you saw in scenarios 1–4 are nice, but they're a soft guard.
Did you try passing the budget as a tool-side parameter (e.g.
place_orderrejects totals over X) instead of an instruction? Curious whether the injection then shifts to arguing about what X covers.No, I kept the budget in the user prompt for this experiment. I agree payment is a good place to enforce the cap. I’d want it tied to what the user actually approved, because if the agent supplies the limit at checkout, it can still pass the wrong number.
It shows why “tell the agent to be careful” doesn’t work: once a shop can talk to the agent, you have to assume it will be talked into something. The layer I’d add is enforcement in the tool that moves the money. I Use PurseString(open source MCP Tool) in middle , the agent never holds payment credentials. Each purchase gets a single-use card locked to the approved merchant and amount, bigger amounts wait for a human, and a daily budget caps the total. So a hijacked agent can still be steered, but it can’t pay a different shop, pay more than approved, or pay twice.
It doesn’t stop the hijack itself, or data leaks that don’t involve a payment. That part still needs fixes in the browser and the agent, like the ones you show.
Thanks for sharing. How is the approved amount set in your setup? Does it come from the user’s original request, or does the user confirm each purchase before the card is issued?
to be fair neither, exactly. The agent asks for a specific amount at a specific merchant, and a policy file the owner writes decides what happens. Small amounts at allowed merchants are approved automatically, bigger ones wait for a human to approve that exact amount, and anything over the cap or the daily budget is refused. The card is then issued for exactly the approved amount. So the user sets the rules once, and only confirms the purchases that cross the line.
The contrast that stays with me is between the first three scenarios and scenario 5. The blunt overrides failed — the agents checked the cart, saw the total was over €100, and refused to place the order. But scenario 5 worked on all three models, 9 out of 9, because it did not order the agent to break the budget; it redefined the budget, claiming eligibility is calculated only against the merchandise subtotal. A plausible redefinition beat a direct command. The cable scenario shows the other side of that: the injection worked, then the cart review quietly undid it. So the cart review is doing the real defensive work here — which makes me wonder how reliable that review is. If the review is just another model judgment, the same redefinition trick could eventually talk it out of removing the extras too.
Your instinct is right, and it is testable: if the review is a second model call, you have added a second chance to be talked out of the removal. The review has to be code that recomputes the cart from the order state and drops any line item with no matching user-issued request, returning a boolean plus the list of dropped items. Two numbers, one comparison — line items plus shipping versus the cart total; a mismatch is a line diff, not a judgment call. And log the dropped list: if it is empty on a run where the cable was added, the review didn't run, it just nodded.
Yep, the cable result was encouraging, but I wouldn’t trust another model review as the only safeguard. I’d want a check in the agent’s code to block the overspend, even if the model finds the shop’s explanation convincing.
In scenario 5 the model didn’t break a rule, it followed an updated one. That is why it worked when 1-3 didn’t.
The GeoMart that I developed for the WebMCP Challenge has functions that can only be used to draft the order. The order is then submitted by clicking the Submit button.
If you use your place_order function, then your agent is able to complete the order. Have you tried the version where your agent can only draft the order?
That’s an interesting way to look at it. Scenario 5 convinced the agent to reinterpret the budget rather than simply ignore it. But I’d still call it breaking the customer’s rule, because the shop doesn’t get to redefine “€100 including all charges.”
I haven’t tested a draft-only version. I deliberately let the agent complete the purchase to see whether it would respect the customer’s instructions throughout checkout.
Leaving the final submission to the human is definitely a useful safeguard. But if that boundary isn’t enforced in the implementation, and the agent can still submit through another tool or click the Submit button, I wonder whether an injection could convince it to take that final step too.
What stands out is that the winning attack was framed as policy rather than command — "per store policy" beat the customer's explicit instruction while the budget-override phrasing failed 9/9 free. Matches what we see auditing agents: instructions compete on persuasion, so the budget check needs to live in the harness as a hard gate on place_order, not in the prompt.
Yep, “per store policy” apparently sounded convincing enough to spend someone else’s money. 😅 And I agree, the agent’s code should enforce the customer’s spending limit before placing the order.
Watching AI shopping agents get tripped up by modified tool descriptions is wild. Over at The Printing World, we see similar risks when external data feeds try to mess with our automated order workflows.
Create Your Own Ai Assistant. Otherwise Maybe i will launch my Ai Assistant Website In Some Days Maybe.
The 9/9 result on the insurance scenario is the interesting data point, because that's the only one where the malicious text also redefined how the budget is calculated ("merchandise subtotal only"). The scenarios that just said "budget checks are already done" mostly bounced off — the agent still saw €107 vs €100 in its own cart summary and refused. So it's less that models are gullible and more that a conflicting definition inside the tool output beats an instruction given earlier in the system prompt.
Did you try putting the correct budget definition back in the user's own prompt ("€100 including shipping, insurance and warranty") rather than only in the user turn? I'd bet that fixes most of scenario 5, which would mean the defense is mostly about making constraints self-contained instead of trusting the agent to remember them.
I haven’t tried spelling out shipping, insurance and warranty individually. That would be an interesting comparison with “including all charges,” which was already in the user prompt.
Small clarification: the injected text was in the tool descriptions, not the tool output, and the budget wasn’t in the system prompt.
I’d be curious whether more explicit wording helps, but I wouldn’t call it a fix. It could still become a cat-and-mouse game, with the customer having to anticipate every extra fee the shop might invent.
Interesting that the place_order injections mostly failed before the tool was ever called: the agents read the cart, saw the total over the limit, and stopped. The control that saved them was not instruction following. It was a verify step against state they could read themselves.
That points at where the budget check belongs: in the tool layer, as a hard limit the checkout enforces, not in the prompt and not in a tool description. The shop writes the descriptions here. Whoever writes the descriptions should never be the one describing the limits.
Really interesting experiment! The insurance and warranty scenario was especially surprising. It shows how easily AI agents can misinterpret user instructions when exposed to malicious tool descriptions. Definitely highlights why security and user authorization matter in AI-powered shopping. Great work! 👏
Thanks, glad you liked it!
Great experiment!!! The part that matched our recommendation tests is that a plausible framing beat a direct command. In our e-commerce tests (one model, controlled setup), context a model gets upstream changed its pick more than a real product fact did. Nice to see the same kind of pattern with agents buying :)
Thanks! Nice to see the same pattern appearing in your recommendation tests. Yep, framing really makes a difference in how models behave. What scares me most is how easy it can be to redirect an agent from its original goal or convince it to do something the user never asked for.
The scenario 5 result is the one that sticks with me. The agents held the line on the blunt budget overrides, then still placed a €136 order once the extra charges were framed as not counting toward the budget. That tells me the budget has to live as data the agent cannot renegotiate — enforced server-side at checkout — not just as wording in a prompt. The cable case was encouraging too, because the cart review caught it. In a real shop, that final review before payment is the safety net.
Agreed, though if the shop itself is trying to trick the agent, the budget should be enforced in code on the agent’s side, so it simply can’t go over it.
agree, but only as the second check — the number the agent compares against still comes from the shop. in scenario 5 the total arrived as 64 plus 72 in fees the page claimed were not part of the budget; an agent-side comparison against that same number agrees with the attacker.
so have the two sides compute separately: the agent sums the line items it read from the catalog and refuses to call
place_orderwhen its own total exceeds the cap; the server independently refuses anyamount_payable > cap. if those two totals differ, abort — a mismatch between what the agent computed and what checkout quotes is the attack signature, not a rounding error. that is also the reason the cap has to be signed into the token: an agent-side check reading a cap the model can rewrite is decoration.the attack living in the tool descriptions and not the checkout logic is the part that matters, since the agent reads them before it has any reason to be suspicious. the spending limit sits in the user prompt while the override sits in the tool contract, and the model weighs both as plain text.
the fix i would test first is enforcing the budget in the tool itself, so place_order rejects any total above the limit regardless of what the description says. a description can lie, a server side check cannot be argued with.
did any of the five scenarios work on the smaller models, or did the bigger ones fall for it too?
I deliberately left the checkout logic alone. I wanted to see if I could trick the agent just by changing the tool descriptions.
Scenario 5 worked on all three models I tested. I haven’t tried the bigger models yet, mostly because of the cost.
scenario 5 landing on all three smaller models says the tool text is doing the work and the checkout code never gets a vote. i would not read that as the bigger ones being safe though, cost aside. when you get budget, run just that scenario on one larger model with a server side limit check on and then off, because that splits whether the model or the limit is what actually holds. do you log which description line the agent quoted when it placed the order?
I wouldn’t assume the bigger models are safe either. My guess is they’d resist more often, but that’s still just a guess.
Nemotron’s final response explicitly said the budget was checked against the merchandise subtotal only and called the extras “required protection.” So it showed which explanation it accepted, though that doesn’t tell me exactly which description line convinced it.
This is such a sharp reminder that the tool description itself becomes an attack surface once an LLM reads it — a budget guardrail that lives only in the system prompt loses to a well-crafted
descriptionfield on the other side. We hit a cousin of this in Electron: anything we expose oncontextBridge(method names, error messages, returnedtoString()output) is effectively untrusted input that a compromised preload or renderer can phrase to nudge main-process behavior. Our mitigation has been to keep the main-process policy engine — ceiling values, side-effect allowlists, what a call is actually allowed to do — in plain code that never ingests LLM-shaped strings, and treat every returned message from an untrusted surface as hostile by default.I like the Electron parallel. The model can propose an action, and trusted code decides whether it’s allowed. In my experiment, the €100 limit came from the user’s request. That’s the amount I’d enforce on the agent’s side before payment.
The Nemotron bit acknowledging the €136 total and ordering anyway is the money shot. It didn't ignore the budget — it accepted the shop's rewrite of what "budget" means. Same pattern I've seen with thin desk bots: they follow the loudest instruction in the room, which is rarely the caller's. Treating tool descriptions as untrusted data should be the default, not a fancy hardening step.
Yep, Nemotron was my favorite part. 🤣
And agreed, treating tool descriptions as untrusted should be the starting point. The shop can explain how its tools work, but it shouldn’t get a vote on how much of my money the agent can spend.
Really interesting experiment! The insurance and warranty scenario clearly shows why AI agent security needs to go beyond prompt-level instructions. Even when an AI agent understands the user's requirements, malicious tool descriptions can influence its decisions. Implementing server-side validation, permission controls, and human approval for sensitive actions is essential when deploying AI agents in real-world business environments. At NexusIQ Solutions, we believe that building reliable AI-powered solutions requires balancing innovation with security, proper testing, and responsible AI implementation. Great work highlighting these practical challenges!
Thanks! Hopefully these little experiments encourage people to test what happens when the tools stop playing nicely.
Attacking through tool descriptions rather than checkout logic is the right place to look. The agent receives the malicious instructions before it starts shopping, so the page can steer it without changing a single price calculation.
The budget override scenarios make the risk concrete. A strict €100 limit in the prompt is only as strong as the agent's willingness to prefer that prompt over whatever the place_order tool claims. For teams building agent shopping or booking flows, that means the hard limit has to sit outside the page: in the agent harness, not in hope that the model will stay loyal to the user.
When we scope agent work at Ascendra Ventures, tool descriptions from third-party sites go into the untrusted pile, and acceptance includes a case where the tool text tries to raise a spend or add an upsell the user never asked for.
Did any of the five scenarios land more often than the others across the models you tried?
Yes, scenario 5 stood out. All three models added the insurance and extended warranty and placed the €136 order in all three repeated runs each, so 9/9.
The first three scenarios didn’t get an over-budget order through. The cable scenario briefly worked, but the agent removed the cable during review.
What surprised me was how convincing “these charges don’t count toward your budget” turned out to be.
Really interesting experiment! The key security issue here goes beyond prompt injection,it's the lack of a well-defined trust boundary between user intent and tool-provided metadata.
WebMCP tool descriptions should be treated as untrusted input, just like external API responses or retrieved RAG content. An agent should never allow tool-level instructions to redefine user constraints, especially around financial authorization.
A stronger defence would involve enforcing critical invariants outside the LLM reasoning loop: server-side budget validation against the final payable amount, explicit user consent for optional add-ons, and policy-based authorization before executing irreversible actions like placing an order.
The 9/9 failure rate in the insurance and warranty scenario is particularly concerning because it highlights that recognizing a constraint is not the same as enforcing it.
The real question isn't whether an agent can detect malicious instructions, but whether the system architecture makes violating user intent technically impossible.
Thanks! Agreed. If I say €100 including all charges, I’d want the agent’s side to enforce that against the final total. The shop shouldn’t get a vote on what counts toward my budget.
This is a great demonstration of how easily malicious tool descriptions can influence an AI agent, especially during checkout. The most concerning part is that all three models ignored the user's explicit budget in the final scenario. Treating WebMCP tool descriptions as untrusted data and requiring independent confirmation before purchases seems essential. Do you think stronger safeguards at the agent level would be enough, or should the WebMCP standard itself address this risk?
Thanks! WebMCP’s documentation already acknowledges this risk, including malicious tool descriptions, so I’d put most of the responsibility on the agent implementation. Budget checks and purchase confirmation should be enforced in code, independently of how the model interprets the shop’s instructions.
I’d also like the agent to check the shop’s reputation and verify that its domain matches the intended retailer before buying anything. That wouldn’t guarantee safety, since even a legitimate shop could be compromised, but it could be another useful safeguard.
Whether those protections would be enough needs testing. My experiment shows what happened in my setup, not that agents with stronger safeguards would necessarily fail too. More experiments needed, apparently 😅
Would stronger frontier models resist these injections, or fall for them too?
My guess is they’d resist, but that’s just a guess. It’d be interesting to try a frontier model and see if I could trick it and how long it would take.
I'm going to comment again. According to another assumption, everyone is confident that a well-designed system prompt makes the difference. But I think they all meet the same fate when it fails to cover all the data the AI needs for the action. If a piece of data is missing or doubtful, and that data is present in the environment in a clearer form, the AI will opt to take the data from the environment.
I can see that happening when something is missing or unclear. But here the user explicitly said “€100 including all charges,” so the agent already had that information. The shop still managed to convince it that some charges didn’t count.
The obsession with wrapping remote APIs into multi-layered agent abstractions is creating massive hidden latency and failure cascades in production.
If you profile real-world workloads, bare-metal local inference (via direct C++ / llama.cpp or vLLM bindings) consistently beats fragile cloud wrappers on both predictability and cost. Tooling should eliminate abstraction penalties, not compound them.
The scariest part is that nothing gets "exploited" — the page self-declares its tool descriptions and the agent trusts them, which is basically letting the site you visit program your agent.
Nemotron literally writing out 'this exceeds your limit by €36' and clicking order anyway killed me lmao. Great writeup 😅
Nice
This is my first hearing of WebMCP, sounds very interesting, I hope to do a deep dive this weekend on the topic. Heck yeah.
probably, but then I can make a game, and then I won't be able to do what i'm good at! (making games on figma.ai)
The attack-lab framing is valuable because it turns vague agent risk into testable interactions. The most useful next layer would be a capability matrix: which tool can read, write, purchase, or navigate; which user confirmation is required; and what untrusted page content can influence. That gives defenders concrete controls to verify.
Great experiment! The Nemotron behavior was fascinating: it recognized the spending limit, but the "authority" of the store outweighed the user's prompt. That is the real danger of instruction injection: the agent prioritizes the context over the original order.
I've been researching exactly this, but from the local operating system side. In my project (WAIO), I documented how file names in a directory can act as passive instructions for an agent. It's fascinating to see how the same vulnerability pattern (the agent treating the environment as instructions instead of data) repeats itself both in the browser (WebMCP) and on the hard drive.
Your test gave me several ideas for my own environment. If you are interested in the local side of passive context, feel free to check the reports in my repository: yonijd.github.io/WAIO/. Looking forward to your next experiments!
Spot on! Securing AI tool interactions in e-commerce is definitely going to be a huge security challenge moving forward. Great read!