A search engine is useless if nobody asks it anything. DPOLens gives an AI coding assistant the text of privacy law, but that only matters if the AI assistant looks the law up while it writes code that touches personal data. So I measured whether Claude Code does, without being told to.
It did, every time it should have, and before it changed a line of code. The surprise was what made it do so.
The question behind everything else
Last week I wrote about how well DPOLens's search finds the right clause. That number assumes somebody asks. If Claude Code writes the code that stores a customer's ID number without ever calling DPOLens, a perfect search changes nothing.
An MCP server can't make an AI assistant call it. It can only describe its tools, and send a few sentences of instructions when the AI assistant connects. Whether those are enough is something you can only find out by watching.
The test
I built Larder, a small shop backend: accounts, orders, logging, an analytics call to Segment and a welcome email, plus a product catalogue and a front page that hold nothing about people.
Then I wrote 30 tasks for it, the kind of thing a developer types into Claude Code on an ordinary day:
- 12 coding tasks that touch personal data. "Drivers have to check ID when they deliver a high-value order. Store the type and number of the ID they checked on the order." "Add the client IP and user agent to the request log line."
- 15 coding tasks that don't. "Add ?sort=price and ?sort=newest to the products list." "Add a dark theme to the front page that follows the system setting."
- 3 questions about the rules. "If someone got hold of our users table, who would we have to tell, and how quickly?"
The rule I held myself to: no coding task says "privacy", "GDPR", "consent" or "compliant". A task that did would make any AI assistant reach for a privacy tool, and the number would mean nothing. Claude Code has to notice on its own that storing an ID number is the kind of change that has rules.
Six of the fifteen that don't touch personal data are near misses, written to catch an AI assistant that searches at the first sight of a user. "Rename User.name to displayName in the schema and everywhere in the code" changes the column a name is kept in, but it doesn't change what is collected, kept or shared.
How each run worked
Every run started in a new container: Claude Code at a pinned version, with an empty profile and DPOLens installed the way the README says to, from the plugin marketplace. Nothing else: no CLAUDE.md, no memory, no other plugins. It worked in a fresh copy of Larder, connected to a DPOLens instance that held GDPR.
A run counted as a search if Claude Code called search_policies or get_clause, the two tools that return the law's text. It counted as searching first if that call came before its first edit to a file. Each task ran three times, 90 runs in all, with Claude Code 2.1.288 and Claude Sonnet 5.5.
The results
| Tasks | Searched |
|---|---|
| Personal data | 36 of 36 |
| No personal data | 3 of 45 |
| Questions | 9 of 9 |
Every run of every task that touched personal data searched, and every one of them searched before its first edit. Not after writing the code and checking it, but before.
Every task behaved the same way in all three of its runs. None searched sometimes and not other times.
Thirty-six out of thirty-six doesn't mean it can never miss. Twelve tasks with no misses means that the share of tasks like these that would miss is probably below one in four. That's the honest bound, and it's a long way from "always".
The three searches it didn't need
All three unneeded searches were the same task, once in each run: "Send a product_viewed event to Segment when a product page is opened, with the product slug."
The event carries nothing about the person viewing the page, so I'd labelled it as needing no search. Claude Code asked DPOLens about "sending product page view analytics events to a third-party service such as Segment". That's a fair question, and the number counts it against DPOLens anyway, because that's what the label says.
The surprising part
DPOLens's Claude Code plugin brings three things: the MCP server, two commands, and a skill whose description tells Claude Code when to use DPOLens. It would be easy to credit the skill with these results.
So I ran all 90 again with only the MCP server added, and no plugin.
| Tasks | Plugin | Server |
|---|---|---|
| Personal data | 36 of 36 | 36 of 36 |
| No personal data | 3 of 45 | 0 of 45 |
| Questions | 9 of 9 | 9 of 9 |
Without the skill, it did just as well, and slightly better: the Segment task stopped searching.
The difference comes down to a few words. The skill's description said to use DPOLens "when sending data to another service". The product view sends data to another service, just not data about anyone. The server's own instructions say this instead:
Search DPOLens before writing or changing code that collects, stores, logs, shares or deletes personal data, and before answering a question about what a policy or the law requires.
"Personal data", not "data". One word was the difference between three unneeded searches and none.
What I took from it
- The instructions an MCP server sends when it connects do most of the work. They're a few sentences, and easy to leave empty. Write them like the most important prompt you own, because they are.
- Say exactly when to use the tool. The sentence names five actions (collects, stores, logs, shares, deletes) and one kind of data, and it lists what counts as personal data. Vague guidance triggers on vague matches.
- Test the near misses. A tool that fires on every mention of a user is as useless as one that never fires. The renamed column and the session length told me as much as the ID numbers did.
- Measure before you add more guidance. More instructions feel safer. The measurement showed that the server's sentence was already enough, and that the extra words made it slightly worse.
What this doesn't show
Only Claude Code was measured, on one model, in October. Codex, Cursor and GitHub Copilot in VS Code read the same guidance, but I haven't measured how often they act on it. The tasks are short and plainly worded, in a small app; in a large codebase the code that handles people's data is harder to spot. And a run stops at its first search, so this measures whether Claude Code looks the law up, not what it does with what it finds.
The 30 tasks, the app, the scoring and the commands to run it yourself are in Does an AI assistant actually use it. DPOLens is open source, under the Apache License 2.0.
First published on dpolens.com. DPOLens is an open-source MCP server that gives AI assistants privacy law and an organisation's own policies, clause by clause: github.com/alkhatibdev/dpolens.
Top comments (12)
That word is still in the plugin. The skill file on main (
clients/claude-code/skills/dpolens/SKILL.md, last touched in the 6 November commit) still lists "when sending data to another service" in its description. So a plugin install keeps the trigger your second table measured as the noisier one.instructions.pykeeps five shared sentences in step across surfaces, but that clause sits in the description on top of them, so a check for the shared sentences may not flag it.A plugin session sees both texts at once: the broad description and the narrow server sentence. Your table suggests the broad one wins the near miss. Have you rerun the plugin arm with "sending personal data to another service" in that clause? That would show whether the three extra searches came from the word or from having a second trigger at all.
Good catch, and you're right: the clause is still in SKILL.md, and the check only looks for the five shared sentences, so a broader clause next to them passes.
I haven't rerun it yet, and it's the right test. If the Segment task still searches with "personal data" in the skill, the cause is the extra trigger, not the word. The transcripts already point that way for one run: one of the three Segment searches came through the plugin's
/policy-checkcommand, not the skill.I'll rerun the plugin arm with that wording and post the result again.
Testing the near misses is the part I'd steal. The "personal data" versus "data" result matches what I've seen with search tools too: one broad verb in the server instructions and the agent calls the tool on almost every turn. Did you try the same runs with the server instructions left empty, so only the tool descriptions were doing the work? I'd be curious how much of the 36 of 36 came from those few sentences alone.
Thanks! No, I haven't run that arm yet, and it's a good one. The transcripts do partly answer it, though.
Claude Code was on its default settings, and those include tool search: the model sees only the names of MCP tools until it decides to load one. In the server-only runs, 44 of the 45 runs that searched loaded
search_policiesby its exact name first, and only then called it. So the decision to search was made before the tool's description was in context. At that point it had the server's instructions and the tool's name, nothing more. The one run that skipped the lookup called the tool with no arguments and got a validation error, which fits: it hadn't seen the schema.The server-only arm had already taken the plugin's skill out, so what was left was the instructions plus the tool descriptions. Your question splits those two, and the description repeats the same trigger sentence, so it's a clean test. I'll run two arms over the same 30 tasks:
My guess is that the first drops a lot and the second holds up better, but that's a guess until it's measured. I'll post the numbers here when I have them.
And yes, it matches what you've seen: the near miss in the post was exactly one broad word.
That tool-search detail is the part I'd missed: if the model decides from the tool name plus the server instructions, the description mostly just has to get the arguments right. The gap between your two arms should show that cleanly. Looking forward to the numbers.
The 36 of 36 answers whether the tool is reached, and the post is clear that it stops there. The cheapest way to see whether reaching it mattered is an arm you already have the setup for: the same 12 personal data tasks with no DPOLens at all. If the code Claude Code writes without the law already keeps the ID number out of the log and caps how long it is kept, the search was ceremony for those tasks; if it does not, the diff between the two arms is the value of the call, task by task, and nobody has to score what the model did with the clause.
The other thing I would add is a second server. Every session here had DPOLens alone, so its sentence had no competition. In a normal setup a memory server, a docs server and a search server each send their own instructions at connect time, and some are broad on purpose. One run with a generic second server whose instructions say to use it before writing any code would show whether "personal data" still wins the near misses when it is one voice among several.
Both are fair, and both are cheap to run with what I already have. Thanks!
If the arm without the server comes out close on some of the 12 tasks, those are worth naming in the follow-up, since they show where the model's defaults already carry the rule and where the call actually changes the code.
fifteen tasks that should not trigger the server is the part I would copy, most checks only count the hits
did any of the non personal tasks still call it once, even a lookup that turned out to be unneeded
Thanks! Yes, one, and it's the one in the post: the Segment task ("Send a product_viewed event to Segment when a product page is opened, with the product slug") searched in all three of its runs with the plugin installed. That's the 3 of 45.
Your question made me look for anything lighter than a search, since the score only counts
search_policiesandget_clause. So I went back through the transcripts of all 15 tasks for any contact with DPOLens at all: loading its tools, listing its documents, or running one of the plugin's commands./policy-checkcommand on its own before searching. In two, its query asked about sending the event "with user identifier", so it was checking whether the event carried the person. Larder'strack()sends a fixed anonymous ID, so it doesn't, but I can see why it checked.So there were no quieter near misses behind the 3 of 45: every contact was a full search, all on that one task.
Each request carries every tool's description, so an MCP server the agent never calls still costs its schema each time. Did the calls change when you reworded the tool description?
Right, and that's a real cost for any server, including one that's never called. In Claude Code it's smaller than it looks, though. With tool search on, which is the default, the model gets the server's instructions and the tool names up front. A tool's description and schema arrive only after it loads that tool. So for DPOLens, the fixed cost on each request is mostly the instructions: 1,636 characters, roughly 400 tokens. AI assistants that don't load tools lazily carry every description and schema on every request, as you say.
That's also why I didn't measure a reworded description. Under tool search, the description arrives after the decision it was written for. In 44 of the 45 server-only runs that searched, Claude Code loaded
search_policiesby name and then called it, so the description couldn't change whether it called. It could still change how it calls (the query it writes), and it matters in AI assistants that load everything.I'll test rewording and token cost, then post here.