DEV Community

jidonglab
jidonglab

Posted on

Pydantic Lax Mode Hid 689 Type Errors in My Claude JSON Outputs

My contract extraction pipeline had a 99.7% validation pass rate. Then a Node script summed the contract values and printed 01200.50300.

That's not a number. It's what JavaScript gives you when you add 0 + "1200.50" + 300. The amount had come back from Claude as a string, Pydantic had quietly accepted it, and I had stored the raw JSON. Pydantic lax mode is the default in v2, and it had been coercing my LLM's type mistakes into valid-looking objects for weeks.

So I re-validated all 4,000 stored outputs with strict mode switched on. 588 records failed. Here's what was hiding in them.

TL;DR

  • Pydantic v2 validates in lax mode by default. It turns "1200.50" into a float, "yes" into True, 12.0 into 12, and an integer like 2023 in a datetime field into a Unix timestamp.
  • Re-running 4,000 stored Claude outputs with model_validate_json(raw, strict=True) failed 588 records (14.7%) that lax mode had passed, with 689 coerced fields total.
  • Most coercions were harmless. 4 were silently wrong: year-only dates became 1970-01-01T00:33:43Z.
  • The real damage came downstream: I stored the raw LLM JSON, so a non-Python consumer saw strings where Python saw floats.
  • Fix: ConfigDict(strict=True), feed validation errors back for one repair call, and track the strict failure rate as a drift metric.

What is Pydantic lax mode?

Pydantic lax mode is the default validation behavior in Pydantic v2: when an input doesn't match the declared type exactly, Pydantic tries a well-defined set of conversions before failing. A float field accepts the string "42.5". A bool field accepts "yes", "no", "true", "1" and friends. An int field accepts 12.0 because it has no fractional part.

For form data and query strings, this is exactly what you want. Everything arrives as a string, and lax mode saves you a hundred lines of int(request.args["page"]).

For LLM output, it means you never find out whether the model followed your schema. You find out whether the model's output could be bent into your schema. Those are different questions.

The setup

The pipeline is boring on purpose:

  1. Pull a contract PDF, extract text.
  2. Send it to Claude with a prompt describing the fields and asking for JSON only.
  3. Parse with Contract.model_validate_json(raw).
  4. On success, store the raw JSON string plus a few columns in Postgres.

The model looked like this:

from datetime import datetime
from pydantic import BaseModel

class Contract(BaseModel):
    counterparty: str
    total_value: float
    term_months: int
    auto_renews: bool
    signed_at: datetime | None
Enter fullscreen mode Exit fullscreen mode

Over a few weeks it processed 4,000 documents. 13 failed validation, mostly amounts written as "1,200.50" with a thousands separator (lax mode won't parse that). I retried those by hand and felt good about a 99.7% pass rate.

Step 4 was the mistake. I stored the raw string for audit purposes instead of contract.model_dump_json(). The Python side never noticed, because Python only ever touched the validated object. The reporting script, written in Node, read the raw column directly.

How do I find out what Pydantic coerced?

Re-validate the same input with strict=True and diff the results. You don't need to change the model. model_validate_json takes a strict argument that overrides the model's config for one call:

from pydantic import ValidationError
from collections import Counter

coerced = Counter()
failed_records = 0

for row in rows:  # 4,000 stored raw outputs
    Contract.model_validate_json(row.raw)  # lax: passes
    try:
        Contract.model_validate_json(row.raw, strict=True)
    except ValidationError as e:
        failed_records += 1
        for err in e.errors():
            coerced[(err["loc"][0], err["type"])] += 1
Enter fullscreen mode Exit fullscreen mode

Every record that passes lax and fails strict relied on at least one coercion. Every error in the strict run is one coerced field.

The result from my 4,000 outputs:

What lax mode did Fields
Numeric string to float/int ("1200.50") 452
Bool string to bool ("yes", "no", "true") 163
Whole float to int (12.0 to 12) 70
Integer to datetime as Unix timestamp 4
Total 689

689 coerced fields across 588 records. 14.7% of my "valid" outputs weren't valid by the schema I'd written.

Why did my dates turn into 1970?

Because Pydantic lax mode reads an integer in a datetime field as a Unix timestamp. When a contract only said "signed in 2023", Claude sometimes returned "signed_at": 2023 instead of null or a full date. Pydantic took that as 2,023 seconds after the epoch:

>>> Contract.model_validate_json('{..., "signed_at": 2023}').signed_at
datetime.datetime(1970, 1, 1, 0, 33, 43, tzinfo=TzInfo(UTC))
Enter fullscreen mode Exit fullscreen mode

No error. No warning. A valid, timezone-aware datetime that is wrong by 53 years.

Only 4 of 4,000 records hit this. But those 4 sorted to the top of every "oldest contracts" view, and one of them was flagged by my renewal reminder as overdue by half a century. That's the scary category of coercion: not a type the model got slightly wrong, but a value whose meaning changed during conversion.

The other 685 coercions were technically correct conversions. "1200.50" really is 1200.5. "yes" really means True. Python was fine. The JSON on disk was not.

Is Pydantic lax mode wrong for LLM output?

Lax mode isn't wrong, but it answers the wrong question for LLM output. It tells you "this can be made into a Contract." For a model you're prompting, you want to know "the model produced a Contract." The gap between those two is exactly where model drift hides: a prompt change, a model upgrade, or a weird document that nudges the model from numbers to strings.

If you only ever consume the validated Python object, lax mode is mostly safe, apart from the timestamp trap. If anything else reads the raw output (a JS frontend, a data warehouse, a second service, an audit log someone will query later), lax mode lets inconsistent JSON leak out with a green checkmark on it.

How do I turn on Pydantic strict mode?

Set strict=True in the model config. Every field then rejects inputs of the wrong type instead of converting them:

from pydantic import BaseModel, ConfigDict

class Contract(BaseModel):
    model_config = ConfigDict(strict=True)

    counterparty: str
    total_value: float
    term_months: int
    auto_renews: bool
    signed_at: datetime | None
Enter fullscreen mode Exit fullscreen mode

One nuance worth knowing: strict mode with JSON input still accepts ISO 8601 strings for datetime, because JSON has no native datetime type. So "2023-04-12T00:00:00Z" passes, while 2023 fails. That's the behavior you want.

If you can't flip the whole model, use per-field strict types (StrictInt, StrictBool, or Annotated[int, Strict()]) on the fields that matter most. For me that was signed_at and total_value.

What did fixing it actually cost?

Turning on strict mode moved 588 records from "pass" to "fail," so I needed a recovery path. I added one repair call: send the original output plus Pydantic's error list back to Claude and ask for corrected JSON.

errors = "\n".join(
    f"{'.'.join(map(str, e['loc']))}: {e['msg']} (got {e['input']!r})"
    for e in exc.errors()
)
Enter fullscreen mode Exit fullscreen mode

Pydantic's error messages are good enough that the model rarely needs more context. Results on the 588 failures:

  • 571 fixed on the first repair call
  • 17 sent to manual review, mostly contracts with genuinely ambiguous dates
  • +14.7% more API calls on that batch, the one honest cost of this whole fix

Then I fixed the prompt. I'd described the fields in prose ("the total value of the contract"). I rewrote it to state types explicitly ("total_value: a JSON number, no quotes, no currency symbol, no thousands separator; signed_at: ISO 8601 string or null, never a bare year"). On a fresh sample of 500 documents, the strict failure rate dropped to 12 of 500, or 2.4%.

And I changed step 4 to store contract.model_dump_json(). If you keep the raw output for audit, keep it in a separate column that nothing downstream reads.

The metric I wish I'd had from day one

Lax pass rate is a vanity number. It measures Pydantic's flexibility, not your model's compliance.

I now log two numbers per batch: strict pass rate on the first call and strict pass rate after repair. The first one is the drift alarm. When I swapped model versions last month, first-call strict pass dipped from 97.6% to 93% in a day, and I caught it before a single bad row landed. Lax mode would have shown 99.7% the whole time.

Does Pydantic lax mode hide LLM type errors?

Yes. Pydantic v2 validates in lax mode by default, so it silently converts numeric strings to numbers, "yes"/"no" to booleans, whole floats to ints, and integers in datetime fields to Unix timestamps. In my pipeline, re-validating 4,000 Claude JSON outputs with model_validate_json(raw, strict=True) exposed 689 coerced fields across 588 records that lax mode had passed, including 4 year-only dates turned into 1970-01-01. Use ConfigDict(strict=True) for LLM output, feed validation errors back for one repair call, serialize with model_dump_json() instead of storing raw output, and track the strict pass rate as your real compliance metric.


Written by the developer behind Preterview, an interview prep platform.

Top comments (0)