DEV Community

Damin Mutti
Damin Mutti

Posted on

Best AI tools for forensic accountants: testing LLM extraction against printed balances

The best AI tools for forensic accountants test what the model read against the balances the bank printed, because a language model can misread one figure on a statement and still hand back perfectly valid JSON. That single test is what ZeraBooks' ranking of 9 AI tools for forensic work sorts on, and you can run it on any model's output in a few dozen lines.

We make ZeraBooks, and it sits first on that list. Its AI reads bank, card, check and brokerage statements, then proves every row against the printed balances before anyone downloads a file. The harness below does the same kind of check with no ZeraBooks code involved.

Why isn't schema validation enough?

Because schema validation tells you the shape is right, not the numbers. A model that reads 4,781.80 as 4,718.80 passes every JSON schema you can write. The bank already printed the answer key on the statement: an opening balance and a closing balance. Everything the model extracted has to connect those two.

What does the model output look like?

Here's a summary-level extraction for checking ••4417, March 2025, from Matter 25-117, a made-up matter. One figure is wrong on purpose.

import json
from decimal import Decimal as D, InvalidOperation

# What the model returned for checking 4417, March 2025.
# One figure is wrong on purpose: money_out should be 4781.80.
model_output = """
{
  "account_last4": "4417",
  "period": "2025-03",
  "opening_balance": "14902.35",
  "total_money_in": "8120.00",
  "total_money_out": "4718.80",
  "closing_balance": "18240.55",
  "checks": [
    {"number": "3285", "date": "2025-03-14",
     "payee": "Harbor Marine Supply", "amount": "2450.00"}
  ]
}
"""
Enter fullscreen mode Exit fullscreen mode

Ask the model for amounts as strings. Floats in JSON get parsed as floats, and you lose the cents before you've checked anything.

How do you parse and tie it out?

REQUIRED = ["opening_balance", "total_money_in",
            "total_money_out", "closing_balance"]

def parse(raw):
    data = json.loads(raw)
    missing = [k for k in REQUIRED if k not in data]
    if missing:
        raise ValueError(f"model left out {missing}")
    try:
        return data, {k: D(data[k]) for k in REQUIRED}
    except InvalidOperation as exc:
        raise ValueError("a balance is not a plain number") from exc

def tie_out(v):
    computed = v["opening_balance"] + v["total_money_in"] - v["total_money_out"]
    return computed, v["closing_balance"] - computed
Enter fullscreen mode Exit fullscreen mode

Can the size of the gap say what went wrong?

Often, yes. A few patterns show up again and again in statement extraction:

def diagnose(diff, amounts=()):
    hints = []
    cents = abs(diff) * 100
    if diff == 0:
        return ["ties to the cent"]
    if cents % 9 == 0:
        hints.append("gap divides by 9: look for two swapped digits")
    for a in amounts:
        if abs(diff) == 2 * abs(a):
            hints.append(f"gap is twice {a}: that row's sign is flipped")
        elif abs(diff) == abs(a):
            hints.append(f"gap equals {a}: that row is missing or doubled")
    return hints

data, values = parse(model_output)
computed, diff = tie_out(values)
print("computed closing:", computed)
print("printed closing: ", values["closing_balance"])
print("difference:      ", diff)
for hint in diagnose(diff):
    print(" -", hint)
Enter fullscreen mode Exit fullscreen mode

Output:

computed closing: 18303.55
printed closing:  18240.55
difference:       -63.00
 - gap divides by 9: look for two swapped digits
Enter fullscreen mode Exit fullscreen mode

The 8 and the 1 swapped places, and the gap is 63.00, which is 7 × 9. Any swap of two adjacent digits leaves a gap that divides by 9, so this hint is cheap and right more often than you'd expect. It isn't proof. Plenty of other gaps divide by 9 too.

Now flip the sign on check 3285, the $2,450.00 payment to Harbor Marine Supply:

print(diagnose(D("4900.00"), [D("2450.00")]))
# ["gap is twice 2450.00: that row's sign is flipped"]
Enter fullscreen mode Exit fullscreen mode

A model that files a check as money in moves the balance by double its amount. Row-level extraction gives you the amounts list to test against.

What should a harness like this include?

  • A tie-out per account and, on card statements, per cardholder, since a card statement can tie overall while one cardholder's rows are off.
  • A balance chain across months, so a statement the model never saw shows up as a break.
  • The same statement run more than once. If two runs disagree, at least one of them is wrong, and you know before the tie-out tells you.
  • A stop rule. A statement that doesn't tie goes to a person, not into the workpaper.

That last one is the whole point. In ZeraBooks, a statement that misses by a cent is marked Needs a look instead of moving on.

Where does this leave general chatbots?

A general chatbot can read a statement you paste into it. It doesn't prove the result against the printed balances, and client confidentiality is worth thinking about before you upload a production anywhere. If you want the full picture of the checks a matter needs beyond extraction, here's how ZeraBooks handles a forensic matter, check images and duplicate uploads included.

Let the model read. Let the bank's own numbers grade it.

Top comments (1)

Collapse
 
kolmopdf profile image
KolmoPDF •

Testing extracted balances against printed opening and closing numbers catches the silent digit swaps that JSON schema checks always miss. On multi-page PDF bank statements, two layout issues often break that tie-out before the arithmetic check even runs.

Wrapped transaction memos are the first trap. When a description wraps onto a second line with a reference number or dollar figure inside the text (like Check #1042 ref 245.00), line-by-line parsers either emit a phantom ledger row or slide values across debit and credit columns. Checking the horizontal x-span of each numeric token against the column header bounds keeps embedded numbers out of the ledger math.

Page-break subtotals cause a similar false mismatch. Statements frequently print a "Balance Brought Forward" or running subtotal row at the top or bottom of a page, and treating that summary line as a transaction inflates total_money_in by the prior page's ending balance. Filtering out those boundary markers before summing keeps the tie-out focused on genuine extraction errors.