AI hallucinations in financial data — the guardrails an ERP must have

A person who is unsure hesitates. They frown, re-read the invoice, ask a colleague. That hesitation is a signal, and every finance process ever designed quietly depends on it.

A language model does not hesitate. Ask it to read an invoice and it returns a number. If it misreads 4,820 as 4,320, it returns 4,320 with precisely the same confidence it would have returned the correct figure. The uncertainty signal that finance was built around is missing.

That is the actual risk of AI in financial systems. Not that models are wrong — people are wrong constantly, and we built controls for that. It is that models are wrong without the tells, and our controls were tuned to the tells.

What "hallucination" means here, precisely

The word is loose, so let us split it into three failure modes with different fixes.

Extraction error. The model reads a document and gets a value wrong. Rare on clean documents, less rare on a phone photo of a crumpled delivery order.

Confabulation. The model produces a plausible value that is not in the source at all. Ask for a PO number on a document that has none, and a poorly-constrained system may supply one that looks exactly like a PO number. This is the dangerous one, because the output is well-formed. It passes every format check you have.

Reasoning error. Extraction is right, conclusion is wrong. It matches an invoice to the wrong PO because they are similar, and explains its choice fluently.

That third one is the hardest, because the explanation is persuasive. A human reviewer reading a confident, coherent rationale is far more likely to approve than one facing a bare number. Fluency defeats scrutiny. That is not a model flaw you can patch — it is a property of how people read.

Why "the model is 99% accurate" is not reassurance

Two reasons, and both matter more than the headline number.

Volume. A thousand invoices a month at 99% is ten wrong invoices a month. Whether that is fine or catastrophic depends entirely on which ten. One of them is a payment to a supplier who does not exist.

The errors are not random. They cluster on exactly the documents that are unusual — the handwritten amendment, the credit note, the supplier who redesigned their template. Which are the same documents most likely to be doing something that matters. The error distribution and the risk distribution overlap, and averages hide that completely.

Ask a vendor about accuracy and you get one number. Ask them about the distribution of errors and you find out whether they have thought about this.

The guardrails

Structural, not model-based. The point of every one is to stay correct even when the AI is confidently wrong.

1. Provenance on every extracted value

Every figure the AI produces should be traceable to where it came from — this document, this region, this line. Not because users will check every one, but because when something is wrong you need to reconstruct what happened in seconds rather than reverse-engineering a black box.

A system that cannot tell you why it thinks the total is RM4,820 is unauditable, regardless of accuracy.

2. Thresholds with a named human above them

The simplest and most effective control. Above an amount you choose, a person approves. Full stop. Not a person who can approve — a person who must.

This is not distrust of AI. It is the same control you would place on a new junior, and for the same reason: the cost of an undetected error scales with the amount.

3. Immutable audit trail, including the AI's own actions

Who did what, when, and if it was a machine — which one, on what input. When a regulator or auditor asks why a payment went out, "the system did it" is not an answer. "The system extracted these values from this document, matched this PO, and this named person approved it at 14:32" is.

This is why we treat the audit trail as structural rather than a setting. In a world where generation is cheap, the record of what happened is the thing you actually own.

4. Independent duplicate detection

Run it separately from the extraction. If the same component both reads the invoice and decides whether it is a duplicate, a systematic misreading defeats both at once. Independent checks fail independently — that is the entire point of having them.

5. Out-of-band verification for payment details

Supplier bank-detail changes must be verified through a channel that is not the incoming document. Phone the number you already had. Not the one on the email.

Invoice fraud works by looking legitimate, and a model trained to accept things that look legitimate is not a defence — it is arguably a liability. This is a process rule and no AI feature replaces it.

6. Make the human review meaningful

The hardest one, and the most neglected.

If a system asks a person to approve four hundred items a day, they are not reviewing. They are clicking. You have built a rubber stamp and told your auditors it is a control — and you now have less oversight than before, plus documentation claiming otherwise.

Meaningful review means volume low enough that attention is real, context rich enough to decide in seconds, and the confident cases genuinely automated so human attention lands where it matters. Escalate less, but escalate better. A system that escalates 3% of invoices to a person who actually looks is safer than one escalating 40% to someone who does not.

The uncomfortable trade

Every guardrail above costs you some of the speed you bought the AI for. That is not a flaw in the design. It is the design.

The pitch "fully autonomous accounts payable" and the pitch "every payment has human oversight" are in direct tension, and any vendor claiming both without qualification has not thought it through, or is hoping you will not.

Our position: on anything touching money, take the speed on the boring 90% and spend it buying real attention on the 10% that matters. The tedious is where the value is. The judgement is where the risk is. Do not let a vendor blur those — including us. That distinction is the whole of what AI still cannot do in ERP.

Questions to ask any vendor

• When the model is wrong, how do I find out? If the answer is "you'd notice in the reconciliation", the control is your month-end, which is weeks late.
• Can you show me the provenance of an extracted value?
• What is in the audit trail when the AI takes an action?
• Is duplicate detection independent of extraction?
• What proportion of documents get escalated, and do you track whether reviewers actually catch errors? Almost nobody measures this. It is the number that tells you whether your control is real.

That last question is the one worth asking, because it is the difference between having a control and having documentation of a control.

Common questions

What happens if AI misreads an invoice amount?

It hands you the wrong number with exactly the confidence it would have given the right one. A person who is unsure hesitates and re-reads the document; a model does not, so the uncertainty signal every finance process was built around is missing. The defence has to be structural rather than a better model: every extracted figure traceable back to the document, region and line it came from, so when something is wrong you can reconstruct what happened in seconds.

Is 99% extraction accuracy good enough for invoice processing?

Not on its own, for two reasons. A thousand invoices a month at that rate is ten wrong invoices a month, and whether that is fine depends entirely on which ten — one of them may be a payment to a supplier who does not exist. The errors are also not random: they cluster on the handwritten amendment, the credit note, the supplier who redesigned their template. Ask a vendor about the distribution of errors, not the headline number.

Do I still need human approval if the AI is accurate?

Yes. Above an amount you choose, a person approves — not a person who can approve, a person who must. It is the same control you would place on a new junior and for the same reason: the cost of an undetected error scales with the amount. Changes to supplier bank details need verifying out of band as well, by phoning the number you already had rather than the one on the incoming email. No AI feature replaces that process rule.

How do I tell whether a review step is a real control?

Look at the volume. If the system asks a person to approve four hundred items a day, they are not reviewing, they are clicking — you have built a rubber stamp and documentation claiming it is oversight, which leaves you with less scrutiny than before. Meaningful review means the queue is small enough for attention to be real, the context rich enough to decide in seconds, and the confident cases genuinely automated. Escalate less, but escalate better.

If you want to work out what this would look like in your business, talk to us — including if the honest answer is that you are not ready yet.

Related: AI in accounts payable for how the function evolved, and the new shadow IT for governance beyond the ledger.