Why confidence scores matter more than accuracy in finance

The most useful thing an accounting system can tell you is not the answer. It is how sure it is about the answer.

That sounds like a technicality. It is the difference between automation you can safely run at volume and automation that requires you to check everything — which is not automation at all.

What a confidence score is

When a model proposes a treatment, it also produces an internal measure of how strongly the evidence supports it. A recurring invoice from a supplier with two years of consistent history scores high. A first invoice from an unknown supplier with an ambiguous description scores low.

That number is what allows the split: high-confidence transactions process automatically, low-confidence ones go to a person. Without it there are only two options — check everything, or check nothing.

Rule-based systems cannot produce this. A rule either fires or it does not; there is no gradient. Which is why the absence of any confidence indicator in an interface is a reliable signal about what is underneath, and worth checking in a demo.

Why it beats raw accuracy

Two systems processing a thousand transactions:

System A: 99% accurate, no confidence scoring. Ten errors, indistinguishable from the 990 correct ones. All ten reach your accounts.

System B: 96% accurate, with confidence scoring. Forty errors, of which thirty-five score low enough to be flagged. Five reach your accounts.

System A wins on the metric in the brochure. System B puts half as many errors into your ledger and hands you a queue containing most of the problems.

What matters is not how often it is right. It is whether it knows when it might be wrong.

Setting the threshold

The threshold determines what posts automatically, and it is a real business decision rather than a technical setting.

Set it too high and everything goes to review. The exception queue is unmanageable, reviewers start rubber-stamping, and you have automated nothing while adding a step.

Set it too low and errors post silently. You find out at year end.

The approach that works:

1. Start high. Accept a large exception queue initially. You are gathering evidence, not saving time yet.
2. Track what the reviewers change. If they approve 98% of items in a confidence band without amendment, that band does not need review.
3. Lower it in steps, one band at a time, watching what happens.
4. Vary it by risk, not just by confidence. Coding a RM 40 stationery invoice and a RM 40,000 capital purchase should not share a threshold, however confident the system is. Amount and category belong in the decision.

That last point is the one most often missed. Confidence is about the model's certainty; risk is about consequence. Both should govern what posts.

Where confidence is misleading

Two honest caveats.

Confidence is not probability of correctness. A model can be confidently wrong, particularly on cases that resemble something they are not — an invoice from a supplier who has changed what they sell, or a new supplier whose name resembles an existing one. High confidence lowers the odds; it does not eliminate them.

Confidence says nothing about systematic error. If a supplier has been miscoded consistently for two years, the model is highly confident and consistently wrong, because it learned the error. Confidence measures agreement with history, not correctness. This is why sampling high-confidence transactions is a separate and necessary control.

What to ask a vendor

"Show me the confidence level on a transaction." If it is not visible anywhere, ask directly what determines whether something is reviewed.

"Can I set different thresholds for different categories or amounts?" A single global threshold is a blunt instrument.

"What happens between the thresholds?" There should be a middle band that is proposed-but-reviewed, not just a binary.

"Does confidence improve as I correct it?" It should. If corrections do not affect future confidence, the system is not learning.

The practical test

If you are already running automation, one question tells you whether this is under control:

Can you say what percentage of your transactions posted without human review last month, and what the threshold was?

If the answer is no, the threshold is whatever the implementer set, nobody has revisited it, and you do not know how much of your ledger nobody looked at. That is a common position and an uncomfortable one to be in when an auditor asks.

Common questions

What is a confidence score in accounting automation?

It is the system's own measure of how strongly the evidence supports the treatment it is proposing, based on how closely a transaction resembles ones it has seen before. It is what allows high-confidence transactions to process automatically while lower-confidence ones are routed to a person, which is what makes automation safe to run at volume rather than requiring every item to be checked.

Why does confidence matter more than accuracy?

Because errors that the system flags are caught, and errors it does not flag reach your accounts. A system that is slightly less accurate but reliably signals uncertainty puts fewer undetected errors into the ledger than a more accurate system that cannot distinguish its confident correct answers from its confident wrong ones.

How should confidence thresholds be set?

Start high, accept a large exception queue while you gather evidence, track which confidence bands reviewers approve without amendment, and lower the threshold one band at a time. Thresholds should also vary by risk rather than confidence alone, since a small stationery invoice and a large capital purchase warrant different treatment even at identical confidence.

Can a system be confidently wrong?

Yes, in two ways. It can misjudge a case that resembles something it is not, such as an invoice from a supplier who has changed what they sell. More importantly, it will be confidently wrong about systematic errors it learned from your history — a supplier miscoded consistently for two years produces high confidence and a wrong answer, which is why periodically sampling high-confidence transactions is a necessary separate control.

Related: how accurate is AI in accounting · the exception queue as a control · sampling automated transactions