What happens when the AI is wrong

Every evaluation asks how accurate a system is. Almost none ask what happens on the day it is wrong.

That is the more useful question, because the answer to the first is always "mostly right" and the answer to the second varies enormously between products. Four capabilities determine whether a bad day is an afternoon's work or a fortnight's.

One: can you find out?

The first requirement is detection, and it is the one most likely to be missing.

Automated errors do not announce themselves — nothing crashes, nothing fails to balance. Detection has to be a deliberate activity: sampling confident transactions, decomposing balances into nameable items, reviewing after changes.

The question for a vendor: what in this system would tell me something is systematically wrong? If the honest answer is "nothing, until you look", that is worth knowing — it means detection is entirely your responsibility, which is fine as long as it is planned rather than assumed.

Two: can you determine the scope?

You find one invoice coded wrongly. Is that one invoice, or every invoice from that supplier since March, or every invoice processed under a rule that changed in August?

Answering this needs the system to record, per transaction, which rule or model version produced the treatment and at what confidence. With that, the query is straightforward: everything processed under this rule between these dates. Without it, you are working from the ones you happened to notice, and correcting a population you cannot define.

This single capability is the largest determinant of how expensive an error is, and it is almost never asked about in evaluations.

The question: show me how I would identify every transaction processed under a particular rule version in a date range.

Three: can you unwind it?

Correction has to be a designed feature, not an improvisation.

What you need:

• Bulk reversal of an identified population, not one transaction at a time
• The reversal itself on the record, so the correction is visible rather than the history being rewritten
• The original entries preserved, because an auditor will want to see what happened, not a tidied version
• Correction at the cause — fixing the rule, not just the transactions, or it recurs next month

The anti-pattern is a system where correcting means deleting and re-entering. That destroys the trail, and the trail is what makes the whole arrangement defensible.

The question: four hundred transactions were processed under a wrong tolerance. Walk me through fixing that.

Four: can you explain it afterwards?

Eventually someone asks: what happened, how did it happen, how long did it run, what did you do about it.

That someone is an auditor, a tax authority, a board, or a lender. The answer needs to be reconstructable from records rather than memory, which means the system has to have captured — at the time — the rule in force, the confidence, the review that did or did not occur, and the correction.

The question: what does this system record that would let me reconstruct a processing error six months later?

The response when it happens

A working sequence, worth agreeing before you need it:

1. Stop the process producing the error. Not the whole system, the specific rule.
2. Establish the start date and the cause.
3. Query the full affected population.
4. Assess materiality across all of it, not the sample you found.
5. Fix the cause.
6. Reverse and re-post, with the correction on the record.
7. Write down what happened, including how it was found and what changed to prevent recurrence.
8. Tell whoever needs to know, early. Auditors are markedly more comfortable with a discovered, documented, corrected error than with one they find themselves.

Step seven is the one skipped under time pressure, and it is the one that matters most six months later.

What it is reasonable to expect

Not perfection. A well-run automated finance function will still have periods where something was wrong.

What is reasonable:

• Errors are found within a period or two, not at year end
• The affected population can be identified precisely
• Correction takes hours, not weeks
• There is a written record of what happened
• The same error does not recur, because the cause was fixed

That is a functioning control environment. A vendor uncomfortable discussing any of it is telling you something useful — the honest ones have thought about this, because their customers have had bad days.

Common questions

What should I ask a vendor about errors?

Four things: what in the system would reveal that something is systematically wrong, how to identify every transaction processed under a particular rule version and date range, how to reverse an identified population in bulk with the correction on the record, and what is recorded that would let you reconstruct the incident six months later. Accuracy claims matter far less than the answers to these.

How do you correct a large number of wrong automated entries?

By identifying the full affected population first — which requires the system to record which rule or model version produced each treatment — then fixing the cause before the transactions, reversing in bulk with the reversal itself recorded rather than the history being rewritten, and documenting what happened. Deleting and re-entering destroys the trail that makes the correction defensible.

What determines how expensive an automated error is?

Mainly whether you can define the affected population. If the system records the rule version and confidence against each transaction, the correction is a query and a bulk reversal; if it does not, you are correcting only the transactions you happened to notice and cannot demonstrate the scope to anyone. This capability is rarely asked about during evaluation and largely determines the cost of a bad day.

Should we tell the auditor about an error we found ourselves?

Yes, and early. Auditors are considerably more comfortable with an error that was discovered, documented and corrected by the business than with one they find themselves, and most comfortable of all when there is a written record showing how it was detected and what changed to prevent recurrence. An error found and not written down is the worst of the available positions.

Related: what an AI accounting error looks like · who is accountable for an automated entry · rolling back an automation that failed