Explainability in accounting automation

In most fields, explainable AI is a research topic. In accounting it is an operational requirement, because you will be asked to justify individual entries — by an auditor, a tax authority, a lender, or a business owner who does not believe a number.

"The system decided" is not an answer to any of them.

What has to be explainable, specifically

Three different things, frequently conflated.

Why this treatment. Why was this invoice coded to this account, matched to that invoice, allocated to this period? A statement of the basis: this supplier's history, this purchase order, this rule.

Where this figure came from. For any number in a report, the transactions behind it, and the documents behind those. This is traceability, and it is the more fundamental of the two.

What the system was doing at the time. Which rules were in force, which model version, what the thresholds were. Necessary because a system that changed in August means an entry from July was produced by different logic than one from September.

Products commonly deliver the second and claim it covers all three. It does not. Being able to drill into a total is not the same as being able to say why a transaction was treated as it was.

The standard worth holding

A practical test, applicable in a demo: pick any figure and get to the source document in three clicks, then get a plain-language statement of why that transaction was treated that way.

If drilling down works but the "why" is unavailable, you have traceability without explainability. That is enough for most auditor questions about balances and not enough for questions about treatment — and treatment is where the difficult questions are.

Why language models make this harder and easier

Harder, because a model can produce a fluent explanation that is a plausible reconstruction rather than a record of what actually happened. An explanation generated after the fact is not evidence. It sounds exactly like one that is.

Easier, because when the system records the actual basis at the time of the decision, a language model can render it readable. "Coded to freight because this supplier's last 47 invoices with similar descriptions were coded to freight, confidence 0.94" is a record. Presenting it in a sentence is presentation, not invention.

The distinction to insist on: is the explanation retrieved from what was recorded at the time, or generated now? Ask directly. Vendors who have thought about it answer immediately; the question visibly lands as new with those who have not.

What the record needs to contain

Per transaction, at minimum:

• The source document, or a durable reference to it
• The treatment applied and the basis for it
• The rule or model version in force
• The confidence level
• Whether a human reviewed, who, and when
• Any subsequent amendment, with its own reason and author

That set answers essentially every question that gets asked afterwards. Missing any one of them creates a category of question you cannot answer — and you discover which category during an audit, which is a poor time to find out.

Reports have to be explainable too

Explainability at transaction level is necessary and not sufficient. If a system answers questions in plain language — "why is margin down two points" — then every figure in that answer needs to be traceable.

The failure mode is specific and serious: a confident, fluent, wrong answer, which passes casual review precisely because it is well-argued. The architectural defence is that figures are computed by the system and retrieved, never produced by the model's reasoning. We cover this in AI hallucinations in financial data.

Practical test: ask a question, then click into a number in the answer. If you cannot, treat the answer as a claim rather than a fact.

What auditors actually want

Less than people assume, and different.

They generally do not want the model explained. They want to know that automated processing operated under documented controls, that exceptions were handled by a defined process, that someone was accountable, and that individual entries can be traced to evidence.

An auditor is comfortable with "this was coded automatically based on supplier history, under a rule requiring human review above RM 10,000, and here is the invoice". They are not comfortable with "the system did it and we can't reconstruct why" — which is a statement about your records rather than about AI.

Common questions

What does explainability mean in accounting automation?

Three distinct things: why a particular treatment was applied to a transaction, where any figure in a report came from in terms of underlying transactions and documents, and what rules, model version and thresholds were in force at the time. Products often deliver the second and imply it covers the others, but being able to drill into a total does not tell you why a transaction was treated as it was.

Is an AI-generated explanation acceptable evidence?

Only if it retrieves what was recorded at the time of the decision rather than reconstructing a plausible account afterwards. A generated explanation sounds identical to a recorded one while being an inference about what probably happened, so it is worth asking a vendor directly whether explanations come from stored decision records or are produced on demand.

What do auditors want to see for automated entries?

That processing operated under documented controls, that exceptions followed a defined process, that a named person was accountable, and that individual entries trace to supporting evidence. They generally do not need the model explained — the objection is to entries nobody can reconstruct or attribute, which is a records problem rather than an AI problem.

How do I test explainability in a demo?

Pick any figure in a report and try to reach the source document in about three clicks, then ask why that transaction was treated as it was and see whether a plain-language basis is available. If you can drill down but cannot get the reasoning, you have traceability without explainability, which is sufficient for questions about balances and not for questions about treatment.

Related: AI hallucinations in financial data · evidence an auditor will accept · what full traceability actually means