Where AI goes wrong, and how to build around it.

The first AI automation I put into production read invoices for a whole week without a problem. In the second week it read "Total: 1.250,00" as 125000. The invoice went through with an amount a hundred times too high and landed in a register. Nothing blew up, because someone was watching. But the lesson stuck.

A classic program, when it doesn't know, stops. A model, when it doesn't know, guesses. And it guesses in the same confident voice it uses when it's right. That's not a defect you fix; it's how it works. So I don't build for it to never be wrong. I build for the mistake not to matter.

How it fails, concretely

• It invents. An invoice number that doesn't exist, a product name close to the real one, a plausible date.
• It misreads. The comma and the dot in amounts, a date in American format, a "1" that is really an "l", the second page skipped.
• It answers differently. The same text twice, two slightly different results. Fine for a summary; not for an amount.
• It gets carried away by the text. An email containing "ignore the previous instructions" really can make it ignore them, if you weren't careful.

The four safety nets

1. Rule-based validation. Everything the model outputs goes through a classic filter. The amount has to be a number. The tax ID has to exist in the supplier list. The date has to be within the last 90 days. The total has to equal the sum of the lines. If anything fails, the document goes to a person, not onward.

2. "I don't know" is a valid answer. I ask the model to say how sure it is, and I treat "unsure" as a normal path, not an exception. A "to check" folder with three files a week is a success, not a failure.

3. A human signs. Everything that goes to a customer, to the accountant or to the tax office has a step where someone looks. One click, not an hour. The difference is that the mistake stops with you, not with the customer.

4. A log. Every call: what went in, what came out, how confident it was. When something looks off three months later, you know exactly what happened, and you can fix the instruction, not just the invoice.

What this means for you

That an AI automation doesn't remove control, it moves it. Instead of reading 200 invoices, you check five. Instead of writing 40 replies, you approve 40. If someone promises you a "fully autonomous" system on messy data, ask them what happens on the day the model reads 125000.