Where AI automation fails: an honest list

Decisions the law keeps with a person

UK data protection law draws a line that no improvement in model quality will move. The ICO explains it this way: automated individual decision making means making a decision solely by automated means without any human involvement, and Article 22 of the UK GDPR adds rules where that decision has a legal or similarly significant effect on someone.

If a decision falls into that category, it can only be taken when necessary for a contract, authorised by law or based on explicit consent. The ICO also expects a route for people to ask for human intervention, challenge the decision and check that the system is doing what it was built to do.

In practice this covers credit, insurance, hiring, tenancy and access to services. A small business may never touch those areas. The habit is still worth copying: if a decision cannot be undone and matters to someone's money, health or work, a person signs it off.

Work that starts with something a machine cannot read

Automation is at its best when the input arrives in a predictable shape: a form, a booking, a structured email or an API message. It struggles when the first step is interpreting something physical or messy.

Handwritten job sheets, photographs of a boiler, a price scribbled on an invoice or a long email thread where the customer changes their mind can be partly processed. The missed detail is often the important one: a serial number or access code can send the wrong person to the wrong job.

Look at your last ten pieces of work and ask how many arrived in the same format. If the answer is three, automation needs a person at the front of the queue, and that person is part of the real cost.

Complaints, disputes and negotiations

Anything with an emotional or legal edge belongs with a human from the first message. A complaint about damage, a disputed bill, a discount request or a change of scope halfway through a job is a test of whether the business is reasonable.

Software can collect facts, draft a summary, log a promise and remind you to reply. It should not decide the outcome or send a settlement. At higher volume, a short first reply can be drafted by the machine and flagged for a person before anything is agreed.

Systems that have no usable way in

Plenty of small business software still expects a person at a keyboard. Desktop accounting packages with no interface, a supplier that only sends PDFs, a spreadsheet on one laptop or a booking system whose only export is a printed list.

Connecting to those is possible, and the connection is often where the budget goes. When there is no supported way in, every sync is fragile engineering someone has to maintain.

When the data going in is already wrong

Automation does not clean up an underlying mess. Duplicate customer records, two systems that disagree about a price, a missing field that everyone fills in from memory: a machine will process all of it and return a tidy, confident answer that happens to be wrong.

This is the least glamorous part of the work and the part that decides whether the result is trusted. Some of the effort always goes into matching records, agreeing which system owns the record and writing down what happens when they disagree.

Advice that carries legal, medical or tax weight

Bookkeeping, contract drafting, tax treatment and clinical questions can be prepared by software and owned only by a qualified person. The preparation is genuinely useful: pulling figures together, drafting a first version, checking that nothing obvious is missing. The advice itself is a different thing, with a name attached to it.

Anyone automating this area should also keep an eye on the rules, because they are moving. The ICO's guidance on AI and data protection was updated in March 2023 and the page now carries a notice that it is under review following the Data (Use and Access) Act, so it may change again.

When the volume is too low to be worth it

This is our opinion, not a published fact, and it is the one that loses us the most work.

If a task happens a handful of times a week and takes a few minutes, the honest advice is usually to leave it alone. Setting something up, checking its output and fixing the exceptions can cost more than the original job did. Automation earns its place where the same task repeats often enough that small savings add up, or where the real damage is a missed message at the wrong time.

The failure that worries us most

The visible failures are not the dangerous ones. A message that does not send, a booking that does not appear, a form that refuses to submit: you find out within the hour.

The dangerous failure is the one with no error message. A call is answered politely, the details are captured slightly wrong, the summary reaches the wrong folder, and nobody notices because every screen shows success. By the time a customer rings to ask why nobody turned up, the trail is three days old.

The practical answer is unglamorous: one named person owns the queue, exceptions are visible rather than buried in an inbox, every automated step leaves a record you can read back, and the first weeks run with a person checking the output before it goes out.

What we do about it in our own work

Our own product is built around the same limits. Enquiries are handled, details are captured and passed on, and the awkward ones are handed to a person with the context attached, which is why our line for calls and our chat assistant both end with a notification rather than a decision. The areas we cover, and the ones we leave alone, are listed at amoskalets.com/solutions.

Five questions before automating anything

1. Does this decision affect someone's money, health, work or legal position? If yes, a person signs it off.

2. Do the inputs arrive in one predictable shape, or does someone have to guess what a document means?

3. Who owns the exceptions, by name, and where do they see them?

4. If the output is wrong and no error appears, how would you find out, and how quickly?

5. Does this task happen often enough that the saving is worth the setup and the checking?

If the answers to the last two are unclear, the honest next step is a small test on one process, run in parallel with the way you work today, before anything is switched off.

Where the facts in this article come from

• Automated decision making under the UK GDPR, including Article 22 and the requirement to offer human intervention: ICO, Rights related to automated decision making including profiling.
• The review notice on the ICO's AI guidance and the March 2023 update date: ICO, Guidance on AI and data protection.
• The statements about where automation struggles, the value of setting up low volume work and the nature of silent failures are our own reading of how these systems behave. They are marked as ours because no published UK dataset measures them, and we are not going to invent one.