I recently spent a day taking a mixed commercial and technical team through AI from first principles, and then building working software with them before they went home. By the end, not one of the things that made the difference was a product name. They were distinctions: how the model actually works, which mode you are running it in, how much autonomy you are handing over, and which failure you are designing against.
Skip those four and you get the results that made the news. Air Canada's chatbot invented a bereavement fare policy, the airline argued in court that the chatbot was a separate legal entity, and the court disagreed. McDonald's pulled AI voice ordering from a hundred restaurants after one customer's drink order turned into two hundred dollars of McFlurries. Klarna replaced the work of 700 support agents and then rehired, with its CEO conceding that the AI's ability to mimic emotional intelligence was very limited.
None of those were model failures. All three were design failures.
The pilot habit
The standard way in is a pilot. Pick a tool, pick a department, run it for a quarter, see what happens. That made sense while the technology was unfamiliar and the risk was unclear, and it has produced a whole genre of internal report concluding that the technology is promising but not yet ready.
The pilot answers the wrong question. It tells you whether people enjoyed using a product. It does not tell you which work in your company can be handed over, under what supervision, with what proof attached. That is the question you need answered, and no vendor demo contains it.
The model is not the product. The context is.
Here is the piece of theory that changes behaviour fastest, and it takes ten minutes to teach.
A model reads and writes tokens, not words. Roughly three tokens for every four English words. Everything it can see at once sits in a context window, which is best thought of as a desk. What is on the desk, the model can use. What is not on the desk does not exist, and the model will not tell you something is missing, because it has no way of knowing.
What it does with that desk is predict the next token. That is the entire mechanism. There is no comprehension underneath it and, more importantly, no internal signal for I do not know. When information is missing the model does not stop. It predicts what a plausible answer would look like, and plausible is exactly what a confident wrong answer is made of.
Ask a model about your returns policy for damaged goods without giving it the policy, and it will invent one that sounds like yours. Paste the real policy into the context and ask again, and you get the right answer. Same model, same question, different desk.
This is why hallucination is not a defect awaiting a patch. It is a property of the architecture. The fix is not better wording, it is putting the right things on the desk and building a system that notices when something is absent.
The same logic applies to the dials people reach for first. Turning the temperature down does not stop the model being wrong. It makes the wrong answer more consistent.
The weak spots are predictable, and none of them yields to a cleverer prompt: arithmetic, anything that happened after training, anything sitting in your private systems, long multi-step procedures, and instructions buried on page 47. Each has a design response rather than a prompting response. Use code for maths. Fetch the data and hand it over. Connect the systems through tools. Break the procedure into separate steps you can inspect.
Two modes, and you have to pick one
The most useful distinction from the whole day is also the simplest.
- AI as producer. The model makes something: a piece of copy, an analysis, a block of code. A human reads it, decides, ships it. One shot, reviewable, finished.
- AI as engine. The model runs inside your operations, deciding on live data, continuously, with nobody reading each individual output. Classifying incoming tickets and routing them. Matching payments to settlements. Flagging stock levels.
The technology is identical. Everything else is different. Producer work fails privately and cheaply, and a human is the safety net by default. Engine work fails in production, at volume, in front of customers, and the safety net has to be built on purpose. Different risk profile, different observability requirements, different cost model.
Deciding which mode a task belongs in, before anyone builds anything, is most of the job. All three headline failures above were engine deployments run with producer-level supervision.
From prompt to agent, one rung at a time
An agent is not a cleverer model. An agent is a prompt inside a loop, with tools and rules. Something happens, the agent checks a condition, takes an action, verifies the result, and waits for the next event. The intelligence per step is unchanged. The exposure is not, because now it acts.
So autonomy is a dial, not a switch, and three settings are enough to plan with.
- Human in the loop. The agent drafts, a human approves every action. Correct wherever errors are expensive: anything touching money, contracts, or a named customer.
- Human on the loop. The agent acts, a human monitors and can intervene. Correct for high volume with reversible consequences: classification, routing, drafting replies.
- Human out of the loop. The agent acts alone. Correct only where the action is cheap, reversible and heavily instrumented.
Start low. Move up when the logs give you a reason to, not when the roadmap says so.
The same progression shows up in the tooling. Business intelligence tells you what happened last quarter. An operational dashboard tells you what is happening now. A command centre shows what is happening and lets you act on it, with the agent proposing and a human confirming. Most companies own the first, have built parts of the second, and have never seen the third. The third is where the operational return sits.
Name the failure before you automate
Agents fail in a small number of recognisable ways, and each has a countermeasure you build in advance.
- Loops. The agent processes the same event ten thousand times, so you track processed IDs and check before acting.
- Stale data. The decision rests on yesterday's inventory, so you fetch fresh and timestamp everything.
- Hallucination inside an action. A routine case gets classified as urgent and escalated, so you set a confidence threshold below which the case goes to a human.
- Cascade. One bad decision triggers a chain of consequences across connected systems, so you put hard limits and human checkpoints between stages.
The test is blunt: if you cannot name the failure mode, you are not ready to automate the process. That single question kills more bad ideas in a workshop than any cost estimate.
What this changes for you
Getting started is not a procurement decision and it is not a pilot. It is four decisions taken per process: what goes on the desk, which mode you are in, which rung of the ladder you are standing on, and which failure you are designing against. Answer those and the tool question largely answers itself. Match the model to the task, expect to run two or three different ones, and remember that the answer to volume is rarely the most expensive model available.
The job of the person leading this is no longer choosing AI. It is deciding, process by process, what is allowed to act without asking. That is a governance skill more than a technical one, and it is what separates the companies quietly running real work through agents from the ones still reading their pilot report.
Related reading: AI agent or plain automation?