Why most AI projects die between the demo and production

There is a widely-cited pattern in enterprise AI reporting right now: a large majority of organizations say they have adopted agentic AI in some form, while only a small fraction have anything running in production. The exact numbers move depending on who is counting. The shape does not.

That gap is not a technology problem, and it is not a talent problem. It is a scoping problem. The demo is genuinely easy now — an afternoon, maybe two. What follows is three months of work that nobody put in the plan, because from the outside it looks like the project is already finished.

What the demo actually proved

A demo proves that the model can produce the right output for an input you chose, in an environment you controlled, with a failure mode you were present to explain away. Every one of those conditions disappears in production.

This is not a criticism of demos. Demos are how you find out whether the idea is worth pursuing at all, and building one first is correct. The mistake is treating the demo as ninety percent of the project when it is closer to ten.

The five things that sit in the gap

Nearly every stalled AI project we have looked at stalled in one of five places. They are boring, which is exactly why they get skipped.

1. 01Data access with real permissions. The demo read from a folder someone exported. Production has to read from the live system, respecting who is allowed to see what — and a support agent that can quote another customer's ticket back at someone is not a bug you get to fix quietly.
2. 02Failure behavior. What does the system do when it does not know? 'Answer anyway' is the default and it is almost never the right one. Deciding the confidence threshold, the escalation path, and who gets told is design work, and it has to happen before launch.
3. 03Evaluation. Software has compilers and test suites; model behavior has neither. Without an evaluation set built from your real cases, you cannot tell whether a prompt change improved things or broke them — you can only tell whether the three examples you happened to check still look fine.
4. 04Monitoring. Volume, failure rate, escalation rate, cost. If nobody can see these without asking an engineer, nobody will look, and the system will degrade silently for months.
5. 05Adoption. A system your team does not trust gets checked every time, which means it saves nothing. Trust comes from visible sources, predictable failure, and someone accountable — not from accuracy alone.

Evaluation is the one people most regret skipping

Of those five, evaluation is the one that comes back to bite hardest, because its absence is invisible until you need it. Everything works. You change a prompt to fix an edge case. Everything still appears to work. Three weeks later someone notices the system has been handling a common case worse ever since, and there is no way to know when it started.

An evaluation set does not need to be sophisticated to be transformative. Thirty to fifty real cases with known-good answers, run on every change, is enough to convert 'I think that's better' into 'that's better on 46 of 50, worse on 2, and here are the two.' That is the difference between engineering and vibes.

You cannot improve what you cannot measure, and with model behavior you genuinely cannot measure it by looking.

The honest scoping fix

The practical correction is to stop scoping AI work as 'build the thing' and start scoping it as two separate pieces of work with a decision point between them.

• A short, cheap prototype phase whose only job is to answer 'is the model good enough at this task at all?' — days, not weeks, and genuinely disposable.
• A build phase that assumes the answer is yes and budgets properly for access, failure behavior, evaluation, monitoring, and rollout.

Splitting them protects you in both directions. If the prototype fails, you spent days rather than a quarter. If it succeeds, nobody is under the illusion that the hard part is behind them.

How to tell whether a project is actually in production

A test we find useful: can somebody who is not you answer these four questions without asking an engineer?

1. 01How many times did this run last week, and how many times did it escalate to a human?
2. 02What does it do when it is not confident?
3. 03How would we know if it got worse?
4. 04Who is responsible when it does?

If the answers exist and are written down, the thing is in production. If they do not, what you have is a demo that happens to be reachable on the internet — which is a genuinely different and much more fragile object.