NOMDAD
Discuss a project
§ AI INTEGRATION

Why your AI pilot never shipped

6 min read ← All entries

The demo worked. The board was impressed. Eight months later it is still a demo. The failure is almost never the model.

01

A demo proves the wrong thing

A pilot shows that a model can produce a good answer under conditions you chose. Production asks a different question: what happens on the input nobody anticipated, at 3am, when the upstream system returns an empty field. Pilots are built to succeed. Systems are built to fail predictably. Those are different engineering problems, and the second one is where the work actually is.

02

Start at the workflow, not the model

The useful question is not which model to use. It is: which decision, made how many times a day, by whom, currently costs the most attention. Answer that and the shape of the integration follows — where it sits, what it is allowed to touch, what a wrong answer costs. Answer it late and you get a chatbot bolted onto the side of a product, which is what most AI pilots are.

03

Evaluation is the deliverable

Without a test set you cannot tell an improvement from a regression, which means you cannot change anything with confidence — so nobody does, and the pilot freezes. Fifty real cases with known-good outcomes are enough to start. They do not need to be elegant. They need to be real, and they need to run on every change. The team that has this ships; the team that argues from screenshots does not.

04

Assume the model will be replaced

Whatever you pick today will be outclassed or repriced within a year. That is fine, as long as the model sits behind an interface you own: your prompts, your retrieval, your tools, your evaluation. Then a swap is an afternoon. Wire provider-specific behaviour through your business logic instead, and the same swap becomes a rewrite you keep postponing.

05

What shipping actually looks like

One decision, automated end to end, with a measured error rate and a human path for the cases it declines. That is smaller than most pilots and worth more than all of them, because it runs on Monday without anyone watching. Ship that, then widen. The second decision costs a fraction of the first.

◆

Working on something like this?

Tell us what you are building. We will tell you honestly whether we are the right studio for it.

Related service: AI Integration

→

Next entry

One codebase, two stores — and what it costs

Start a project → ← All entries