From pilot to production: why automations stall after the demo

The pilot worked, everyone saw it run, and eight months later it is still the only automation in production. This is the most common outcome in the region, and the cause is almost never the technology: the pilot proved feasibility and nothing else it needed to prove.

Below: what the demo does not test, the four things production requires, the ownership question nobody answers, how to choose the second process, and what a real pilot measures.

A pilot answers "can this be automated?" — a question that, for most administrative processes, has been answerable with yes for a decade. It is the wrong question to spend three months on.

The questions that decide whether anything reaches production are different, and they are usually not asked until the pilot is over.

The pattern is consistent across Colombia and Mexico, and it is not a maturity problem: the teams running these pilots are competent and the tools work. What is missing is an operating model, and nobody notices its absence until the demo is over and the first real month begins.

What the demo does not test

A demo runs the happy path with prepared data, at a moment when the person who built it is watching. Production is the opposite of every one of those conditions.

It does not test what happens when the source system is slow, when a field arrives empty, when the volume is ten times the sample, or when the automation runs at 3am with nobody watching. Each of those is ordinary and each of them breaks a demo-grade build.

Nor does it test the boring parts: how a failure gets noticed, who gets called, and how the process runs manually while the automation is down. Those are not edge cases — they are the operating conditions.

The four things production requires

Error handling that is designed, not incidental. Every step needs a defined behaviour when it fails: retry, skip and log, or stop and alert. A build without this makes the decision by accident.

Monitoring somebody actually receives. An alert into an unwatched mailbox is not monitoring. The test is whether a failure at 3am on a Saturday reaches a person before the business notices.

Credential management. Pilots run under a personal account. Production cannot, and the change is not cosmetic — it usually surfaces permissions the automation was borrowing from whoever built it.

A documented manual fallback. The process still has to run when the automation does not, and the team that stopped doing it by hand six months ago will not remember how.

The gap between a build and a service

What separates the two is not code quality. It is that a service has an agreed availability, a known cost to run, and someone whose job includes it — and a build has none of those, however well written it is.

The practical consequence is that a production automation carries a running cost that the business case rarely includes: licences, the infrastructure it runs on, and the support time it consumes when a source system changes. Two or three hours a month per automation is a reasonable planning assumption, and it is not zero.

Leaving that cost out is what makes the second-year conversation difficult. The savings were presented as permanent and the support effort appears as a surprise, which makes an otherwise successful programme look like it underdelivered.

The ownership question nobody answers

Who maintains it. Not who built it — who changes it when the source system gets an update, who is accountable when it produces a wrong result, and whose budget pays for the change.

When this is unanswered, the automation belongs to whoever built it until that person moves on, and then it belongs to nobody. The failure is not dramatic: it degrades, someone works around it, and eventually the process goes back to being manual without anyone deciding that.

Answering it is not a governance formality. It is the difference between one automation and a portfolio, because the second process cannot start until the first one has somewhere to live.

How to choose the second process

Not by size. The instinct after a successful pilot is to go after the biggest saving available, and that process is usually the one with the most exceptions, the most stakeholders and the least stable inputs.

The better second candidate is adjacent to the first: same team, same systems, similar shape. It reuses what was learned, it strengthens the same ownership arrangement, and it produces a second result quickly enough that the programme keeps its sponsor.

Ambition is better spent on the third or fourth, when the operating model has been proven on something that could survive being wrong.

The people part, which is the part that fails

An automation removes work from a specific team, and that team is usually the one asked to validate it. If nobody has said what happens to the freed capacity, the incentive to find problems is stronger than the incentive to find fixes.

This is not resistance and it should not be treated as a communications problem. It is a rational response to an unanswered question, and the answer — redeployment to work the team considers better — has to be given by someone with the authority to give it, before the pilot rather than after.

Where that answer exists, the same team becomes the best source of exception cases, because they know the process and now have a reason to want it working. Where it does not, every edge case arrives as an objection.

What a real pilot measures

A pilot worth running measures three things the demo does not: the true exception rate against real volume, the time the process takes end to end including the waits, and how often the source systems are unavailable or slow.

Those three numbers size everything downstream — the return, the build effort, the support load. Without them the business case is an estimate dressed as a calculation.

A fourth is worth adding where it applies: how often the process output has to be corrected today. An automation inherits the accuracy of the process it copies, and a manual step with a four per cent correction rate does not become accurate by being automated — it becomes fast and four per cent wrong.

They also cost almost nothing to collect, because they can be measured on the manual process before a line of automation is built. Most pilots skip this and then spend months arguing about a saving nobody can substantiate.

What changes at scale

The first automation is a project. The tenth is an estate, and an estate has properties a project does not: a change to a shared system breaks several things at once, and the support load is continuous rather than a phase.

Organisations that get past the pilot stage almost always did one thing early — they treated the operating model as part of the first delivery rather than something to formalise later.

That is why the honest question after a successful pilot is not which process is next. It is whether anything exists to run the next one on.

Where to start

With the numbers the pilot did not produce. Real volume, real exception rate, real system availability, and a named owner for what already runs.

A process automation assessment establishes those and calculates the return with local costs, which is what turns a stalled pilot into a sequenced automation plan rather than a second demo.

Frequently asked questions

Why do automation pilots not reach production?

Because a pilot proves feasibility, which was rarely in doubt. It does not test error handling, monitoring, credential management or the manual fallback, and those are what production requires.

What does a demo fail to test?

Slow source systems, empty fields, ten times the volume, and running unattended at night. It also never tests how a failure gets noticed and who gets called.

What has to exist before going to production?

Designed error handling, monitoring somebody actually receives, credentials that do not belong to a person, and a documented way to run the process manually.

Which process should be the second one?

One adjacent to the first — same team, same systems, similar shape. The biggest available saving is usually the process with the most exceptions and the least stable inputs.

What should a pilot measure?

The true exception rate against real volume, end-to-end elapsed time including waits, and how often source systems are slow or unavailable. All three can be measured on the manual process.

Who should own an automation in production?

A named owner accountable for changes when source systems update and for wrong results, with a budget attached. Without one it belongs to whoever built it, and then to nobody.