Testing automations: quality control for processes that run alone

An automation that runs unattended has no user to notice when it goes wrong. That single property is why testing it is a different discipline from testing an application, and why the tests that matter are the ones covering conditions nobody will be watching for.

Below: what to test that an application would not, building a regression suite against systems you do not control, test data, what to check after every change, and who runs it.

Testing is the step most automation projects compress when the date moves, on the reasoning that the process is simple and the build is small. Both are usually true, and neither is the reason testing matters here.

What matters is that the automation will run three hundred times before anyone looks at it closely again.

That changes what a test is for. In an attended application, testing reduces the number of defects a user will encounter and report. Here it is the only mechanism that exists for noticing at all, because the alternative discovery route is a downstream consequence weeks later.

What to test that an application would not

Applications are tested against inputs a user could provide. An automation has to be tested against conditions the environment provides: a slow response, a session that expired mid-run, a file that is present but still being written, a source system returning yesterday's data without saying so.

None of these appear in a functional specification, and all of them occur within the first few months of running.

The second category is volume behaviour. An automation that works on ten records and is run against four thousand can fail in ways that have nothing to do with logic — timeouts, memory, a downstream system rate-limiting it, a report that takes longer to generate than the wait allows.

The regression suite against systems you do not control

The difficulty is obvious: the systems change and they are not yours. The temptation is to test against copies, which removes the difficulty and most of the value with it.

The arrangement that works is layered. A small set of tests against a real non-production instance where one exists, a larger set against recorded responses for the logic, and a single end-to-end run in production against a known safe record — a designated test customer, a reversible transaction — executed on a schedule.

That last one is what catches the class of failure the other layers structurally cannot: the supplier changed something and nobody told anyone.

Test data is the hard part

Realistic test data is difficult for a familiar reason — real records contain personal data, and copying production into a test environment is a common practice that quietly creates an exposure. Habeas Data in Colombia and the federal data protection law in Mexico both apply to the copy exactly as they apply to the original.

Synthetic data avoids that and misses the cases that matter, because the awkward records are awkward in ways nobody thinks to invent: the trailing space, the accented character, the field that was reused for a different purpose in 2019.

The workable middle is a masked extract built from real records — real shapes, real anomalies, substituted identifiers — refreshed periodically. It takes a day to set up and it is the difference between tests that pass and tests that mean something.

What to check after every change

Not everything, and not nothing. The set worth running on every change is small: the happy path, the two or three exception cases that have actually occurred in production, and the outputs other systems consume.

That third one is the most commonly skipped. An automation whose logic is correct can still break the thing downstream by changing a file name, a column order, or a date format — and the automation's own tests will all pass.

Anything that has broken once belongs in this set permanently. A defect that recurs after being fixed is the clearest available signal that the suite is not covering what it should.

Kept that way, the suite grows in the right direction on its own. It ends up describing the failures this particular estate actually has, which no generic testing checklist can do.

The environment problem, stated plainly

Most organisations in the region do not have a usable non-production copy of the systems their automations depend on. The core banking instance, the supplier portal, the government filing site — none of these come with a test environment, and the ones that do are often stale enough to be misleading.

Pretending otherwise produces a test plan that cannot be executed, which is worse than admitting the constraint, because it defers the discovery to the week before go-live.

The honest response is to design the testing around what exists: mock the systems with no test instance, use the real one where it exists, and accept that a defined slice of the risk will only be observable in production — then instrument production accordingly, which is what the next section is about.

The check that runs in production

Beyond testing, an unattended automation needs a continuous assertion about its own output: the number of records processed is within an expected range, the totals reconcile, the run finished inside its normal window.

These catch the failure mode that no test catches — the automation ran successfully and did the wrong thing, because the input changed shape. A run that processes eleven records when it normally processes four hundred has not failed by any technical measure.

The threshold does not need to be sophisticated. A range derived from the last few months, with an alert when a run falls outside it, catches most of this class at almost no cost.

Who runs it

Whoever owns the automation, which is a question worth having answered before this point. In practice the tests get written by the build team and then run by nobody, which is the same failure as not having them.

Automating the suite is the obvious answer and it is only half of one — someone still has to look at the result and act on a failure. An automated test suite reporting into an unwatched channel is the testing equivalent of an alert nobody receives.

The lightest arrangement that holds: the suite runs on a schedule and before every change, failures go to a named person, and the result is reviewed as part of whatever cadence already exists for the estate.

Where to start

With the exception cases that have already occurred. They are the highest-value tests available, they cost nothing to identify, and they are already documented in the support history. Turning each one into a repeatable test is usually a morning of work per automation.

A process automation assessment reviews the existing estate alongside new candidates, so the automation plan accounts for what it takes to keep the automations correct, not just to build them.

Frequently asked questions

How is testing an automation different from testing an application?

An application is tested against what a user could do. An unattended automation has to be tested against what the environment does — slow responses, expired sessions, stale data, and volumes far above the sample.

How do you test against systems you do not control?

In layers: a few tests against a real non-production instance where one exists, more against recorded responses for the logic, and one scheduled end-to-end run in production against a known safe record.

What test data should be used?

A masked extract from real records — real shapes and anomalies with substituted identifiers. Copying production creates a data protection exposure; purely synthetic data misses the awkward cases.

What should be checked after every change?

The happy path, the exception cases that have actually occurred, and the outputs other systems consume. That last one is most often skipped and breaks downstream while all tests pass.

What catches a failure that tests cannot?

A production assertion on the run itself: record count within an expected range, totals reconciling, and completion inside the normal window. A run processing eleven records instead of four hundred has not technically failed.

Who should run the tests?

The named owner of the automation. Written by the build team and run by nobody is the same outcome as not having them, and an automated suite still needs someone who acts on a failure.