The right question is not how often a piece of data can be refreshed, but how often it needs to be refreshed for the decision it supports to stay sound. That frequency is set by the business, not by the infrastructure.
What follows: why real time by default is expensive, how the useful frequency is determined, what requires judgement, and how to sequence the project.
In almost every data project somebody asks for information in real time. When asked which decision changes if the data arrives an hour late, the usual answer is none.
That does not invalidate the request: it reveals it for what it is, a reasonable preference for having the most recent figure. But a preference and a requirement cost very different things.
Why real time by default is expensive
Continuous refresh demands a different architecture: event processing, fault tolerance in the pipeline, and permanent monitoring. It is a valuable capability and it is also the most expensive to operate.
The cost is not only infrastructure. A continuous flow forces inconsistencies to be resolved in the moment that a batch process would have caught with a prior validation, and that shifts complexity to where it is hardest to correct.
When that capability is applied to an indicator reviewed in a monthly committee, the organisation pays for an architecture of seconds to support a decision of thirty days.
How the useful frequency is determined
Start from the decision. Every piece of data exists to support a concrete decision: replenish stock, contact a customer, authorise a transaction, review a budget. Identify which one.
Ask how often it is taken. If the decision is made once a day, data refreshed hourly is already more than enough. If it is made while serving a customer, the requirement is different.
Estimate the cost of deciding on old data. This is the question that orders the conversation. If acting on yesterday’s information does not change the outcome, daily is the right frequency and further debate is time lost.
Record the answer alongside the indicator. Frequency stops being a technical detail and becomes part of the data’s definition, visible to whoever consumes it.
What requires judgement
There are data where freshness is not a preference but a condition. Fraud detection, the availability of a running service and plant control lose all their value if they arrive late, and there the cost discussion runs the other way.
It is also worth separating freshness from query latency. A dashboard can answer in one second on yesterday’s data, and that misleads whoever uses it: speed of response says nothing about the age of the information.
Which is why a figure on screen should carry its timestamp. A number without a date invites the assumption that it is current, and that assumption is what produces wrong decisions from correct information.
What changes by source
Not every source permits the same frequency. An in-house transactional system can expose changes continuously; a third-party service may impose query limits that set the real ceiling.
When the source constrains, the frequency promised to the business has to reflect that limit. Promising a five-minute refresh on a source that allows two queries an hour is a promise that breaks on day one.
And if the data comes from a manual process — a load somebody runs — freshness depends on a person, not on the architecture. Better to say so than to dress it up as an automated process.
How to sequence the project
Build the inventory of indicators in use and, for each, record three things: which decision it supports, how often that decision is taken, and how often it is refreshed today.
The resulting table usually shows two mismatches in opposite directions: indicators refreshed far more often than needed, and critical indicators that depend on a weekly load.
Correct the second group first. Lowering the frequency of what is over-provisioned frees cost, but raising the frequency of what arrives late changes decisions.
What has to exist first
An owner per indicator, able to say which decision it supports. Without that person the conversation turns technical and ends up settled by preference.
And an acceptance that the right answer is usually modest: most management indicators live well on a daily refresh, and saying so plainly is what leaves budget for the few that genuinely need continuity.
Three frequencies that cover almost everything
Continuous. The data updates as it happens. It is justified when the decision is made in the instant and delay changes the outcome: authorising a transaction, detecting fraud, reacting to a service failure. It is the most expensive to build and to operate, which is why it is worth reserving.
Intraday. Several refreshes a day, typically hourly. It covers most of the operation: stock, receivables, service, availability. It offers room to react within the day without requiring an event architecture.
Daily. One consolidated load, usually overnight. It is enough for almost every management and board indicator, and it has an advantage rarely mentioned: it allows validation before publication, so what the business sees has already passed a quality check.
Most of the mismatches an inventory turns up are resolved by moving indicators between these three categories, without building anything new.
When is real time justified?
When the decision is made in the moment and delay changes the outcome: fraud, service availability, operational control. Outside those cases it is usually a preference, not a requirement.
Are freshness and dashboard speed the same thing?
No. A dashboard can answer in one second showing yesterday’s data. Which is why the data’s timestamp is worth showing next to the number.
What if the source does not permit the frequency requested?
That limit sets the real frequency, and it has to be declared. Promising a refresh the source cannot support produces a promise that breaks immediately.
Where is it best to start?
With an inventory that pairs, for each indicator, the decision it supports against its current refresh frequency. The mismatches surface on their own, and almost always in both directions.
