Your team made a big decision last week. Probably based on data. A metric. A report. A dashboard someone pulled up in a meeting. Everyone nodded because numbers feel objective.
But when was that data collected? What cohort does it represent? Has anything changed since then?
Nobody asked. Numbers don't come with expiration dates.
The data freshness problem
Product decisions based on last quarter's data are like driving with a rearview mirror. You can see where you were. You can't see where you're going.
Usage patterns shift. Customer demographics change. The feature you launched last month altered behavior in ways that make old baselines meaningless. But the dashboard still shows the old numbers until someone updates the query.
When data becomes misleading
Data doesn't go from useful to useless overnight. It degrades slowly. A retention metric from January is slightly off in February. By April, it's telling a different story entirely. But it still looks like a solid number on a dashboard.
The most dangerous data is data that looks current but isn't. A metric displayed on a live dashboard feels real-time even if the underlying cohort is six months old.
The aggregation problem
Averages hide decay. "Average NPS is 45" might mean your longtime users love you and your recent signups hate you. The average hasn't changed. The reality underneath it has completely shifted.
Teams that rely on aggregated metrics miss the fact that their product is serving two completely different populations with two completely different experiences.
What to do about it
Put a timestamp on every metric you use to make a decision. Not when the dashboard was built. When the data was collected. Then ask: has anything material changed since this data was gathered?
If you can't answer that question, the data isn't evidence. It's comfort.
Tom Pinder