The insight around which this post is built might expose some serious internal government dysfunction. Even though that’s the goal of many The Audit pieces, I think this one goes both deeper and wider.
As you may recall, I recently assessed the usefulness of result metrics reported by a few government departments. Many entries from Health Canada and Canadian Heritage were depressing examples of narrowly-focused box-checking that didn’t even try to connect to the real world. The examples we saw from Department of Veterans Affairs on the other hand were an encouraging contrast.
But that was all more or less random. For all I know, the samples I chose were one-off anomalies and failed to representative of the work of individual departments or of the government as a whole.
To fix that, I thought I’d expand my analysis. So I used the complete 2024-2025 dataset from the Government of Canada’s InfoBase open datasets relating to departmental results reports. That dataset contains activity names and descriptions for all programs authorized to achieve policy goals. For each activity, there’s also a target status, whose value could be “met”, “not met”, “to be achieved” or “not available”.
To give you an idea of what I’m talking about, here are some examples of target descriptions that I consider meaningful:
Degree to which Agriculture and Agri-Food Canada advances the resolution or the mitigation of market access barriers, World Trade Organization disputes and technical trade issues
Percentage change in the economic performance of the agriculture and agri-food sector
Revenue growth rate of firms supported by ACOA programs
% of supervisors who report improved performance of employees; in particular for those employees in management and leadership development programs
And here are some targets that I felt were largely meaningless:
Percentage of surveyed Indigenous partners who indicate their experience engaging on policy initiatives led by the Canada Water Agency as meaningful.
Number of international commitments through which Canada works with partners to address strategic peace and security challenges.
Number of women’s organizations and women’s networks (international and local) advancing women’s rights and gender equality that receive GAC support for programming and/or institutional strengthening
Amount of climate finance announced for specific initiatives to help developing countries transition to low-carbon, climate-resilient economies
Dollar value of CALA loans registered to established and beginning farmers per fiscal year ($millions)
Notice how the only events referenced in those examples were designed as part of a process. There’s nothing indicating that any real world problems were actually addressed. There’s no direct suggestion that water is cleaner, foreign poverty and suffering are alleviated, women’s right are actually advanced, climate harms are measurably reduced, or agricultural outcomes are improved. Good things might have happened, but there’s nothing in those metrics that proves it.
Since there were more than 2,700 distinct results reported for 2024-2025, I’m unable to manually assess each one. So I tasked the Claude AI chat model with the job instead, asking it to rate each official indicator of success as “largely meaningful”, “largely meaningless”, or “neutral”. That last option was to avoid forcing Claude to make a weak decision under pressure. I’m really only looking for clear examples one way or anther.
When its analysis was done, Claude had classified 1,571 of the results as neutral, 609 and meaningless, and 545 as meaningful. Obviously, we can’t blindly rely on those numbers. But after looking through a sample, I’d say that the AI model appeared to understand what I was looking for and was at least mostly on target. You’re welcome to take a look through the results yourself.
What that means is that just 545 of the 2,725 activities (20 percent) were unambiguously useful.
There’s more. Among all results that Claude designated as “meaningful”, only half were officially self-rated as meeting their targets. By contrast, department officials gave the thumbs up to nearly 59 percent of the indicators that were designated “meaningless”.
Given the scale of the data we’re working with, that’s a statistically significant divergence. Perhaps it suggests that people are more comfortable overrating the success of unrealistic targets. Or maybe the higher scores are simply the product of how easy it is to satisfy fake requirements.
I also asked Claude to rate each activity success indicator as either “within government control”, “outside of government control”, or “neutral”. My thinking is that, by and large, targets that lie within the reach of governments are more likely to be detached from real-world outcomes.
When I ran the numbers, I discovered that only 28 percent of those activities with meaningful result indicators lay within government control. By sharp contrast, nearly 86 percent of the meaningless indicators were within government control!
I can’t tell whether departments looking to use meaningless targets to game the reporting system consciously look for targets that they control, or whether it just works out that way. But there is an obvious connection between fake, box-checking results and domains that aren’t touched by the real world.
Here’s a simple visualization of those two metrics:
Finally, I took a look at results at the department level. Global Affairs Canada, Employment and Social Development, and Public Works and Government Services were responsible for the highest concentrations of meaningless targets: 63, 55, and 34 respectively. Department of National Defence and Correctional Service of Canada scored 54 and 34 meaningful targets (respectively).
In any case, I’ll repeat what I’ve said in the past: governments have no business legislating policies whose outcomes can’t be meaningfully measured: they’re wasteful by design. Even if all those garbage metrics are the products of inadvertent coincidence, their existence undermines government credibility.
You might also enjoy:
How Much Value Does Your Job Generate?
It was arguably the invention of the printing press that - in the Early Modern period -doomed countless powerful and wealthy monasteries whose primary economic function had been the manual transcription of books. As a recent article by John Carter would have it, the one-two punch of the internet and artificial intelligence is in the process of unleashin…
Are Parliamentary Committees Underachieving?
We can be forgiven for assuming that the Canadian government exists to spit out legislation that’s poorly planned, overtly political, and often legally dubious.
How Compliance with the Canadian Gender Budgeting Act Drains Government Capacity
I don’t have enough information to share a useful opinion about the intrinsic value of the Canadian Gender Budgeting Act itself. Although there are some questions I’d love to be able to answer, like: