Assessing Government Departmental Self-Assessments

As a (commendable) exercise in transparency, federal government departments are required to file annual reports. Well, not just reports. There’ll often also be plans, reviews, compliance documentation, and audits.

I’ve already discussed the costs involved with generating all those publications (not to mention the misaligned incentives resulting from setting targets internally). But there’s another problem that might loom yet larger: who’s actually reading all this stuff?

Now it’s true that Treasury Board works very hard to make that information as accessible as possible. The Results tab on the GC InfoBase page for a given department will definitely make it easier to quickly visualize an organization’s programming performance. But even with all that information, we’re still missing context.

So I decided I’d feed a small subset of that data to an AI model to see what insights would come out the other end. I went with departmental results reports from 2019-20 and 2024-25 from Canadian Heritage. For good measure, I also uploaded InfoBase results data. I then asked for assessments based on these points:

• The extent to which key deliverables were met or missed

• Whether there were changes to frames of reference that make it difficult/impossible to compare

• Were any statements or descriptions in the documents in conflict with reality?

• Were any predictions proven wrong?

• Were any goals/accomplishments more about words and ideology than real-world changes?

Here are some quick observations:

1. Most formal departmental targets were met, but that’s because the targets were usually activity-based rather than outcome-based.

2. There are few outright factual conflicts with reality, but there are many instances where spending, consultation, or representation are treated as evidence of success without demonstrating real-world effects.

3. Some broad ambitions—especially around social cohesion and language vitality—show weaker evidence of success than the reports imply.

4. The 2024–25 report contains substantially more ideological and values-oriented language than the 2019–20 report, and success is more often defined in terms of alignment with those values rather than measurable societal outcomes.

And here’s the full response:

---

The reports generally demonstrate that Heritage Canada became better at reporting activities, consultations, funding distributions, and inclusion initiatives, but not necessarily better at demonstrating concrete societal outcomes.

---

1. Changes in the Frame of Reference

This is perhaps the biggest issue.

The 2019–20 report focused heavily on:

• cultural exports

• arts funding

• official languages

• Indigenous languages

• multiculturalism

• youth engagement

• participation metrics

The 2024–25 report increasingly focuses on:

• inclusion

• equity

• representation

• accessibility

• anti-racism

• gender-based analysis

• online harms

• platform governance

• diversity of creators and audiences

Many performance indicators shifted from measuring outputs and participation toward measuring representation and inclusiveness.

For example:

2019 framing

• How many projects were funded?

• How many Canadians participated?

• How much Canadian content was produced or exported?

2024 framing

• Were underrepresented groups included?

• Were equity objectives advanced?

• Were funding tools modernized?

• Were diverse voices represented?

These are not necessarily bad metrics, but they make longitudinal evaluation difficult because the underlying definition of success changed.

---

2. Deliverables That Were Clearly Achieved

Several initiatives can reasonably be called successes.

Indigenous Languages Act implementation

In 2019 the Department predicted that the Act would become a major pillar of activity.

By 2024:

• permanent funding streams exist

• Indigenous Screen Office funding was secured

• implementation structures were created

• language revitalization programs expanded

The prediction that Indigenous-language policy would become institutionalized was largely correct.

Creative Export Strategy

The 2019 report highlighted:

• export funding

• Frankfurt Book Fair preparations

• international market access

Those programs survived and expanded over the next several years.

The prediction that exports would remain a major policy focus proved correct.

Digital Policy Agenda

The 2024 report emphasizes:

• Online Streaming Act

• Online News Act

• platform regulation

The groundwork for this direction was already visible in earlier Heritage planning documents.

The Department successfully shifted from traditional cultural policy toward digital-platform governance. Whether the policies themselves are successful is a different question.

---

3. Goals That Were Not Clearly Met

Several areas appear weaker.

Official Languages

The Department continued to report activities and agreements.

However, broader demographic trends moved in the opposite direction:

• declining relative weight of French outside Quebec

• continuing assimilation pressures in minority-language communities

• ongoing concerns about bilingual service delivery

The reports frequently measure:

• agreements signed

• money spent

• organizations funded

rather than actual language retention outcomes.

This creates a gap between activity and demonstrated success.

Social Cohesion and Inclusion

One recurring departmental goal is a more inclusive and socially cohesive Canada.

Yet between 2019 and 2025 Canada experienced:

• heightened polarization

• rising antisemitism

• rising anti-Muslim incidents

• increased social conflict around race, gender, and identity issues

The reports rarely engage with these outcomes.

Instead, success is generally defined by:

• number of funded projects

• number of consultations

• awareness activities

rather than measuring whether social cohesion actually improved.

This is probably the largest gap between reported accomplishment and observable societal outcomes.

---

4. Predictions That Were Proven Wrong

Several implicit assumptions from 2019 did not age well.

“Inclusion programming will strengthen social cohesion”

This appears throughout Heritage reporting.

The evidence by 2025 is mixed at best.

Many indicators measure participation in programs, but Canada became more politically and culturally polarized during the same period.

The reports provide little evidence that funded inclusion initiatives produced measurable improvements in cohesion.

Public Service Media Stability

2019-era reporting assumes a continuing central role for traditional broadcasting.

By 2025:

• television audiences fragmented further

• streaming services gained market share

• CBC audience challenges persisted

This is partly why Heritage shifted toward Online Streaming Act implementation. The original assumptions about the broadcasting environment proved too optimistic.

---

5. Statements Potentially in Conflict with Reality

I would not say there are outright falsehoods.

However, there are several areas where the reports arguably present inputs as outcomes.

Example:

funding was provided to support inclusion

That is factually true.

But it does not demonstrate:

• reduced discrimination

• increased trust

• improved cohesion

• improved life outcomes

The reports often move rhetorically from:

“money was spent”

to

“progress was achieved”

without showing the causal link.

The same issue appears in:

• anti-racism programming

• multiculturalism programming

• youth engagement programming

• diversity initiatives

This is common across many federal departments, not unique to Heritage.

---

6. Growth of Ideological Language

This is where the difference between the reports is most noticeable.

The 2019 report certainly contains language around:

• diversity

• inclusion

• reconciliation

But it remains largely program-focused.

By 2024, language emphasizing:

• equity

• representation

• intersectionality

• accessibility

• anti-racism

• gender-responsive approaches

appears much more frequently and often serves as a success criterion in itself.

A recurring pattern is:

2019

We funded X projects.

2024

We funded X projects that advanced equity, diversity, accessibility and inclusion objectives.

The latter may be entirely legitimate policy, but it is a shift from measuring outputs toward measuring alignment with normative values.

---

7. The Most Important Structural Weakness

The reports tend to evaluate:

• activities

• consultations

• funding decisions

• partnerships

• participation rates

far more rigorously than they evaluate ultimate outcomes.

For example:

A strong outcome measure would be:

Percentage increase in Indigenous-language speakers.

A weaker measure is:

Number of Indigenous-language initiatives funded.

The reports heavily favor the second type of metric.

As a result, Heritage Canada generally appears successful according to its own indicators because the indicators are often measuring departmental activity rather than societal change.

---

---