Two. That's the risk threshold that now triggers independent peer review for federal automated decision systems in Canada. The Treasury Board of Canada Secretariat updated the Algorithmic Impact Assessment and published a Guide on Peer Review to sit alongside the Directive on Automated Decision-Making, formalizing a questionnaire that scores systems from 1 to 4 and that requires peer review for any system rated 2 or above. Departments must complete the AIA before deploying public-facing automated decision tools. The change turns a self-assessment into a two-step gate, while leaving important questions about scope and implementation open.

Departments now must complete the Algorithmic Impact Assessment before deploying public-facing automated decision-making, which means answers to a structured questionnaire determine whether a system is subject to independent review.

How the two-step workflow actually works

The Directive on Automated Decision-Making, published on April 1, 2019 and extended across federal institutions in April 2020, set a mandatory, risk-based regime for public-facing automated decisions. At its centre is the Algorithmic Impact Assessment, a structured questionnaire that asks about a system's design, the underlying algorithm, the kind of decision it makes, expected impacts, and the data used. Responses are converted into a numerical score that places the system into one of four risk levels. When a system scores 2 or higher, the framework requires an independent peer review before the system may be put into service.

The Treasury Board of Canada Secretariat oversees the AIA's design and ongoing updates, and members of its oversight team have described the AIA as a work in progress as the instrument's criteria and scoring logic have been refined since introduction. In parallel, the government has published supporting materials on its digital government portal, including a Guide on the Use of Generative AI, the AIA tool itself, a minimum workable product of the Government of Canada AI Register, and policy implementation notices for specific enterprise tools.

The Guide on Peer Review, produced with Treasury Board staff and led by the University of Ottawa, adapts established practices from privacy impact assessments and research ethics boards. Its aim is to standardize how technical, ethical, and social risks are evaluated by reviewers who are independent of the project team. The Guide sets out best practices, a model review process, and an interdisciplinary approach intended to bring technical and social science expertise to bear on departmental assessments.

Academic contributors who helped design the Guide stressed that cross-disciplinary review is a core value. Independent reviewers and civil society analysts have said the model improves consistency across departments while making reviews more likely to surface non-technical harms such as impacts on eligibility and family separation.

Mastheads

In practice, several federal institutions have already used the AIA and the peer review trigger: employment and transportation agencies, the Department of Veterans Affairs, and the Royal Canadian Mounted Police have completed AIAs for automated systems they use.

Publicly available AIAs have also become a resource beyond central agencies. Lawyers working on immigration and refugee cases consult them to understand how automated systems affect client outcomes. That transparency has practical consequences: it gives external actors a document to read when assessing how an automated decision contributed to an eligibility outcome or a family separation decision.

Strengths and the limits the framework still faces

The system is influential because it creates a clear procedural sequence: score first, review second. That sequence reduces ambiguity for departments and sets a consistent bar for when independent oversight is required. The Guide on Peer Review brings comparative institutional practices into federal work, and the availability of complementary tools on the government portal gives departments concrete instruments to meet their obligations.

Still, the framework has clear limits. Observers point out that the AIA depends heavily on departmental self-assessment to produce quantitative risk scores, a trait it shares with many international AI governance tools. Self-assessment raises obvious questions about consistency and incentives, especially when internal pressures favour deployment. The Directive itself is explicit about scope: it focuses on public-facing automated decision-making and doesn't cover many internal government processes or national security systems. Those limits mean the regime can leave important decision-making systems outside its formal safeguards.

Treasury Board staff and outside researchers have continued to iterate on the AIA's questions, scoring logic, and implementation guidance to address these gaps. The process of refinement is ongoing, which reflects both the complexity of measuring algorithmic risk and the desire to make the scoring and review steps more robust in practice.

The Guide on Peer Review has been released for departmental use, but its effectiveness will depend on how departments apply it and how genuinely independent reviewers are in the field.

What to read this framework as

Read as a governance pattern, the Canadian approach is an orderly attempt to separate two decisions that too often get conflated: the internal choice to deploy a system, and the independent question of whether its technical and social risks have been adequately considered. That separation is the working logic behind what some practitioners call a decision subtraction approach, where the review removes or isolates the decision to ensure independent scrutiny.

That logic is practical in a government context. It channels scarce expert review capacity toward systems that score above a defined threshold, rather than requiring uniform external review of every tool. The trade-off is the reliance on the initial, internal scoring exercise. If that exercise understates risk, the peer review gate may never be triggered. The Canadian framework accepts this trade-off while trying to reduce its downside through updated scoring guidance, a university-led peer review Guide, and an inventory of complementary resources hosted on the Government of Canada digital government portal.

Related Articles

Two is the single number that governs whether federal automated decision systems receive independent scrutiny in Canada. Paired with the AIA, the Guide on Peer Review and Treasury Board oversight, it's now the concrete hinge between departmental self-assessment and independent review. The real test will be implementation: whether departments reliably trigger and integrate genuinely independent reviews when systems score 2 or higher, and whether those reviews change deployment decisions.

This article was created with AI assistance.