Carney government testing use of AI in prisons to create profile reports of offenders

But the prison trial, which CSC says has not yet been used in real cases, is raising concerns from AI experts, criminal defence lawyers and the federal NDP’s public safety critic, who argue a widespread adoption could lead to crucial errors, exacerbate racial biases and put offenders and victims at risk.

Criminal profile reports, as they are called, are detailed “foundational documents” prepared by CSC staff during a prisoner’s intake process that identify risks and play a role in major decisions like access to programs and likelihood of parole.

Drawing from scores of official documents, they include details about an offender’s criminal history, the circumstances of their crimes, patterns of violence or behavioural, mental health and addiction issues, family and social background, trauma history, education and employment records, and even victim impact statements.

“This is what defines your offence cycle,” criminal defence lawyer Nora Demnati said of those reports. “It will have an impact on everything else that comes.”

Following queries from the Star, a spokesperson for CSC said it’s “exploring whether AI can help staff review and organize information from existing documents more efficiently when preparing a criminal profile during intake,” while maintaining “human review, quality and accuracy.”

“The focus is on helping staff with time-intensive document review, analysis and information extraction from source materials used to prepare the criminal profile,” wrote Esther Mailhot, who added an evaluation is expected to be done by the end of June and no final decision is made.

The Carney government inked a $123,000 contract with consulting giant Accenture to run the pilot from February to end of May, according to documents released in Parliament in response to questions from a Conservative MP about details of all federal AI contracts.

Accenture is only using “anonymized sample documents” or “artificially created” information for the trial, Mailhot said, and the tool “has not been used in any operational setting.”

But if Ottawa adopts this tool moving forward, mistakes are “very likely” said Jennifer Evans, principal at the consultancy and research firm PatternPulse AI.

That’s because AI is a “probabilistic technology” based on pattern recognition, she said, and “it is always architecturally going to make errors.”

“There is no dispensing with that. No amount of training, no amount of what people will call better data will ever erase the issues of hallucination, and in fact, when proper nouns, name, information, where there’s a lot of very specific components to the data, the hallucination rate is higher,” Evans told the Star.

And putting in the work to catch those mistakes could cancel out any time savings, she said.

“Errors are hard to detect, and they propagate, and unless you do have somebody paying very close attention to the accuracy of each individual record, you’re not going to know if it was conducted properly or not, and that almost obviates the utility of the software itself in this particular use case,” Evans said. 

That’s why the Carney government should slow down and consult widely, including with the CSC union, its lawyers and the Privacy Commissioner of Canada before going further, said NDP MP Jenny Kwan, the party’s public safety critic. Neither the Union of Safety and Justice Employees or the Office of the Privacy Commissioner have been consulted yet, they told the Star.

Kwan warned of a multitude of legal concerns that go both ways and can have a “cascading impact”: Violating the rights of inmates if mistakes are added to reports, on one hand, or hurting victims and prison staff if crucial information is missed by the AI summaries, on the other.

“When you have those kinds of risks associated with correctional policing matters, you can imagine what the huge ramifications might be,” Kwan told the Star. “You could potentially compromise people’s legal rights.”

AI use by the Canadian government, including military contracts with the controversial American tech company Palantir, have been under scrutiny in recent weeks.

Howard Sapers, the executive director of the Canadian Civil Liberties Association who was Canada’s Correctional Investigator from 2004 to 2016, said he commonly received complaints about outdated information in offenders’ files, an issue that will not improve with mistake-prone AI.

“That stale dated or inaccurate information, when it gets replicated, it replicates the same problem, and that can result in a negative recommendation for parole. It can result in a higher than necessary security classification, and it can result in somebody not being able to get into a useful correctional program that would help them avoid criminality,” Sapers told the Star.

Demnati, a member of the Canadian Bar Association’s committee on imprisonment and release, said she is also concerned introducing AI to the creation of criminal profile reports could exacerbate biases against Black and Indigenous people.

“We already have concerns with assessments that are being done with humans,” Demnati told the Star.

CSC said that the “ability to assess potential bias was constrained” given the limited testing, but said that issue and “ethical considerations” have “been identified as risks, and more comprehensive testing would be required if the work proceeds further.”

Citing a recent auditor general report that concluded CSC “failed to identify and eliminate systemic barriers that persistently disadvantage certain groups of offenders,” Public Safety Minister Gary Anandasangaree told the Star “there’s a world in which individual biases may be better addressed through a neutral system, as opposed to individual human decision making,” though he stressed humans are still getting the final say.

That’s a “bit naive,” Demnati said in response.