OpenAI – which created each ChatGPT and Whisper – instructed RNZ: “Addressing hallucinations is an ongoing area of research.”
Photo: Jonathan Raa / NurPhoto / NurPhoto by way of AFP
Police officers trialling generative AI considerably misused it and the challenge was dumped, solely to be restarted six months later after officers used unapproved AI models.
The police high-tech crime group has now been utilizing the accredited Whisper AI in investigations for 9 months after the challenge was restarted with additional controls.
The trial and use of the AI programs have been revealed in Official Information Act paperwork launched to RNZ.
Officers examined Whisper on translating and transcribing non-English language audio for six months, as much as March final 12 months.
The paperwork say Whisper is 45 % inaccurate on Māori and Pacific languages and can’t be used on these.
But officers broke that ban on utilizing it to transcribe English throughout the trial, so police dumped Whisper in March final 12 months.
“Trial misuse demonstrated limitations of relying on behavioural controls alone,” stated a doc.
Police restarted it in September, and instructed RNZ that solely educated accredited employees can use it and its output isn’t used in proof.
Hallucinations
The use of AI for legislation enforcement investigations on this method is controversial around the globe, with critics saying it’s too unreliable with such excessive stakes. But proponents – together with firms promoting AI – say it saves officers enormous numbers of hours in paperwork.
NZ Police had a ban on generative AI (GAI) till mid-2024 and stated it might be very cautious about introducing any programs.
A Cornell University research in May 2024 discovered Whisper AI hallucinated one % of the time, together with making up racial commentary, violent rhetoric and imagined medical therapies.
OpenAI – which created each ChatGPT and Whisper – instructed RNZ: “Addressing hallucinations is an ongoing area of research.”
Whisper was an open-source analysis mannequin launched in 2022.
“As we’ve consistently stated publicly, speech recognition systems are not perfect and should be evaluated carefully for their intended use case,” stated an organization spokesperson on Tuesday.
It stored bettering models and had newer ones, like GPT-Realtime-Whisper, that it stated diminished hallucinations and had been extra correct throughout varied languages.
The trial started six months earlier than police launched their first coverage on generative AI in March 2025. It started three months after police stated that they had no plans to carry a ban on utilizing GAI.
A spokesperson for the Muslim neighborhood stated they trusted police and supported use of expertise, however Whisper clearly was used with ethnic teams.
“We should have been consulted,” stated Abdur Razzaq of the Federation of Islamic Associations.
‘Trial terminated’
The trial concluded that the AI “was efficient and valuable for intel”.
But the OIA paperwork included a minute from the police headquarter’s tech watchdog group which additionally stated: “Significant misuse famous (English audio processed regardless of restrictions).
“Continued use not endorsed; trial terminated.”
That identical month, a March 2025 public presentation by police about GAI stated the results of misuse or errors had been excessive for the general public.
They have additionally had inside warnings that the possibility of irreversible harm from utilizing AI – together with to the fame of police – required excessive due diligence from the beginning.
“This is even more crucial given there is no specific legal framework for the use of AI in New Zealand,” stated a warning.
‘Unapproved options’
Whisper got here up once more on the tech watchdog group in September 2025.
“Memo presented seeking reconsideration of Whisper AI following earlier termination,” stated a minute.
“Strong operational demand acknowledged; turning off the tool increased risk of staff using unapproved alternatives.”
NZ Police director of the nationwide legal investigations group detective superintendent Keith Borrell didn’t point out the misuse in his assertion to RNZ.
Having famous Whisper’s worth for investigative work, Borrell stated: “This tool was briefly turned off while the trial results were analysed, additional safeguards were put in place and guidance was updated.”
Police didn’t comply with an interview.
‘Not conscious of any materials points’
Police thought-about Whisper was probably the most appropriate mannequin to check, stated Borrell.
As for US studies about it fabricating content material that by no means existed, he stated: “We are not aware of any material issues to date arising from the inaccuracies inherent in automated translation and transcription.”
Users of it now needed to have accomplished obligatory AI consciousness coaching and been made conscious of the “limitations of these types of tools”. The use for recordings primarily in Te Reo Māori or Pacific languages was not authorised.
“Text is added to the outputs to make it clear that transcripts must not be relied upon for critical decision making or evidential purposes without verification by an appropriately qualified person,” stated Borrell.
The police coverage launched in March 2025 stated if GAI was used on a draft, that should be disclosed to the courtroom if the ultimate human-reviewed transcript was used evidentially.
Police coverage requires six-monthly use audits of GAI.
RNZ has requested to see these audits for Whisper AI.
Late on Tuesday, Borrell stated police had not executed an audit of Whisper’s use.
“Although no formal audit has been undertaken because the trial and analysis final 12 months, TranScriptor [as police call the model] is managed underneath stringent utilization controls.
“The service is subject to ongoing oversight to ensure reliable operation, with system performance continuously monitored and every use is logged against the user for any future auditing purposes.”
Police paperwork emphasise having “a continuous evaluation cycle” and open communication with the general public concerning the use of AI.
The Justice Ministry stated it was seeking to use transcription expertise of what went on in courtroom.
“As the New Zealand Police is [sic] also progressing the use of Speech to Text for their transcription requirements, both agencies are in regular contact to share and learn from their respective approaches,” a spokesperson stated.
It has a young out.
Sign up for Ngā Pitopito Kōrero, a each day publication curated by our editors and delivered straight to your inbox each weekday.