AI-powered transcription tool used in hospitals invents things no one ever said

Public

On October 26, 2024, Associated Press published a story detailing findings of researchers that the OpenAI transcription tool Whisper, used by clinics and hospitals, is prone to making up chunks of text or even entire sentences according to interviews with more than a dozen software engineers, developers and academic researchers.

Experts said that such fabrications are problematic because Whisper is being used in a slew of industries worldwide to translate and transcribe interviews, generate text in popular consumer technologies and create subtitles for videos.

More concerning, they said, is a rush by medical centers to utilize Whisper-based tools to transcribe patients’ consultations with doctors, despite OpenAI’ s warnings that the tool should not be used in “high-risk domains.”

The prevalence of such hallucinations has led experts, advocates and former OpenAI employees to call for the (US) federal government to consider AI regulations.

... Over 30,000 clinicians and 40 health systems, including the Mankato Clinic in Minnesota and Children’s Hospital Los Angeles, have started using a Whisper-based tool built by Nabla, which has offices in France and the U.S.

... It’s impossible to compare Nabla’s AI-generated transcript to the original recording because Nabla’s tool erases the original audio for “data safety reasons,” Raison said.

Nabla said the tool has been used to transcribe an estimated 7 million medical visits.

Saunders, the former OpenAI engineer, said erasing the original audio could be worrisome if transcripts aren’t double checked or clinicians can’t access the recording to verify they are correct.

... A California state lawmaker, Rebecca Bauer-Kahan, said she took one of her children to the doctor earlier this year, and refused to sign a form the health network provided that sought her permission to share the consultation audio with vendors that included Microsoft Azure, the cloud computing system run by OpenAI’s largest investor. Bauer-Kahan didn’t want such intimate medical conversations being shared with tech companies, she said.

“The release was very specific that for-profit companies would have the right to have this,” said Bauer-Kahan, a Democrat who represents part of the San Francisco suburbs in the state Assembly. “I was like ‘absolutely not.’ ”

Web links:

https://apnews.com/article/ai-artificial-intelligence-health-business-90...