Study finds that AI-powered models’ predictions are limited to specific trials and cannot be generalized.

According to a recent study published in the journal Science, researchers have found that AI-powered prediction models were successful in making accurate predictions within the specific trial they were designed for. However, outside of this trial, the models provided random predictions, indicating that generalizations of these AI-based models across various study centers cannot be guaranteed at present. The study also highlighted that these models are highly dependent on the specific context in which they were developed.

Combining data from different experiments didn’t help matters either, the team found.

A team of researchers, including the universities of Cologne (Germany) and Yale (USA), tested the accuracy of AI-based models in predicting the response of schizophrenic patients to antipsychotics in several independent clinical trials.

The current research was related to the field of precision psychiatry, where data-related models are used in targeted treatments and medicines suitable for individuals.

“Our goal is to use new models from the field of artificial intelligence to target patients with mental health problems,” said Joseph Kambeitz, professor of biological psychiatry at the University of Cologne Medical Faculty and University Hospital Cologne. .

“Although numerous preliminary studies show the success of such AI models, the durability of these models has not yet been demonstrated,” said Kambeitz, adding that safety was of utmost importance in daily clinical use.

“We have strict quality requirements for clinical models and we also have to make sure that the models in different contexts give good predictions.

“The models should provide equally good predictions whether they are used in a US, German or Chilean hospital,” Kambeitz said.

That these AI models have very limited generalizability was an important signal for clinical practice and indicates that further research is needed to improve psychiatric care, the researchers said.

The team hopes to overcome these obstacles and is currently studying large cohorts of patients and data sets to improve the accuracy of the AI models, they said.