Predictive models have been used in healthcare for many years to determine the risk of patients developing certain conditions and to aid decision-making. Developing AI predictive models for clinical use is complex, and even if they perform well, it doesn’t guarantee they will be used effectively. This study outlines nine stages for developing and evaluating AI predictive models, highlighting challenges clinicians may face and offering practical tips. These stages include clarifying clinical questions, selecting predictors, choosing datasets, developing and validating models, presenting predictions, licensing, maintaining the models, and evaluating their impact. Introducing AI models in clinical practice involves various components like the accuracy of predictions, understanding and use by doctors and patients, and the effectiveness of actions taken based on predictions. Evaluating these models in real-world settings is crucial to understand their true impact.

Abstract

Background Predictive models have been used in clinical care for decades. They can determine the risk of a patient developing a particular condition or complication and inform the shared decision-making process. Developing artificial intelligence (AI) predictive models for use in clinical practice is challenging; even if they have good predictive performance, this does not guarantee that they will be used or enhance decision-making. We describe nine stages of developing and evaluating a predictive AI model, recognising the challenges that clinicians might face at each stage and providing practical tips to help manage them.

Findings The nine stages included clarifying the clinical question or outcome(s) of interest (output), identifying appropriate predictors (features selection), choosing relevant datasets, developing the AI predictive model, validating and testing the developed model, presenting and interpreting the model prediction(s), licensing and maintaining the AI predictive model and evaluating the impact of the AI predictive model. The introduction of an AI prediction model into clinical practice usually consists of multiple interacting components, including the accuracy of the model predictions, physician and patient understanding and use of these probabilities, expected effectiveness of subsequent actions or interventions and adherence to these. Much of the difference in whether benefits are realised relates to whether the predictions are given to clinicians in a timely way that enables them to take an appropriate action.

Conclusion 

The downstream effects on processes and outcomes of AI prediction models vary widely, and it is essential to evaluate the use in clinical practice using an appropriate study design.