Poor performance of clinical prediction models: the harm of commonly applied methods

Publication date: Available online 24 November 2017
Source:Journal of Clinical Epidemiology
Author(s): Ewout W. Steyerberg, Hajime Uno, John P.A. Ioannidis, Ben van Calster, Chinedu Ukaegbu, Tara Dhingra, Sapna Syngal, Fay Kastrinos
ObjectiveTo evaluate limitations of common statistical modeling approaches in deriving clinical prediction models and explore alternative strategies.Study Design and SettingA previously published model predicted the likelihood of having a mutation at the time of diagnosis of colorectal cancer. This model was based on a cohort where 38 mutations were found among 870 participants, with validation in an independent cohort with 35 mutations. The modeling strategy included stepwise selection of predictors from a pool of 37 candidate predictors and dichotomization of continuous predictors. We simulated this strategy in small subsets of a large contemporary cohort (2,051 mutations among 19,866 participants) and made comparisons to other modeling approaches. All models were evaluated according to discriminative ability (concordance index, c) in independent data.ResultsWe found over 50% bias for 5 out of 6 originally selected predictors, unstable model specification, and poor performance at validation (median c=0.74). A small validation sample hampered stable assessment of performance. Model pre-specification based on external knowledge and using continuous predictors led to better performance (c=0.836 and c=0.852 with 38 and 2,051 events respectively).ConclusionPrediction models perform poorly if based on small numbers and developed with common but suboptimal statistical approaches. Alternative modeling strategies to best exploit available predictive information need wider implementation, with collaborative research to increase sample sizes.



from ! ORL Sfakianakis via paythelady.61 on Inoreader http://ift.tt/2A76lRN
via IFTTT