Characteristics of Biomarkers on Predictive Ability of Risk Models in Development and Validation Populations
Suman Kundu*
Department of Cardiovascular Medicine, Vanderbilt University Medical Center, USA
Submission: February 06, 2017; Published: February 27, 2017
*Corresponding author: Suman Kundu, Division of Cardiovascular Medicine, Vanderbilt University Medical Center, 2525 West End, Ste 300-A, Nashville, Tennessee 37203, USA, Tel: +1-615-875-9426; Fax: +1-615-322-3837; Email: suman.kundu@vanderbilt
How to cite this article: Suman K. Characteristics of Biomarkers on Predictive Ability of Risk Models in Development and Validation Populations. Biostat Biometrics Open Acc J. 2017;1(1): 555552. DOI: 10.19080/BBOAJ.2017.01.555552.
Prediction models are increasingly important for clinical decision making. For example, the predicted risk of an outcome may guide for a treatment initiation: high risks may lead to starting treatment, whereas for low risks treatments may be withheld or delayed. For several common complex diseases including different forms of cancer, diabetes, and cardiovascular disease, a number of prediction models have been developed in various populations. The predictive performance of these risk models is typically assessed by evaluating discrimination, which is the ability of the model to separate those with events from those without events. After developing a risk model, it is essential to investigate the model’s discriminative performance within an external population to judge about the generalizability and reliability of the risk model. A risk model that does not have an appreciable predictive ability in an external population may have limited opportunity to investigate further, aiming implementation in practice. It is often assumed that when a prediction model is validated in an external population, the discriminative ability expressed by the area under the receiver operating characteristic curve (AUC) decreases. However sometimes the AUC is able to increase as observed in number of validation studies . External validity may be assessed in other populations with different ethnic groups or different demographic. And, some predictor effects can be different for the validation population. However, the distributions of population characteristics (case mix) also differed between the development and validation populations. For example, the Framingham Heart Study risk score was developed in a US white, middle-class population and validated in Native Americans. White persons often have high blood pressure than the Native Americans. The external validity of a risk prediction model in new subjects is an important step for the model’s generalizability to other populations. Several studies have shown how the AUC is impacted by a different case-mix within the validation sample, and heterogeneity in the effect sizes of risk factors among development and validation samples. Simulation studies also investigated the impact of correlation among predictors on the AUC in the development sample and how correlation should be interpreted in the light of the other parameters such as the distributions of predictors among cases and controls, and different strengths of predictive effects that may alter the AUC in external populations. Simulation results showed that the AUC depends on the fitted regression coefficients, the case mix of the validation sample and the correlation among the predictors in the development and external validation sample. Lower discriminative ability in the validation sample as compared with the development sample can be the result of a more homogeneous validation case mix. Increasingly negative correlations improve the AUC. Interestingly and maybe not expected, the AUC in external populations can be either lower or higher as compared to the AUC estimated in the derivation cohort. Since AUC is impacted by the differences in population characteristics and in predictor effects between the derivation and validation populations, these need to consider for proper interpretation for difference of AUC in validation populations. This suggests, although the AUC estimated in the derivation sample is not promising, the same risk model can have a promising AUC at external validation. Conversely, even though the AUC in both derivation and a particular validation dataset is high, the same risk model can perform poorly in another external population. Since there may be heterogeneity in population characteristics, the predictive ability of risk models should be assessed in different external populations in order to ascertain generalizability. Opinion
References
- Collins GS, de Groot JA, Dutton S, Omar O, Shanyinde M, Tajar A, et al. (2014) External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC med res methodol 14: 40.
- Vergouwe Y, Moons KG, Steyerberg EW (2010) External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol172(8): 971-980.
- Debray TP, Vergouwe Y, Koffijberg H, Nieboer D, Steyerberg EW, et al. (2015) A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol 68(3): 279-289.
- Demler OV, Pencina MJ, D’Agostino RB Sr (2013) Impact of correlation on predictive ability of biomarkers. Stat med 32(24): 4196-4210.
- Riley RD, Ensor J, Snell KI, Debray TP, Altman DG, et al. (2016) External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ 353: i3140.