The CCOS Communications Team interviewed researchers at the 2023 CTSA Fall Program Annual Meeting poster session in November as part of a series to feature ongoing projects across the CTSA hubs. In this article, we’re featuring Dr. Adam Dziorny, NIDDK K23 Scholar in Pediatrics Critical Care and Informatics at the University of Rochester who presented his work on validating a machine learning model to predict acute kidney injury in children with critical illness.
Research Question
Machine learning has allowed for better prediction of adverse events in children with critical illnesses, including acute kidney injury (AKI). As most machine-learning models are developed and tested using data from a single-center, there is a need for multicenter datasets because single center datasets lack generalizability. Dr. Dziorny and his team aim to use existing multicenter datasets to address this issue.
Research Plan
Dr. Dziorny and his team created two novel, multicenter datasets to externally validate an already existing single-center-derived AKI prediction model. The team combined data from the Virtual Pediatric Systems (VPS) and PEDSnet for one dataset and extracted data from the PICU Data Collaborative for the other. The team first identified cases that fit into their inclusion criteria and set the baseline AKI score. Features for analysis were identified and included pre-admission cardiac arrest, pH, total bilirubin, platelet count, and blood urea nitrogen. Thresholds were set based on those used in the single-center dataset.
Dr. Dziorny and his team were successful in externally validating the existing AKI prediction model using the two derived multicenter datasets. Model discrimination was reported lower than in the original single-center model alone, which was partially attributed to differences in features within datasets.
Next Steps
The team concluded that external validation of a single-center derived model is possible with multicenter datasets, but attention must be paid to cohort selection, outcome definition and feature selection. Ongoing work includes re-calibration of the model, adding additional features for analysis, and generalizability testing.
Reflection with the Researcher: What do you want the CTSA community to know about the work you presented?
Dr. Dziorny wants to remind the CTSA community that “external validation is so important!” He hopes his model used for predicting acute kidney injury among critically ill children will soon be applied to additional patient settings." His work on the project continues, as the model is currently undergoing "silent validation," in which it will be implemented in real-time with electronic health records and run "in the background" to ensure validity.
