The CCOS Communications Team interviewed researchers at the 2023 CTSA Fall Program Annual Meeting poster session in November as part of a series to feature ongoing projects across the CTSA hubs. In this article, we’re featuring Dr. Barbara Tafuto, part of the Workforce Development Core at Rutgers Biomedical and Health Sciences, who presented the work of her and her colleague, Dr. Doreen W. Lechner on advancing pharmacovigilance through health system collaborations.
Research Question
At a basic level, machine learning (ML) requires external validation on unlearned samples to evaluate strength and estimate generalizability. This presents a major challenge for leveraging electronic health record (EHR) data from multiple health systems. Variability in data formats, quality, and completeness of data from different health systems can impede the harmonization and integration of data necessary for external validation of these algorithms. To overcome these challenges, Drs. Tafuto and Lechner developed a checklist to support generalizability for future federated EHR ML pharmacovigilance projects among disparate health systems. The checklist is based on the methods they developed to validate predictive models assessing adverse events for immunotherapy/checkpoint inhibitors arising out of multiple institutional and industry collaborations.
Research Plan
To create a generalizable methodology, the researchers searched recent literature to identify challenges or methods related to this process and culled a list of key themes and important questions to map out a checklist. They then recruited key medical informatics leaders within Rutgers Health for feedback.
This mixed methodological approach identified six general themes that were combined into 24 checklist-style questions. The six themes used in the checklist are as follows: (1) General applications for predictive modeling; (2) Data needs and identification for targeting a population; (3) Strategies for defining variables; (4) Optimizing model deployment; (5) Assessing variable measurement; and (6) Achieving generalizability. The goal of checklist development is to guide multidisciplinary teams to a more efficient, accurate, and generalizable predictive model, and avoid human use bias during model validation.
Next Steps
A checklist of this kind has the potential to improve team organization, build a more structured workflow, reduce workload, and increase generalizability of pharmacovigilance predictive modeling projects using EHR data.
Reflection with the Researcher: What real-world benefits do you envision might come from work like this, and how would patients or community members benefit?
Dr. Tafuto stated that “[m]ethods and processes are key components of Clinical and Translational Science. Our goal is to build a stronger model that can be generalized beyond our institution, and eventually contribute to better patient outcomes.”
