2023 Fall CTSA Program Annual Meeting

Introduction

Machine learning predictive modeling requires external validation on an unlearned sample to evaluate strength and estimate generalizability. Challenges in leveraging EHR data for artificial intelligence and machine learning lie in harmonizing data from multiple health systems to complete algorithm validation needs. Once disparate health systems agree to collaborate, variability in data formats, quality, and completeness can impede the integration necessary for external validation. Differences in clinical practices and patient demographics among health systems can also introduce bias into the validation process. Such disparities require detailed project management, vigilant communications with the collaborating institutions, and appropriate statistical techniques to ensure that predictive models are robust and generalizable.
Rutgers University in partnership with the Drug Information Association and Georgetown University developed a collaboration to build and test a predictive model for signal detection of adverse events for immunotherapy /checkpoint inhibitors. This abstract will illustrate the methods and processes of that collaboration to support its generalizability for future federated EHR, AI/ML, pharmacovigilance projects among disparate health systems.

Methods

In order to generalize a methodology that would best serve to externally validate a predictive model for adverse events using EHR by a collaborating institution, we began with a search of recent literature to identify challenges, checklists, or methods relating to this process. Once relevant articles were identified, key themes and important questions were mapped to outline a checklist.

Results

The search process identified 2 key articles that identified 6 general themes: (1) General applications for predictive modeling; (2) Data needs and identification for targeting a population; (3) Strategies for defining variables; (4) Optimizing model deployment; (5) Assessing variable measurement; and (6) Achieving generalizability. 24 checklist-style questions were developed to guide multidisciplinary teams to a more efficient, accurate, and generalizable predictive model.

Conclusion

The purpose of this checklist is to assure predictive model alignment and avoid human use bias during model validation. A checklist for collaboration among health systems in predictive modeling for adverse events detection has the potential to improve multidisciplinary team organization, build a more structured research workflow, reduce pharmacovigilance and technology unit workload, and increase the generalizability of pharmacovigilance predictive modeling projects using EHR data.

Objective/Goals:
The global threat of antimicrobial resistance (AMR) varies regionally. This study explores whether geospatial analysis and data visualization methods detect both clinically and statistically significant variations in antibiotic susceptibility rates at a neighborhood level.

Methods:
Patient level antibiotic susceptibility data was collected from three regionally distinct Wisconsin health systems (UW Health, Fort HealthCare, Marshfield Clinic Health System [MCHS]). We included the initial Escherichia coli isolate per patient per year per sample source with a patient address in Wisconsin (N?=?100,176). Isolates from U.S. Census Block Groups with less than 30 isolates were excluded (n?=?13,709), resulting in 86,467 E. coli isolates. The primary study outcomes was the detection of statistically significant local hot (high susceptibility) and cold spots (low susceptibility) for variations in antibiotic susceptibility by U.S. Census Block Group. UW Health isolates collected represented greater isolate geographic density (n?=?36,279 E. coli, 389?=?blocks, 2009–2018), compared to Fort HealthCare (n?=?5110 isolates, 48?=?blocks, 2012–2018) and MCHS (45,078 isolates, 480 blocks, 2009–2018). Percent E.coli susceptibility to each antibiotic was calculated by block group. Choropleth maps enabled a spatial AMR data visualization. At the local level, we identified hot and cold spots at all three health systems (90%, 95%, and 99% CIs). This analysis identified blocks with statistically significant higher and lower susceptibility. AMR spatial clustering was observed in urban areas but not rural areas.

Significance of Impact:
We provide a novel method for analysis of antimicrobial susceptibility data geographically. Unique identification of AMR hot spots at the neighborhood level provides a foundation for future analyses and hypotheses. Clinically meaningful differences in AMR could inform clinical decision support tools and translate this data to practice.

Background: Despite immunization, patients on antineoplastic and immunomodulating agents have a heightened risk of COVID-19 infection. However, accurately attributing this risk to specific medications remains challenging.

Introduction
Septic shock is characterized by the need for vasopressors – typically starting with norepinephrine – to maintain blood pressure. Despite ubiquitous use of norepinephrine during septic shock, norepinephrine-to-mean arterial pressure (MAP) dose-response relationships are uncharacterized in critically ill patients. In this study, we sought to determine the risk-adjusted change in MAP response across norepinephrine doses in patients with septic shock.

Methods
We used Medical Information Mart for Intensive Care-IV (MIMIC-IV) database (2008-2019) to identify patients for study inclusion. Included patients were adults with a diagnosis of sepsis who were started on norepinephrine alone at a dose between 0.01 and 0.5 mcg/kg/min. We then identified the first episode when norepinephrine alone was increased by at least 0.005 mcg/kg/min, the starting norepinephrine dose before the dose increase (exposure of interest), and the norepinephrine dose change (new norepinephrine dose minus prior norepinephrine dose). We defined MAP change (outcome) as the maximum MAP in the 120 minutes after norepinephrine dose increase minus the minimum MAP in the 120 minutes before dose increase. Using ANCOVA, we constructed linear models for MAP change including terms for the starting norepinephrine dose, norepinephrine dose change, minimum MAP before dose increase, comorbidities, and multiple measures of critical illness severity. From the ANCOVA models, we calculated the marginal mean MAP change across starting norepinephrine doses while holding variables at their sample means.

Results
Among 34,789 adult patients with sepsis, we identified 5,569 patients started on norepinephrine alone, of whom 2,903 had a norepinephrine dose increase. The mean minimum MAP prior to dose increase was 63 (standard deviation [SD] 11) mm Hg and mean maximum MAP afterwards was 72 (SD 12) mmHg. Nadir MAP measurements occurred ~15 minutes prior to norepinephrine dose increases and returned to pre-dose-increase baselines after ~60 minutes. The mean starting norepinephrine dose was 0.08 (SD 0.07) mcg/kg/min and the mean norepinephrine dose change was 0.04 (SD 0.05) mcg/kg/min. The predicted mean MAP change from a 0.04 mcg/min/kg dose change was 8.7 mm Hg (95% confidence interval [CI] 6.7, 10.7) at a starting norepinephrine dose of 0.1 mcg/kg/min; 8.5 (95% CI 6.3, 10.7) at 0.2 mcg/kg/min; 7.1 (95% CI 4.4, 9.7) mmHg at 0.3 mcg/kg/min, 3.5 (95% CI -1.4, 8.5) at 0.4 mcg/kg/min, and -0.5 (95% CI -9.7, 8.7) at 0.49 mcg/kg/min.

Discussion
Using time-varying electronic medical record data, we show evidence for a smaller increase in MAP in response to a norepinephrine increase for patients with septic shock receiving norepinephrine doses above 0.2 mcg/kg/min. This potential threshold for a change in norepinephrine effectiveness helps to inform septic shock clinical practice and marks a rational target for future studies of adjunctive vasoactive treatments for septic shock.

Cancer ranked second in U.S. mortality in 2020, following heart disease. Breast cancer is the most common cancer among women (30% of cases), while prostate cancer leads among men (20% of cases). African Americans face elevated risks, with black women and men being 1.5 and 2.1 times more likely to die from breast and prostate cancer, respectively, compared to whites. This disparity is most pronounced in the Southeast, home to a significant African American population. Advances in radiotherapy (RT) through Stereotactic Body Radiotherapy (SBRT) offers the delivery of high radiation doses in fewer sessions, boosting convenience, quality of life and cost-effectiveness. The goal of this study is to validate the prospective use of AI-assisted radiotherapy treatment planning at a community cancer center to demonstrate the potential in reducing inequities in cancer care access in rural Alabama. We hypothesize that clinically deploying AI software to automate SBRT planning 1) is feasible and 2) will reduce required radiotherapy planning times by 75%.

Brain-based biomarker of treatment recovery for deep brain stimulation for treatment-resistant depression

Sankaraleengam Alagapan1, Stephen Heisig2, Ki Sueng Choi2, Allison C. Waters2, Ashan Veerakumar3, Vineet Tiruvadi1,3, Mosadoluwa Obatusin2, Tanya Nauvel2, Jungho Cha2, Andrea Crowell3, Martijn Figee2, Patricio Riva-Posse3, Robert Butera1, Helen Mayberg2, Christopher Rozell1

Georgia Institute of Technology1, Icahn School of Medicine at Mount Sinai2, Emory University3

Deep brain stimulation (DBS) of the subcallosal cingulate cortex (SCC) has shown effectiveness in treating patients with treatment-resistant depression (TRD). However, inter-individual variability in recovery and intra-individual variability in hard-to-measure symptoms hinder clinical decision-making, including adjustment of stimulation parameters. To overcome this obstacle, which also limits the scalability, we aim to identify electrophysiological markers of recovery from local field potential (LFP) recordings.

10 participants with TRD were recruited for this study. LFPs were acquired weekly from 6 participants during 24 weeks of SCC DBS therapy using implanted pulse generators (Activa PC+S, Medtronic Inc, USA). 4 participants were excluded for missing LFP data and LFP data contaminated with artifacts. Spectral features were extracted from the LFPs, and a neural network classifier was used to differentiate the beginning and the end of the 24 weeks. A novel explainable artificial intelligence (xAI) tool was used to identify a biomarker that explained the feature changes that contributed to classifier performance. Depression severity was measured using the Hamilton Depression Rating Scale (HDRS). Instead of tracking weekly HDRS scores, the HDRS was binarized into ‘sick’ and ‘stable response’ states and compared to the states derived from the biomarker.

The neural network classifier was able to differentiate the beginning and the end of 24 weeks (Area under ROC curve: 0.87 ± 0.09) from the LFP features. The features that contributed to classifier performance were identified to be alpha (8 – 12 Hz), beta (12 – 30 Hz) and gamma (30 – 40 Hz) bands. The binarized states (‘sick’ and ‘stable response’) derived from the biomarker predicted HDRS-derived states with high accuracy (Area under ROC curve: 0.94 ± 0.04). In addition, the biomarker predicted relapse in a participant whose LFP data was not used for training the different machine learning models. Notably, the biomarker exhibited changes in the week after the stimulation dose was adjusted.

The results indicate that the biomarker derived from LFP satisfies two essential requirements - tracking relevant changes in disease state and responding to intervention. Thus, the biomarker can potentially be utilized to inform clinical decision-making thereby improving the scalability of this promising approach beyond a few highly specialized centers and reducing variability in outcomes.

The management of cancers of the head and neck is complicated and allows for a variety of disparate approaches from providers. However, data from several hundred or thousands of patients is necessary to decipher the optimal decisions for populations of patients. And prospective trials would take years to accrue and often are financially not possible. Instead, we collated the entire electronic medical record for all patients at our institution treated for head and neck cancers. We employed a variety of clinical informatics and natural language processing to gather text data into large data frames. We found key conclusions in the diagnostic, treatment, and surveillance of our patients. First, obtaining post-operative PET/CT changes in management in over one-third of patients, highlighting the utility of optimizing diagnostic imaging. Second, using a newer silicone-based cream instead of just a moisturizer decreased the absolute risk of grade 2+ radiation dermatitis by almost 15%. Lastly, we are deploying novel autosegmentation frameworks to better understand tissue decomposition in the head and neck to identify patients in need of further nutritional support while undergoing radiation therapy. Collectively, we showcase the value and opportunity of mining oncological data for the improvement of patient care.

Context: Climate change is causing prolonged periods of extreme heat. The incidence of heat-related illness is rising in an inequitably distributed manner, with some people and communities at greater health risk than others. The corresponding increase in healthcare utilization could lead to significant cost increases. Objective: To estimate the excess number of heat related emergency room (ER) visits and hospital admissions (AD) caused by heat days in Virginia, estimate the additional monetary costs for Virginia, extrapolate those costs nationally, and identify the communities with higher rates adverse events. Study Design: Retrospective observational analysis and ZCTA code evaluation of Virginia with national extrapolation. Dataset: We used weather station data from 15 sites in or near Virginia and 298 sites nationally to identify heat event days, Virginia All-Payer Claims Data to capture healthcare utilization, American Community Survey to measure community characteristics, and the Healthcare Cost & Utilization Project to estimate cost of care. Analysis: Between 2016 and 2020, we calculated the per capita rate of ER visits and ADs for heat-related and heat-adjacent illness. From daily climate and census data, we determined the number of heat event days experienced by each ZCTA and the size of the exposed population. We compared health care utilization by ZCTA on heat event days to non-heat event days, and applied Virginia’s health event rates to national measures of heat events to extrapolate a national cost. Results: There were an average of 80 heat-event days per year in Virginia between 2016 and 2020. During heat event days, ER visits for heat related and adjacent illness increased 179% and ADs increased 217%, for an additional 6967 ER visits and 1667 AD annually. Extrapolating Virginia’s findings nationally and assuming average ER visit and AD costs of $757 and $14,900, respectively, heat event days resulted nationwide in an average of $1.03 billion in costs every summer. Rates were higher in more vulnerable communities, and risk factors contributing to that rate were identified. Conclusions: These findings highlight the stark impact of climate change on health and demonstrate the value of integrating health, community, and environmental data. Action is needed to slow rising temperatures, raise public awareness about the risks of extreme heat, increase community resilience, and strengthen health care services.

Objective: To incorporate artificial intelligence (AI) in patient-centered precision medicine, machine learning (ML) algorithms must be constructed with high-quality training data representing the target population of interest. A “computable phenotype” can serve as a foundation for ML training and validation. A Clinical Translational Science colloquium student assignment led to the development of a disease-agnostic framework for computable phenotype construction. This Ph.D. nursing student, now a TL1 trainee, continues to develop the framework as a foundation for ML algorithm construction. Male puberty demonstrates utility of the framework.
Methods/Study Population: A computable phenotype of male puberty was constructed in pursuit of answering the following question: “Does early pubertal timing increase the risk of developing type II diabetes (T2D) in boys?” A computable phenotype of males < 18 years old was created in the TriNetX© Diamond Network utilizing Boolean operator data queries. TriNetX© contains patient electronic health record information (ICD-10 diagnoses, anthropometric measures). An exploratory analysis of patient counts reflecting various computable phenotypes allowed for outcome (T2D) comparison of boys diagnosed with precocious puberty (E30.1, ICD code for early pubertal timing) to those without, while controlling for body mass index (BMI).
Results: Subjects (N=12,996,132) displayed the following computable phenotype: Male, < 18 years old, without ever having a BMI documented >85th percentile. Boys diagnosed with precocious puberty (E30.1) were 6.89 times more likely to develop T2D when aged 14-18 years old than those without (OR 6.89, 95% CI: 5.17-9.19, p<0.0001), validating a need for further exploration of the mechanisms underlying this disease pathway. Next steps will involve training a ML model on each computable phenotype groupings’ health records to identify underlying salient pathophysiologic variables.
Discussion/Significance: Between group comparisons of the computed phenotypes in the case example reveal that boys with precocious puberty may be at increased risk for future T2D development, validating the need for further exploring this underlying pathophysiology. The computable phenotypes constructed in this study will now be used to further a TL1 trainee’s research experience through the construction of ML algorithms which describe pathways linking precocious puberty to future T2D development. The generalized computable phenotype approach extracted from this example can be used by multi-disciplinary teams to: 1) answer important exploratory clinical questions in large databases like TriNetX©, and 2) serve as the foundation for future AI algorithmic construction to model disease development.

Authors: Bryan Bunning, Manisha Desai, & Jonathan H Chen

Title: Descriptive Analysis of a Clinical Trial Data Capture Workflow

Abstract: Modern clinical trials involve manual transcription of patient data from Electronic Health Records to Electronic Data Capture (EDC) software, limiting the use of medical knowledge and informatics tools, hindering research speed, safety, and increasing costs.

Despite these issues, there is little work describing data entry workflow in the context of a clinical trials’ design. In this work, we characterize the data workflow of a phase 2 outpatient trial of COVID-19 that occurred at Stanford University using a common but underutilized data type - Audit Logs. Electronic data capture (EDC) Audit logs can provide insight into the temporal nature of data entry in relation to the clinical trials’ design and conduct.

A single trial’s audit log including 185788 Date-User-Action triplets was aligned to the trial’s schedule of events. Approximately 1/3rd of data elements, on average, were entered retroactively for a patient after their last scheduled visit in the trial. Approximately half of the entire dataset was transcribed by 10 staff members. 7/10 of these staff members held an advanced degree such as an RN or an NP. Interestingly, nurses, not clinical coordinators, were the staff who manually entered the largest proportion of data in this trial. Qualitative interviews with clinicians and staff who worked on the trial (n=10) elucidated anecdotes on bottlenecks and workflow tendencies that were supported by the audit log data. Overall, this study highlights the promise of translational science methods to analyze audit log data to gain insight into questions concerning workflow and bottlenecks within a clinical trial, and the burden of data entry in clinical research.

Aggregating multiple audit logs and summarizing data entry across multiple trials could give insight to systemic bottlenecks of clinical research. Incorporating user information from the EHR could give a more holistic understanding of where staff effort is dedicated across multiple software systems. This work is useful when assessing costs and benefits of informatics systems such as “e-source” or other clinical research software tools.