New Machine Learning Framework Uses EHR Data to Assess ICI Effectiveness, Toxicity

Drs. Shaalan Beg and Travis Osterman discuss a machine learning model, recently featured in JCO Clinical Cancer Informatics, that uses electronic health record data to accurately predict the effectiveness and toxicity of treatment with immune checkpoint inhibitors. The new AI model can be used to provide a personalized risk-benefit profile, inform therapeutic decision-making, and improve clinical trial cohort selection.   TRANSCRIPT Dr. Shaalan Beg: Hello, and welcome to the ASCO Daily News Podcast. I'm Dr. Shaalan Beg, your guest host for today. I am an adjunct associate professor at UT Southwestern's Simmons Comprehensive Cancer Center.  Cancer immunotherapy has transformed the treatment landscape by providing new and effective treatment options for many solid and hematologic malignancies. But while many patients experience a remarkable response to immune checkpoint inhibitors, other patients can suffer life-threatening immune checkpoint toxicities. Today, we will be discussing a machine learning solution that can assess a patient's immune checkpoint inhibitor risk-benefit profile based primarily on routinely collected structured electronic health record data. This novel AI solution was recently featured in JCO Clinical Cancer Informatics, and I am delighted to welcome one of the report's authors, Dr. Travis Osterman. He is an associate vice president for research informatics and associate professor in the Department of Biomedical Informatics and the Division of Hematology Oncology at Vanderbilt University Medical Center in Nashville, Tennessee. Dr. Osterman also serves as the director of cancer clinical informatics at the Vanderbilt Ingram Cancer Center.  Our full disclosures are available in the transcript of this episode, and disclosures related to all episodes of the podcast are available at asco.org/DNpod.  Dr. Osterman, it's great to have you on the podcast today.   Dr. Travis Osterman: Thanks, Shaalan. It's great to be here. Thank you for the invitation.   Dr. Shaalan Beg: Congratulations on your recently published article in the JCO CCI titled "Prediction of Effectiveness and Toxicities of Immune Checkpoint Inhibitors Using Real World Patient Data." Why did you decide to address this specific problem?   Dr. Travis Osterman: I am a practicing medical oncologist at Vanderbilt, I specialize in thoracic malignancies. Immunotherapy has been a significant part of my practice from the beginning. And I think for all of us, we have patients in our practices that are tremendous responders. I have stories of my patients, a few of which, at least, are able to get years of benefit even after stopping therapy, and potentially some even stage 4 patients that are amazingly seemingly cured after their treatments. But I also have patients that experience severe toxicities, some of those are life-threatening or life ending, but many of those carry morbidity. In my population, I see a lot of pneumonitis, and that really alters patients' quality of life. And the biggest conversation I have with patients is: “How do I know which of these outcomes I’m going to have, if I’m going to get benefits from these therapies or am I going to get one of these side effects or toxicities?” And we set out to try to answer that question with data.   Dr. Shaalan Beg: When electronic medical records started to make their way into the clinic, I remember all of us thinking about the wonderful applications where we could use the data to help guide the clinical care, assign the right treatment for the right patient at the right time, and learn from other patients' experiences to improve the care of the person who’s in front of us. And my personal opinion is that we haven’t realized our electronic medical records’ potential to that extent. And efforts like the one you published in JCO CCI is the culmination of one of the efforts, and I can only imagine how much time and effort it must have taken to develop that and we’re hoping is the first of many more to come. For our listeners, can you talk us through the steps required to develop such a tool, and why now is the right time, and why we’re starting to see these evolve?   Dr. Travis Osterman: This project would not have been possible 20 years ago. It relies on having what we would call structured data available for our patients that are receiving cancer care, so that’s vital signs, laboratory values, and diagnoses, all of the things that we routinely collect in the electronic health record. So that is step 1. This project required that those systems be not just in place at academic centers but be widely available because our goal is to set up systems that will be able to transform cancer care, not just at academic institutions, but for the entire practice of oncology.  The second piece is you need enough data to be able to train these models. And so, we needed to be practicing with checkpoint inhibitors long enough to see patients that had toxicities, to see patients that had benefit, and then to jump into the data science of actually trying to learn from them. And so this really was the culmination of systems put in place by a lot of people before us and then really the right time [when] we started to have now enough data to really start to learn from.   Dr. Shaalan Beg: The publication discusses the steps of how you validated your tool. Talk me through how you see this being applied to the point of care for the next time you are about to start an immune checkpoint inhibitor for your lung cancer patient?   Dr. Travis Osterman: I think there are two different primary lanes that these types of models can be applied. In the drug development space, I think many of us are familiar that many assets, many drugs that are in the development pipeline are halted because of adverse events in toxicity profiles, but we also realize that not everyone gets those toxicities. And so we envision a future where before a drug that's in the drug development pipeline is taken out of the development pipeline, potentially, you could screen patients that are at lowest risks of actually having side effects from that immunotherapy and only screen those patients into the trial and that would potentially make more drugs available to more patients going forward. So I think that that's 1 lane.  I think the other lane in clinical practice is, let's say that I'm treating a patient who we determine has an increased risk for colitis. Instead of only seeing that patient back in 3 weeks, potentially, now, what if I had one of our nurse navigators, call the patient at weekly check-ins between visits to check in and see whether or not they were having any episodes of diarrhea and trying to intervene earlier. That might allow us to keep patients both out of the hospital, out of the emergency department to treat their symptoms more quickly to decrease the severity of their toxicity and keep them on treatments, especially if they're receiving benefit from it. So, I think there's an opportunity to improve both drug development and making more drugs available to patients and then also to identify patients that are at risk for toxicity, and then to do interventions to help mitigate those risks. Really, the idea of precision risk mitigation.   Dr. Shaalan Beg: One of the problems with electronic medical record-based tools in the past has been that they don’t evolve with time. We develop it, we set it, we deploy it, and it almost feels, to the users at least, that it stops evolving after that. With novel therapeutic agents coming into the clinic, we're seeing new ADCs, new novel checkpoint inhibitors entering the market. How do you envision tools such as yours to be refreshed so they can stay relevant with the modern armamentarium of medications which are being used?   Dr. Travis Osterman: So, if you ask any data scientist, the most requested item they will ask for is more data. And so, this initial set of models that we've described in this publication were trained exclusively on a single institution's data at Vanderbilt University Medical Center as we continue both to see more patients here, and then ideally look forward to collaborations with other centers. We expect that these models will continue to be refined and that the performance will improve as we increase the amount of training data, and we hope that that will do 2 things. One, it will counteract the kind of model drift that you described. But then two, it will allow us to ask some more specific questions that honestly, we weren't really powered to answer in our study here. For instance, we didn't look at cardiac toxicity, which is a concern if you're giving a CTLA-4 along with a PD-1 or PD-L1 inhibitor more so than single agent immunotherapy. We just don't have enough events to be able to train models on that. But with future collaborations, that would be a question we would love to tackle as well.  One of the things that's interesting about the implementation of these models is that we found many of the features that I would have expected to find as a practicing oncologist. For instance, when we're trying to predict the toxicity of pneumonitis inflammation of the lung, I as an oncologist would think that many of my patients that have COPD or interstitial lung disease at baseline seem to be at a higher risk. And so that's one of the features that I was looking to come out in the model. And that's exactly what we found. That was one of the contributing features that helps us predict a higher risk of pneumonitis. But what's interesting is that's certainly not the only feature; there end up being about a dozen features that are in that space that help predict that toxicity.   Similarly, for colitis, we found that the combination of receiving a CTLA-4 inhibitor in addition to a PD-1 or PD-L1 inhibitor, that combination together, which would increase risk for colitis, which is well-documented in our literature. So these models are not entirely black boxes. We've published the top features of these models that contribute to our predictions. And I think clinically the challenge for me has always been if I have a patient who has COPD, but it's pretty well-controlled and their O2 sat is normal, how does that patient's risk bring pneumonitis compared to someone who has poorly controlled COPD with low O2 sat at baseline, etc.? And so these models are really designed to help tease out some of those nuances.   Dr. Shaalan Beg: There are so many wonderful applications to use preexisting data that can improve the lives of our patients and frankly that can improve the work experience for clinicians. They can be used for risk stratification using these preexisting data. Can you talk a little bit about what are the barriers that people face or that your team faced in developing these tools, and what has changed or what's expected to change in the coming years to allow people to continue developing tools such as what was described?   Dr. Travis Osterman: I think it's important to realize that we are not unique in addressing this problem. This is a problem that I think has been a focal point of our cancer informatics community for the better part of the last, probably, decade. I think one of the things that distinguishes the work that we've done here is really this idea of clinical utility. And what I mean is we focused on data that would be collected at any routine oncology visit in the U.S., and I would argue worldwide, to use as features in our model. So, we're not running complex genetic testing that may or may not be paid for. We're not asking for new laboratory values to be sent or for extensive questionnaires that aren’t already in clinical practice. We're using pieces that are already being connected into the pipeline of oncology practices, and I think that's one of the differentiators of this project versus many others in this space.  Right now, these are only EHR data. We have a part of our project that's looking at imaging data and whether that adds value. But one of the pieces that I always advocate for, if we're going to ask practices for instance to upload these imaging files or to send a CD to a central location to improve the outcome, that's harder to work into an oncologist workflow than if all the data are already there in the health record and you can click a button and calculate this person's risk profile. And so, we've really tried to be pragmatic about our approach as we've entered this realm and that's been a real focus of our team.   Dr. Shaalan Beg: Many of the listeners of today’s podcast are busy clinicians, and you talked about how the idea for this project came from the problem you witnessed in your clinic. How can clinicians continue to be involved in such initiatives or drive these initiatives at their own institutions, in office situations where they may not have the resources that your team has? Can you speak to national efforts or collaborations in this regard?   Dr. Travis Osterman: Yeah. So, first of all, I would invite really anyone to reach out to our team, if they're in a position where they'll be interested in validating our models at their local institutions. We would be happy to work with them to provide the models to see how they perform on their data sets. I think that that's an important part of the academic review and informatics is to see how these models translate into other health care settings. And we also are interested to make sure that what I said in the prior discussion is correct, that we're only incorporating things that other institutions already have. So I think that that's certainly one.  The second is a part of a large National Cancer Data standard project called mCODE, the Minimal Common Oncology Data Elements, I chair that executive committee. And one of the pieces of that is trying to find a way to make all of these kinds of structured data interoperable between health records. And so I would just encourage all of my colleagues to always advocate for interoperability and, when there's an option, to store data in a way that makes that data more easily shared in the same formats between institutions. I think that that will pay many dividends for our field going forward. And I just want to plug all the team at mCODE for their work in this and maybe there'll be an integration and connection between mCODE and our project in the future.   Dr. Shaalan Beg: Thank you very much Dr. Osterman for sharing your insights with us today on the ASCO Daily News Podcast.   Dr. Travis Osterman: Thanks, Shaalan. Have a great day.   Dr. Shaalan Beg: And thank you to all our listeners for your time today. You'll find a link to Dr. Osterman’s article in the transcript of this episode. And if you value the insights that you hear on the ASCO Daily News Podcast, please take a moment to rate, review, and subscribe wherever you get your podcast.   Disclaimer: The purpose of this podcast is to educate and to inform. This is not a substitute for professional medical care and is not intended for use in the diagnosis or treatment of individual conditions. Guests on this podcast express their own opinions, experience, and conclusions. Guest statements on the podcast do not express the opinions of ASCO. The mention of any product, service, organization, activity or therapy should not be construed as an ASCO endorsement.   Find out more about today’s speakers: Dr. Shaalan Beg @ShaalanBeg Dr. Travis Osterman @TravisOsterman   Follow ASCO on social media:  @ASCO on Twitter  ASCO on Facebook  ASCO on LinkedIn    Disclosures: Dr. Shaalan Beg: Employment: Science 37 Consulting or Advisory Role: Ipsen, Array BioPharma, AstraZeneca/MedImmune, Cancer Commons, Legend Biotech, Foundation Medicine Research Funding (Inst.): Bristol-Myers Squibb, AstraZeneca/MedImmune, Merck Serono, Five Prime Therapeutics, MedImmune, Genentech, Immunesensor, Tolero Pharmaceuticals   Dr. Travis Osterman: Stock and Other Ownership Interests: Faculty Coaching Honoraria: Amazon Web Services Consulting or Advisory Role: eHealth, AstraZeneca, Outcomes Insights, Biodesix, MD Outlook, GenomOncology, Cota Healthcare, Flagship Biosciences, Microsoft, Dedham Group, Oncollege Research Funding: GE Healthcare, Microsoft, IBM Watson Health Travel, Accommodations, Expenses: GE Healthcare, Amazon Web Services