Opportunities to use EHR-derived RWE to inform HTA decision-making

June 21, 2021
 

generated from an EHR has been shown to be more relevant, timely, and representative for health technology appraisals (HTA) compared to evidence from clinical trials. In this session, use cases demonstrating the value of using RWE in the HTA setting were showcased as well as a discussion on the role of RWE with NICE.

Download the slides

Transcript

Danielle Bargo:

First just let me say, thank you to everyone for joining us today as part of this conversation on how real-world evidence derived from electronic health records can inform HTA decision-making. My name is Danielle Bargo and I'm the Head of HTA program development at Flatiron Health. I'm very excited about this webinar today and even more excited to introduce our speakers.

Our session today includes Dr. Blythe Adamson, Principal Quantitative Scientists from Flatiron Health, Dr. Páll Jónsson, Program Director - Data from the National Institute for Health and Care Excellence, Dr. Scott Ramsey, Director at the Hutchinson Institute for Cancer Outcomes Research at the Fred Hutch Cancer Center and Akshay Swaminathan, Senior Quantitative analysts from Flatiron Health. You will hear from all of them very shortly.

Danielle Bargo:

Some quick housekeeping items before we dive in. First, I would like to draw your attention to the Q&A option available throughout the webinar. If you hover over your screen with your mouse, you will see an option for Q&A towards the bottom middle of your screen with the black bar. At any point during today's presentation, feel free to submit a question through this feature. While we do have a packed agenda, we will have time at the end to ask your questions to the speakers using this function. Please feel free to reach out to us after this webinar if you'd like to discuss any of the content that we present today in more detail. Also, if you have any technical questions or issues, please let us know via the Q&A tool and we will do our best to assist. Lastly, before we jump in, please excuse any interruptions from our pets or loved ones as like most of you, we are all working from home and we are joining you from various corners of the world, from Manchester to Seattle.

Today we are going to explore the opportunities to use real-world evidence for HTA decision-making. We will start with a presentation by Akshay Swaminathan, on how EHR-derived can be used to inform cost-effectiveness analysis. We will then hear from Dr. Páll Jónsson on how real-world data on US patients can be used in an ex-US setting for HTA decision making. Following these presentations, we will be joined by Dr. Blythe Adamson and Dr. Scott Ramsey to discuss these topics further, as well as answer questions from the audience. Before we get started, we would like to learn a bit about our audience and how you currently use RWE for HTA decision-making. You should see a poll pop up on your screen. If you have two monitors, please check both monitors as it may appear on your other screen. Other attendees will not be able to see the responses that you choose. Our first poll question today is, how frequently do you include RWE in your Global Value Dossiers for new oncology medicines? 0 to 20%, 21 to 40%, 41 to 60%, 61 to 80% or 81 to 100%? We'll give everyone some time and then we will share the results. Okay. Let's go ahead and end the poll and share the results. It looks like there's a really good variety attending today. Around 34% say that they use 0 to 20%, then 20% say that they use it 41 to 60% and 20% say they use it 81 to 100%. It's a really good variety and I think today we can dig a bit deeper into the reasons why people use it for HTA submissions.

With that, I'd like to introduce Akshay to present his research on using EHR data for cost-effectiveness analysis. Akshay, the floor is yours.

Akshay Swaminathan:

Thanks, Danielle. Hi everyone. My name is Akshay Swaminathan. I'm a data scientist and researcher specializing in real-world clinical and genomic data. Very excited to share the results of this proof of concept project demonstrating how real-world data can be used to enhance traditional approaches to cost-effectiveness analysis.

When evaluating new therapies, initial health technology assessments typically rely on data from clinical trials, but as these therapies become more widely used in clinical practice, new evidence in the form of real-world data emerges and this evidence can be used to supplement findings from initial HTAs. Now real-world data has several advantages compared to data from clinical trials. It's often more relevant because it's coming directly from routine clinical practice. It's often more representative of the broader patient population and real-world data also often has larger sample sizes and longer follow-up times compared to clinical trial data. Given these advantages, we were interested to see if we use real-world data instead of clinical trial data for cost-effectiveness analysis, how would this change the results?

We replicated a cost effectiveness analysis of non-small cell lung cancer immunotherapies initially developed by the Institute for Clinical and Economic Review and instead of using network meta analysis of clinical trials to derive hazard ratio and survival times, we'll call this the traditional approach. Instead, we derive these quantities using a real-world data cohort of patients with non-small cell lung cancer. How did we use EHR data to select a cohort of patients taking these therapies of interest?

In the next slide you'll see how we started with patients in Flatiron's non-small cell lung cancer dataset and we applied selection criteria to arrive at cohorts of patients who are eligible to take the therapies of interest according to their drug label indication. We arrived at three immunotherapy cohorts, atezolizumab, pembrolizumab and nivolumab and the chemotherapy docetaxel cohort, which was the comparator, and I want to point out here that we were able to use biomarker data captured in the EHR to apply criteria such as selecting for patients with no EGFR mutations and patients who were PD-L1 positive. After selecting these patients, we were interested to see how different are these real-world patients compared to the patients in the original clinical trials.

On this next slide you'll see where we compare the demographic and clinical characteristics between patients in our real-world cohort, shown in the blue bars, with the patients enrolled in the original clinical trials shown in the green bars. The trials we're referring to here are POPLAR for atezolizumab, CheckMate 017 for nivolumab and KEYNOTE-010 for pembrolizuamb. Now, in terms of demographics, we saw that the real-world cohorts were broadly more representative of the overall non-small cell lung cancer patient population, with over 45% female patients across the board and over 20% non-white patients across the board. In terms of followup time, the real-world cohorts had anywhere from five to 10 months greater follow-up time compared to the clinical trials and follow-up time we're defining as time from diagnosis to last clinical activity date. Sample size varied. In the atezolizumab and pembrolizumab cohorts, the original trial populations had greater sample size, but in the nivolumab cohort, the real-world cohort had over 1500 more patients than the original clinical trial.

Now that we understand the makeup of this real-world cohort, how can we use this cohort to conduct a cost effectiveness analysis? This figure on this slide shows the results of probabilistic sensitivity analysis, comparing each of the three immunotherapy cohorts to the comparator chemotherapy cohort, docetaxel. The Xaxis on this figure represents quality adjusted for life years, the Y axis cost in US dollars, and each point represents a simulated ICER, Incremental Cost Effectiveness Ratio. The green points show the simulated ICERs from the traditional cost effectiveness approach and the blue points represent the ICERs from the real-world enhanced cost-effectiveness approach. The table below shows the ICER point estimates with 95% credible intervals. What's immediately evident is that the spread of ICERs from the traditional approach is much larger than the spread of ICERs from the real-world enhanced approach. We actually saw that the 95% credible intervals were shrunk by 37% for atezolizumab, 69% for nivolumab and 83% for pembrolizumab, respectively.

Now, what are the driving factors behind this drastic decrease in uncertainty? The main driver is that in the traditional cost effectiveness analysis approach, the hazard ratios, which were estimated using network meta analysis, had very large confidence intervals and that's what's driving the large spread in simulated ICERs from the traditional approach. Now other contributing factors are that in the real-world cohorts, we saw larger sample size and not just in the nivolumab cohort, but in our chemotherapy cohort docetaxel, which had over 1300 patients. Another contributing factor is the longer followup time in the real-world cohorts, which led to lower rates of censoring.

What can we take away from these results? The point of this analysis is not to propose new price points for the three immunotherapy drugs. That wasn't the purpose of this analysis. Rather, the point is to show how real-world data can be used to select a cohort of patients who are taking therapies of interest, how real-world data can be used to estimate hazard ratios and survival times that can then be inputted into cost-effectiveness models. Also, just show that the result in cost-effectiveness estimates may actually have greater certainty than traditional approaches. Now, I want to point out some limitations of this analysis, as we saw the sample size in the three immunotherapy courts varied substantially and this highlights the fact that this real-world enhanced approach to cost-effectiveness analysis may be best suited for therapies with high uptake in real-world populations. I also want to point out that we did not implement certain clinical trial criteria that involved other variables like baseline ECOG status or sites of metastasis, nor did we implement population adjustment methods, such as matching. Nevertheless, we're very excited about the potential of this new approach using real-world data for enhanced cost effectiveness analysis to inform HTA decision-making. Thank you.

Danielle Bargo:

Thank you very much Akshay for that very insightful presentation on enhanced cost-effectiveness analysis. We will save Q&A for later in the webinar, but do feel free to go ahead and start asking your questions using the Q&A function. We will now have another poll for the audience and the question is which real-world variables do you consider most important for HTA decision making? Overall survival, progression-free survival, clinical response, biomarkers, ECOG or adverse events? For this one, you can check all that apply.

We can go ahead and end the poll and share the results. Let's see what people are thinking. Okay. This one has a bit of a clearer takeaway point than our first question and it looks like 84% of attendees say that overall survival is the most important variable for HTA decision-making. Looks like progression-free survival is a close second with about 58% of people saying that's an important variable. What's really interesting is what's coming in third is adverse events with 44% of people saying that's an important variable. Very interesting insights and I think that ties very nicely with the following presentation.

The next presentation is from Dr. Páll Jónsson at the National Institute for Health and Care Excellence. This presentation will focus on results from our first research project that we are conducting as part of our three-year research collaboration between Flatiron Health and NICE. The objective of our research collaboration is to explore the value of real-world evidence derived from electronic health records to inform the clinical and cost effectiveness of health technologies. Páll, the floor is yours.

Páll Jónsson:

Thank you, Danielle. It's great to be here today. Perhaps I should start with a little bit of context to this. NICE has recently published a new five-year strategy and within this strategy, there's a clear signal that NICE wants to make use of broader types of data to reduce the gaps in our knowledge and essentially being able to drive forward access to innovation for patients. We all know that the regulators, the payers, the HTAs, have traditionally been very reliant on randomized controlled trials and of course that's for a very good reason, but I feel it's absolutely important now to understand how we can make...

the most of the rich healthcare data that's being generated in practice. And we really need to understand how these data can help inform our decision making. So this is the context in which we're in now. So NICE sets out to be active in developing ways in which we can use RWE and data analytics to inform our work. And we're engaging with academics and data experts and research that aims to help us understand how we can make meaningful use of real-world data. And this work is one example of that. So just to start by acknowledging the contributors who contributed to this project. You can see those here. And this work is a starting point, Danielle, as you mentioned in understanding whether EHR data from different jurisdictions can help reduce uncertainty in HTA decision-making.

And this is really sparked off by the fact that the majority of drugs marketed both in the US and Europe are first launched in Europe–no, in the US of course. So we wanted to understand if, at least in principle, if EHR-derived real-world data could inform HTA on our side of the pond. So if we could have the next slide, please. So the typical journey of an oncology drug is an approval by the FDA followed by a marketing authorization in Europe. And then shortly after the marketing authorization, NICE receives a submission from manufacturers. And we will begin our appraisal, which then ends with a final appraisal determination, which then indicates whether the drug is approved for the use in the National Health Service. In some cases, if there are still outstanding evidence gaps after appraisal, the drug is then approved in the context of the Cancer Drugs Fund. The purpose of the Cancer Drugs Fund or the CDF is to enable early access to promising new drugs whilst at the same time enable further evidence collection that could help close the initial evidence gap. So that's the context to this. In this study, we've measured time from an FDA approval to EMA marketing authorization, really as a proxy for authorization in the UK. We also measured the time from FDA approval to manufacturer submission to NICE and the publication of NICE final guidance. And the drugs in scope, oncology drugs that had NICE approvals published between January, 2014 and December, 2019. And we then stratified the results by various parameters such as the therapy class, the cancer type and whether the drug had a biomarker driven indication and so on. And you can see that on the screen. In terms of the clinical data, we used data from Flatiron Health, EHR-derived data. And for the NICE data that we used the publicly available appraisal with documents for single technology appraisals.

In terms of outcomes, we evaluated the total number of patients exposed over time to oncology drugs approved by the FDA and EMA followed by a NICE appraisal. So if we could have the next slide, please. So here you can see the inclusion criteria. We selected a set of NICE technology appraisal of oncology drugs completed between, as I said, 2014 and 2019. We then whittled it down to 60 appraisals associated with 39 cancer drugs across 11 different cancer types. And if we move on to the next slide, please. And this is a breakdown of cancer types within our selection. As I said, we included 11 cancer types with the largest types being advanced non-small cell lung cancer. We had 16 appraisals in that category. Followed by advanced melanoma where we had 11 appraisals and then seven each of metastatic breast cancer and multiple myeloma. So we had a relatively broad group of cancers for this exercise. So the next slide, please.

So in terms of the results. So start with that. So this essentially shows us the journey from an FDA approval to publication of NICE guidance. And the timeline drive from the study now is as indicated here in months. So an FDA approval preceded EMA's approval by a median time of 5.3 months. We then had FDA approval preceded by submission to NICE by 6.4 months. And finally, an FDA approval proceeded to NICE publication by 18.5 months. And just to flag that, within these timelines there is quite a lot of heavy duty evidence review process and economic modeling followed by decision-making. So NICE's decisions are done by independent appraisal committees and is a consultative process. So we produce an initial document called Appraisal Consultation Document, which we then ask stakeholders to comment on before we review the final comments and make a final decision by committee again. But this partly explains why the process takes 12 months as you’ve seen in this example here.

And this represents the picture of the time period from 2014 to 2019, as I said, but the time lags are shorter in the second half of this period. So they're quicker by about 20%. And beyond this period that we studied, I know that the process now is even a little bit shorter today. So if we could have the next slide, please. So this is looking at uptake of drugs after FDA approval. So what we're looking at is the number of uses per drug in the EHR cohort. And as you would expect, the numbers increase with time. At the time of manufacturer submission to NICE, an average of 14 point... Sorry. An average of 147 real-world patients had received the drug of interest on label after FDA approval with a median of 4.5 months of followup time. This then increased to 269 patients, and 6.4 months of followup time at the time of NICE final guidance publication.

And if you look at this stratification according to subsequent NICE decisions, the average count in the US cohort were higher for products than NICE recommended for the Cancer Drugs Fund than the average counts for other decisions. So in essence, we have a greater accumulated follow-up for drugs that go into the managed access with evidence generation pathway process, if you like. Next slide, please. And now looking at split between treatment types. This shows the number of patients when NICE published its guidance. The black bars are the minimum and maximum patient counts. So as you can see, there's a great variation in numbers within each treatment type. But actually one insight we gained from looking at this is that the rate of patient accrual at Flatiron Health for drugs in the chemotherapy and in the immunotherapy classes appear to increase over time more so than in other classes. And the next, please. And finally, here we see the median follow-up time available in the data after the patient received the drug of interest. This slide shows stratification on the Y-axis by the number of months since FDA approval. And again, you can see a great variation. But if you recall, one of the earlier slides that showed the evidence submissions and the timeline is that the evidence submission to NICE comes around the six month mark. So there may be limited follow up at that time. However, for those drugs that go into the Cancer Drugs Fund, the CDF, for further evidence collection, the follow-up time is potentially towards the top two bars. So around six months and potentially much longer. And moving on to my final summary slide. So the time from FDA approval to NICE guidance, as we've seen, may provide an opportunity and provide additional information for HTA decision-makers based on real-world US patients.

There are opportunities to use this data, but that will vary, I think, by cancer type, the nature of the uncertainties that are identified in our appraisals. So these uncertainties are often longer term measures. As we've seen before, the audience thinks overall survival and progression-free survival is important. We often see sort of issues and uncertainty around those longer term outcomes, as well as quality of life outcomes. And other opportunities will depend on whether the data is reflective of the UK patient characteristics, the treatment settings, and the clinical pathways. We need to make sure that we're looking at apples and apples and pears and pears rather than a mix of things in different jurisdictions. And this really is indeed what the next steps will explore in this exercise. So the big question is, is the data appropriate in the decision making context of European HTA? And the last slide again is acknowledgement to the authors of this study. And thank you very much for your attention.

Danielle Bargo:

Fantastic. Thank you so much, Páll, for presenting those results. So we will now do one last poll before we move on to our panel discussion. And the question is which HTA bodies do you consider most accepting of RWD for HTA decision-making? And the audience shouldn't feel pressured given Páll's presence. The options are NICE from England, HAS from France, IQWiG from Germany, PBAC in Australia, CADTH in Canada, or AIFA in Italy.

Okay. We can go ahead and end the poll and share the results. I think the audience might've been swayed, Páll. 76% of the participants said that NICE is the most accepting of RWD for HTA decision-making followed by Canada in second place, 13% IQWiG and HAS had the lower numbers and AIFA the lowest. So thank you everyone. I think that does reiterate the importance of why we're speaking with Páll today. So we will now move into the panel discussion part of our webinar to get more insights into the research. So let's go ahead and start asking some questions. The first question is for you, Akshay. Regarding what you presented, what do you consider the driving difference in the credible intervals between the clinical trial and real-world data?

Akshay Swaminathan:

Thanks, Danielle, for that question. So as I mentioned in the traditional cost effectiveness approach, the hazard ratios that were calculated using the network meta analysis had quite large confidence intervals. And if you compare the confidence intervals of those hazard ratios with the ones estimated from real-world data, that's where you really see the shrinking of uncertainty because those hazard ratios are then inputs into the cost effectiveness modeling. Now, why do we see smaller confidence intervals in the real-world data? So we saw that we had a much larger chemotherapy comparator arm with 1300 patients compared to the clinical trial. Compared to our arms, we also saw a longer followup time, which as I mentioned, led to lower rates of censoring. So those are some drivers.

Danielle Bargo:

Okay. That makes sense. And Scott, I'm going to transition to you now because one question is, how can this type of real-world evidence derived from electronic health records be used by organizations such as ICER in the future?

Scott Ramsey:

Thanks, Danielle. I think this data can be highly relevant for ICER. The problems ICER faces in terms of making decisions quickly for products that have been FDA approved are similar to what NICE has although ICER has to work on a much more aggressive timeline because the US market demands uptake of the products very shortly after FDA approval. But over time, I think when ICER finds that there is more questions about the evidence for efficacy, that the real-world data could be used for a re-evaluation down months from when the original decision was made similar to what's kind of a built-in lag that NICE has right now, and this can be transmitted back to the US decision-makers who, by that time probably will have approval, although it may come with different types of restrictions based on what the individual health insurers view as the level of evidence supporting the product. So, I guess the short answer is I think we can use this type of data for reassessment, particularly in cases where there's a large degree of uncertainty about the effectiveness of products for reassessment and repositioning within US formularies.

Danielle Bargo:

And I think that leads nicely to another question we have for you, Scott, is what do you consider the future of using RWD to influence reimbursement decisions here in the US?

Scott Ramsey:

Well, it's a tricky question because predictions are difficult, especially for the future, as Yogi Berra said, but in the reality, I think there are plenty of places where this information can be used. First of all, I think the most promising area would be in value-based pricing. Over time, as payers get experienced with products, if they can set up contracts with manufacturers to adjust both access and price, based on what's being seen in real-world data, that's an opportunity to address concerns regarding uncertainty, with regard to cost-effectiveness, but also manufacturer concerns about access in situations where the products have a high degree of uncertainty about effectiveness.

Other places could be in simple price negotiations that would change over time, not necessarily tied to value, but tied to observed use and outcomes. And, I think one other area is simply for health insurance plans and providers, to understand what factors drive outcomes in patient populations after products are introduced. This is a high uncertainty area, many physicians, as they gain experience with products over time, find certain cohorts that weren't necessarily represented well in clinical trials that do better or worse, depending on their characteristics. A good example would be patients with poor ECOG performance status because those patients aren't always represented very well in clinical trials. So that real-world data can directly inform clinical decision-making down the line as that experience is reflected in real-world data and presented back to those who are making the decisions.

Danielle Bargo:

Really good point Scott, thank you for that. And, Akshay another follow-up question regarding your presentation, why is a smaller confidence interval around the cost-effectiveness ratio important for value assessments?

Akshay Swaminathan:

Thanks, Danielle. I think this is most relevant when smaller confidence intervals can elucidate a clear difference between two health technologies. So, if ICER or NICE is interested in comparing drug A with drug B, perhaps, you compare the spread of simulated ICERs, it appears that one drug is more cost-effective, but because the distributions overlap, it's unclear, there's uncertainty associated with that effect estimate. Now, if you can somehow shrink the uncertainty and elucidate a clear difference that lends more certainty to HTA decision-making.

Danielle Bargo:

Thank you for that very clear explanation. Moving to Páll, so Páll, what key uncertainties do you typically see in the assessment of oncology medicines?

Páll Jónsson:

Yes, there's potentially quite a lot that we see. Again, to start with a little bit of context, so in England, we only have one payer and that really simplifies things somewhat, but that also means that we can apply a unified approach to ensure that we are maximizing the health we get for every pound or dollar we spend. So typically, we use cost-utility analysis on their treatment. We assess, and this is done because by doing that, we can apply the same value for money framework across different disease areas. So as a consequence, we're reliant on the instrument we use, and the instrument we use at NICE is the QALY, the cost per quality-adjusted year.

And, this is really the key instrument we use to assess cost-effectiveness, but it's a key instrument for us. There are clearly other factors that come in, but the key uncertainties, again, relate to the ingredients that go into the quality. So the length of life gained on that particular treatment compared with standard of care, and then also the quality of life during that gain. And we often see issues due to limited follow-up within trials that affect these measures. So, we don't always have the long-term clinical outcomes that we need to be able to capture the entire health gain from a treatment over the lifetime of the patient, because clearly trials are not run like that.

This is not just an issue just for oncology projects, but generally speaking. And, the most frequently cited uncertainties for cancer products are the longer-term measures, as we've heard before, such as the overall survival, the progression-free survival, in addition to quality of life. And, this is often what we look at in the Cancer Drugs Fund as well to shed light on. And, these are the parameters that are often most of interest, the longer-term parameters. But also, as I mentioned before, we sometimes see uncertainty when our committees are not convinced that the actual trial represents clinical practice as we see it in England. So we have generalized ability issues here as well, which real-world data may or may not depending on what it does or how it's collected and by whom and on what, on which patients can help shed light on. So, yeah.

Danielle Bargo:

Thank you for those thoughts, Páll, and also the context in terms of how you're actually making decisions in the UK for reimbursement and market access. Blythe, we have a question for you. So what key real-world variables are available from electronic health records that make it more appropriate to answer some of these questions relevant for HTA bodies?

Blythe Adamson:

Well, one of the most common things that I'm trying to do when using electronic medical records is to identify a population who aligns with the label indication for a drug. We see a lot of variability in clinical practice, but often, we want to know, do they actually have the biomarker status that aligns with the label indication or, have lab values. And so, as I worked with lots of different types of data, one of the challenges for me before was knowing that a lab was run, but not having the result to it, knowing that a biomarker test was paid for, but not knowing what the result of it was. And so, Scott's also mentioned ECOG performance status. I've also found that to be very important in understanding the likelihood that someone receives one type of drug versus another.

Danielle Bargo:

Thank you for that. And, I think that goes really nicely with what some of the participants were responding to as well in terms of what are some of the key variables that are important to them. Páll, another question for you, how would you say real-world data from US patients can actually be used to inform or address some of these evidence gaps that HTA decision-makers in Europe face and what should be considered when really assessing the appropriateness? I know you've mentioned the generalizability, but if you have any recommendations on what needs to be addressed, that'd be great.

Páll Jónsson:

We can probably go on all day to talk about this question. And, it is essentially also what the subsequent pieces of work are going to be looking at. But I think for me, we have two key points of interest here. So the first point is to, here as we've seen, to take advantage of the time difference of launch to market in different jurisdictions. So here, we're looking at a little bit more time with drugs on the market in the US before they reach the market in Europe and go through subsequent HTA assessments, and that's something that could be taken advantage of.

And, keeping in mind also that the pivotal trials on which regulatory decisions are made are typically the same in the US and Europe. So, we have some additional material here that could potentially input into our decision-making. Clearly, it's better to make decisions with data rather than in the absence of data, but there is a caveat here. We need to make sure that the data that we're looking at reflects our decision problem and that it is of good quality. That's always going to be the first sort of criteria that we look at when we're looking at non-randomized data. And the second point is, so we have real-world data that's generated in practice, and that's a real bonus. So as we heard before, there are potential differences between the characteristics of the patients recruited in clinical trials versus the patients you see in practice. And also, how treatments are used in practice may differ from the trial protocol.

So there is an opportunity to develop further insight into how real-world patients respond to treatment, and that's always useful to get that data, but again, as with any non-randomized data, you clearly need to be aware of the issues that can appear in such data, potential for biases, or quality issues. But if it's done well, there should be useful information here about real-world patients. And you mentioned, what should you be looking at sort of in terms of considering appropriateness? So again, the devil is always in the details. You need to look at the various aspects of the data. And just off the top of my head, some of the considerations could be, is the follow-up period long enough to be meaningful, over in both the clinical trial data that we need, are the patient numbers sufficiently high enough so that you can make robust inferences on the data? So, we're not looking at RCTs where we've done power calculations, so we need to make sure that the patient numbers are sufficient.

Importantly, when you're looking at data from, as I mentioned, different jurisdictions, is the data from the US generalizable to the UK setting? That's a big question that we'll be looking at. You can't really assume that you can generalize or transfer data from one country to another country for various reasons. So, you will need to look at the compatibility of the patient characteristics, age, race, gender, biomarker status, et cetera. You need to look at the treatment setting and look at whether they are equivalent, as, for instance, are patients primarily treated in specialist centers in one country and not in the other? So, it could be a difference of the factors in the way patients are treated that affects the final outcome. You need to look at again, whether the clinical pathways are comparable. For instance, are people receiving the sort of similar sequencing of drugs and at similar clinical time points and many more factors. So there's quite a lot to look at when we're assessing the usefulness and appropriateness of using this data. But clearly, there's a lot of opportunities here and a lot to understand for us.

Danielle Bargo:

A lot of great considerations Páll, and I think our typical answer when asked those types of questions is always, it depends.

Páll Jónsson:

Yes, and it depends.

Danielle Bargo:

I think your explanation is best summarized by it depends. Okay. So we are now going to move, we're going to conclude the panel discussion and transition to the audience Q&A because we've had a lot of questions coming in through the Q&A function. Please do continue to ask your questions there, as we will use the remainder of the time to answer any questions from the audience. So, the first question we have from the audience is for Akshay and Blythe. So Akshay, we can go to you first and it's regarding adjusting for baseline characteristics. So, the question is that RCTs are considered gold-standard due to randomization and balance of baseline patient characteristics. In your analysis, you mentioned that baseline characteristics were not accounted for. How do you think the results would change if the analysis adjusted for baseline variables?

Akshay Swaminathan:

Thanks, Danielle. So, that's correct, we didn't apply population adjustment methods such as matching in our analysis. And before, while these methods can definitely improve balance between cohorts, it's important to first measure the initial levels of covariate imbalance before applying them. And, the reason is that there's always a trade-off between bias and variance. So, it's important to measure initial levels of covariate imbalance, and then assess whether the improved balance from applying something like matching is worth decreasing the sample size and potentially lowering the precision of the resulting estimates.

Danielle Bargo:

Thank you. Blythe, do you want to add any additional comments?

Blythe Adamson:

No, I think that we have many tools in our toolbox for adjusting between patient populations and trying to reduce unmeasured confounding, in these comparative effectiveness studies.

We often try lots of different methods, for example, adjustment in regression, propensity scores, whether for matching or inverse probability weighting. And now we're seeing a lot more emerging methods for instrumental variables. And I think it's encouraging when we try lots of different methods, and the conclusions are consistent across different approaches. So I think that we'll see continuing work in this area.

Danielle Bargo:

Thank you both for those responses. Now, a question for Scott and Páll. So we'll start with Scott. The question is that RWE can be very useful for post-reimbursement assessments, value-based agreements, coverage with evidence development, but how can RWE be used at the time of reimbursement submissions prior to the product actually being available on the market? So Scott, we'll go with you first.

Scott Ramsey:

Yeah. This is a great question. And particularly for U.S. payers, very pertinent because as I mentioned, U.S. payers have to make a decision about putting a product on their formulary, essentially at the time of an FDA decision. So they don't have much time. And the evidence is often limited. And what I would say is, particularly for cancer drugs, the evidence challenge we face has to do with the control group, the comparator group.

In many situations, the control group is not a drug that is used as commonly in practice, but was picked for the trial for particular reasons. In some situations, trials don't have a control group and instead use a synthetic control. This is particularly true for CAR-T drugs that are usually one-arm trials. And in this situation, real-world evidence can be used to really understand incremental benefit by assembling a comparator arm that looks similar to the characteristics of the intervention in the trial, but is drawn from the real world.

And it has two primary advantages. One is that you can do direct matching and a comparison to make the control group look very similar to the clinical trial enrollment criteria. And that can give you an estimate of incremental effectiveness, or of efficacy, I'm sorry. The other advantage is that you can, with that control group, look at extended follow-up. So the other part of HTA would be doing an evaluation where one would model beyond the trial time horizon to estimate lifetime effects. And with a control group that has extended followup, it may be easier to model outcomes for the control group, such that it's a more accurate estimate of incremental effects over a lifetime.

Danielle Bargo:

Thank you, Scott. And Páll, do you need me to repeat the question?

Páll Jónsson:

No, that's fine. So as Scott, I would have said what Scott mentioned was the extrapolation of long-term outcomes. So to use real-world evidence to help inform the extrapolation, which is done by modeling mostly, so inform which parameters are useful for that is really useful. And anything that helps us inform on comparative effectiveness, again, is something we'd be looking at.

But really, I think if you're looking at early stages of submissions, real-world evidence can still be really quite useful. So even at the planning stages and when you're composing the dossier to HTAs, so for instance, to characterize the natural history of disease, what's the journey of a patient, a typical patient, both in oncology? And then also I think it is really important for long-term chronic diseases as well to understand that, real-world evidence is really, really powerful for that purpose to characterize the patient population more generally, and just to inform the trial design. So real-world evidence can be really useful to inform the trial design so that the trial isn't in the end too artificial, but tries to simulate, to the extent it possibly can, the actual reality. So, yeah, but in addition to Scott's excellent answers, I think that's all I would say.

Danielle Bargo:

Thank you so much, Páll. Okay, moving on to the next question. This one's for Blythe. Blythe, moving into a pre-metastatic disease, what real-world data is available on key endpoints such as disease-free survival and EFS?

Blythe Adamson:

Yes. Well, so one of the advantages of having the underlying electronic medical record is it allows abstracters to manually go in and open up these charts. And so often in addition to research ready data sets that have curated variables that are very common for across research questions, it also allows the design and abstraction at scale of new variables and the validation of endpoints that are important to patients and important to the decision. So we've seen a lot of work and studying. I've recently done some work on progression-free survival, and I think that that's the most common one that I've used within Flatiron data. But, I would say an encouragement is that the development of new endpoints is happening all the time, and really having that access to the underlying story about a narrative about patients can really help us link together imaging documents and the patient and clinician experience.

Danielle Bargo:

Thank you, Blythe. Another question for Páll. Any thoughts on how real-world data can be used for tumor agnostic therapies? Are there study designs that we should be considering?

Páll Jónsson:

Yeah. So tumor agnostic therapy is something that we're actively looking at, and we will increasingly see more of those coming through assessments. They pose a bit of a problem for us because you potentially might have to construct multiple economic models based on the actual tumor site, even though the mechanism of action is the same, and the drug is the same, there may be different outcomes and so on. So I think real-world evidence can supplement and give us information on what is often quite small patient populations in each subgroup that's being treated. So early days for that, we've had some trials coming in and some appraisals looking at this, but again, we'll be seeing a lot more and we are actively looking at, how are we dealing with that?

Danielle Bargo:

Thank you, Páll. And Akshay, do you have any additional comments given your role at Flatiron?

Akshay Swaminathan:

Yeah, I would just add that with these histology agnostic therapies, what we often see in submissions is a few tumor types with larger sample size and then a very long tail of much rarer histologies. And I think it's really in that long tail of rare, rare histologies where real-world data can supplement HTA decision- making with the longer follow-up time and increased sample size since that's generally the subpopulation with the greatest uncertainty.

Danielle Bargo:

Thank you, Akshay. Okay. We will conclude with one last question. The question is for Páll. When defining a cohort of data for HTA, should the population align with the clinical trial or the expected license or indication?

Páll Jónsson:

Oh, good question. It sort of depends on what you want to glean from the data. So if you're trying to emulate the trial, then you'd be looking at the patient probably at the trial cohort. My feeling is that you get more out of emulating the patient population for which you're trying to assess the clinical and cost effectiveness. So it's a balanced and nuanced question that needs to be looked at on a case-by-case basis. But perhaps Scott can add to that.

Scott Ramsey:

Yeah, I agree with Páll that that's a really good question. I think ultimately the short answer would be that we want to do an evaluation of how the drug is going to be used in the real world. And for that, I think we need to look beyond the clinical trial and understand how the drug will be used in the clinical indication. The issue, of course, that we face is that clinicians don't often follow the guidance. And so the clinical indication as written through guidelines or guidance can be an intermediate step between what we'd like to see happen and what actually happens. But certainly moving closer to real world, I think, is going to be the better assessment approach.

Danielle Bargo:

Thank you, Scott. So we actually had one last question come in and it's for Blythe about variables. So I just want to squeeze this in in the last minute. Outcomes such as PFS are often challenging to accurately capture from EHRs. How can this challenge be overcome?

Blythe Adamson:

Hm. Well, so we do have a publication by Flatiron in a peer-reviewed journal with some of our approaches for validated a real-world progression event. But I think that we do have to acknowledge that real-world disease progression is going to be different than a trial-based measure like RECIST. It's really not the same thing. We don't always have a baseline scan to be able to measure differences that have changed over time. And patients in the real world are scanned at different frequencies depending on what treatment they're receiving. And we don't know, that can also create bias that we have to adjust for.

So I think that this is an emerging field where lots of different groups now are trying to figure out how we can validate these outcomes. But often I think that some of the true tests are looking at where you have some kind of gold standard of progression that's been detected in a trial, and then you compare it to, try to mimic that type of abstracted variable from the patient's charts and see how close we can get.

Danielle Bargo:

Thank you, Blythe. And with that, we will conclude. So thank you to all of our speakers for today. Thank you to all of our attendees for joining and asking such great questions. If you have any questions about the content, please don't hesitate to reach out to Flatiron, and a friendly reminder to please take the survey upon closing out to help us improve future webinars. Have a great rest of your day. Stay healthy and stay safe. Thanks everyone!