Welcome to the 2022 Season of ResearchX. In this initial episode, we introduce you to the concept of integrated evidence, our vision for the next era of evidence generation and how it will power smarter care for every patient.
Transcript
Carolyn Starrett:
Hello, and welcome to ResearchX. For those of you that I haven't yet met, I'm Carolyn Starrett. I spent the first 15 years of my career working at the intersection of data and technology. And then I joined Flatiron Health six years ago, and led what we call our HC business. This is the work we do partnering with cancer centers on the software and services that they rely on to power a better patient experience and better patient outcomes. And then stepped into the CEO role last April. I have the honor this year of kicking off our ResearchX series, and we're really excited to do something a little different. We've put together an entire season of live ResearchX episodes that are going to run over the next few months where you'll hear from your peers across the industry about how real-world evidence and the next wave of tech and science have and are going to continue to transform the way we think about cancer research.
Flatiron is celebrating an important milestone this year. It just happens to be our 10th birthday. So we've been talking internally, and it's a really interesting time to reflect back on the progress that we've collectively made over this last decade and the opportunities we see ahead for the decade to come. So throughout this season, we'll be celebrating a lot of the great work you all have been doing and sharing some exciting progress that we've made together. And I want to underscore that the progress we've collectively made to learn from the experience of every cancer patient, to this point with real-world data, in no way could have been possible without the ambitions and the hard work of our research partners from biopharma, to regulators, to global health authorities, to academic research institutions. And that's really important context as we consider this season of ResearchX. We do this because forums like this are a really key part of the collaborative work we rely on to advance oncology research for the industry and for patients, just as much as the technology we develop and the data that we curate.
So have a look at the bottom of your Zoom window. The Q&A button is a very powerful thing. Hopefully we've all gotten pretty familiar with it over the last few years, but this isn't Flatiron's show. I really hope that you'll see this series as an opportunity to advance scientific discourse. This is an opportunity to start to lay out what the future path of evidence might look like. And for this to work, we're going to need the participation from all of you. So over the next few months, our series will include your colleagues from Amgen, BMS, Janssen, Genentech, Novartis, Takeda, John Hopkins, and Beat AML, to name just a few. So we're really looking forward to advancing the collective thinking through your participation. Thank you in advance.
So, as I mentioned, we're now 10 years into Flatiron's journey to improve lives by learning from every cancer patient. And today, we're able to reflect on our ambitions for real-world evidence and, from our vantage point, what it's actually capable of. Stephanie, our General Manager for Real-World Evidence is going to share with you a handful of the milestones to date that we're so proud to have been a part of. And that's been both exciting and clarifying for what's needed to further realize our mission.
So for that reason, I'm excited to share with you a preview of the next chapter for Flatiron; building on our bedrock of real-world data. We want to talk to you about how we go further with integrated evidence to achieve transformative benefits for a host of new applications, including new ways we're thinking about interventional clinical trials, as well as new use cases. And in some ways, integrated evidence isn't a new idea, but in other ways, we really think there's an opportunity to stretch its potential through some very new contributions. And we're going to spend a lot of time today and in future ResearchX episodes digging into the substance of it. So to kick things off, we've put together a video that I'd like to share to get the conversation going.
Narrator: Integrated evidence is the knowledge and insight that comes from carefully bringing together different kinds of information, the same way a picture emerges as you put the pieces of a puzzle together. Each piece has its value, like adding a new domain of health information, new patients or even entire populations. And while each piece alone offers limited insight, together, they help us see a fuller representation of reality.
But what more can we learn if we adjust our perspective? What more can we understand if we combine new modalities of data, like structured information about a tumor's progression with the richness of a CT scan or with the nuance of a physician's note? After all, each one is just a lens through which a fact can be observed. Now imagine combining real-world data with prospective data. We can start to picture a world where we can improve our understanding of causal relationships, a world where we can unlock inclusive generalizable data, and from there, a world where we can make better informed decisions and accelerate R&D and access. To realize this vision, it'll take all of us.
Carolyn Starrett: Awesome. So let me pass the mic to Stephanie, our General Manager of Real-World Evidence to tell you a little more. Stephanie.
Stephanie Reisinger: Thank you, Carolyn, and hey, everyone. For those of you that don't know me, my name is Steph Reisinger and I'm the General Manager of Flatiron's Real-World Evidence Business Unit. And while I've only been here at Flatiron for a short time, I've actually been in the real-world evidence space for a much longer time, more than 15 years. And during that time, I've seen some amazing changes. And as Carolyn mentioned, this year is Flatiron's 10th birthday.
So in my time today, what I really want to do is to share some perspective. First, we'll take a look back over the past 10 years, and I'm going to highlight some of the key milestones that have impacted our industry. And then we're going to look ahead and ask the question, "What will the next decade look like for oncology research?" And so my goal today is to provide some high-level context for this year's ResearchX topic, which is integrated evidence. And so let's get started by looking back. And as I said earlier, we've seen some amazing changes in our industry in the past decade. So for instance, in 2010, and that was just a little over 10 years ago, a majority of patient health records were stored on bit paper in big wall-to-wall filing cabinets in the doctor's office. Do you guys remember that? Now let's fast forward to today. In the US, computers have almost completely replaced these paper files. And today, we're regularly using electronic health record data to research important topics. So as an example, the JAMA study shown here is one of just thousands of studies using EHR data during the pandemic that helped us to understand the impact of COVID as it spread across the globe. And just a decade ago, just 10 years ago, this research would not have been possible.
Besides adoption of EHR, another key milestone for our industry was the passage of the 21st Century Cures Act in December of 2016. This was a mandate for the FDA to evaluate the use of real-world evidence for regulatory focused purposes and new drug indications, and also in post-approval drug safety and dosing. And along with others in our industry, Flatiron answered the call to action. So let me show you some examples.
Here's an example of an expanded indication. So in April of '19, the FDA approved an expanded indication for IBRANCE to include male patients. And this was based partly on Flatiron's EHR data showing that the safety profile for men treated with IBRANCE is consistent with the safety profile for women. And what that really means is that today, men with breast cancer have an approved therapy option. And here's a dosing example. In April of 2021, the FDA approved a new biweekly dosing regimen for Eli Lilly's ERBITUX. And this approval was partially based on survival analyses using Flatiron EHR data in patients with colorectal cancer. And what this means is that now patients with EGFR mutations for certain cancers have an alternative, more convenient dosing regimen that's easier to tolerate, and with fewer total days at the infusion site.
And I want to share one more example. And this time it's related to drug safety assessment. Flatiron data was used to study cardiotoxicity risk in breast cancer patients with certain heart conditions. These types of patients weren't included in the clinical trials. And in late 2020, Genentech's KADCYLA label was updated. And now women with these heart conditions have access to HER2-targeted therapy options.
In addition to the advances in real-world data, oncology science has also really seen some amazing advances in the past decade, and these are bringing great improvements to cancer care. But despite these advances, there really hasn't been a commensurate level of innovation in how we actually do that science. In fact, in clinical trials, which is the gold standard for evidence generation, they look basically the same as they did a decade ago, despite some pockets of innovation around decentralized trial technologies.
And as the COVID pandemic has highlighted, performing inclusive research also remains mostly out of reach. So I think it's clear to many of us that our current research operational model really is not meeting the patients where they are today. And so that's where we stand today, a ton of progress, but also a lot of challenges. So now I want to switch gears and look ahead and consider the question I asked earlier in the presentation. And that is, "What could the next 10 years look like for life sciences research?" And to help us consider this question, I want to broaden our lens, and I want to look at what's happening outside of life sciences in the macro healthcare ecosystem.
And so one of the most profound trends in healthcare continues to be digitization. It might have started out with EHR, but today everywhere you look, every aspect of healthcare is being digitized, tracked, app-ified, you name it. And this healthcare digitization is creating more and more and more data. But maybe more importantly than data, it's also causing a massive shift in how healthcare is delivered as these digital capabilities are integrated into point-of-care workflows and patients are getting more control of their own data.
So what does this really mean for life sciences? I would say that the digitization is not only changing the way healthcare is being delivered, but it's also creating a huge innovation opportunity for us within life sciences. So for example, imagine if we can engage a much broader swath of physicians and patients differently by expanding digital point-of-care workflows to include research. And as another example, what if we can move beyond single-source evidence generation by thoughtfully combining and analyzing all of these multiple real-world data streams together? So these are just a couple of examples. As an industry, I believe we're just at the starting line. With this ongoing healthcare digitization, we have a huge opportunity in the next decade to innovate and improve how we research.
So that brings us back to Flatiron. At Flatiron, we occupy a really unique position at the intersection of oncology, healthcare delivery and research. Our oncology care network is really the foundation for everything we do. And within this care network, we're already working to connect physicians and patients to research opportunities at the point of care. And on the other side of the business is evidence, which is really near and dear to my heart. For the past 10 years, Flatiron has invested heavily in really the hard science of oncology real-world evidence. And in fact, in 2021, Flatiron and our partners have published more science using our data than any other oncology real-world data source.
And as we've said, going forward, the next chapter of Flatiron will be focused on this thing called integrated evidence. And at the highest level, what we really mean by that is improving our oncology care network and extending our workflows while learning from all of the data being generated by every patient. And then finally, bringing this evidence-based knowledge back to the point of care to improve patient lives. And by doing all of that, we continue to fulfill our mission, which is to improve lives by learning from the experience of every cancer patient. And with that, I'm going to wrap up my part of the presentation. I look forward to your questions at the end, and don't forget to use the Q/A tool if you have a question. For now, I'll turn it over to my colleague, Shane Woods, to talk more about integrated evidence. Shane, the virtual stage is all yours.
Shane Woods: Thanks, Steph. And thanks everyone for joining. I'm Shane Woods, Chief Commercial Officer atFlatiron Health. As Steph said a minute ago, we're really at the starting line of where this all could go. And we're humbled by the impact of the use case example Steph shared earlier, just given where we in the industry were back in 2016 when Cures first passed. But it's not just the successes across a decade of experience, it's all the shots we've taken, including the ones we missed. It's this experience that has revealed the strength and also, frankly, the limitations of real-world evidence; those examples where we receive critical feedback from health authorities and regulators. It's these learnings taken together with the proliferation of digital healthcare information that inform how real-world evidence fits into this broader evidence ecosystem. And this has been the catalyst for us in defining this vision that we're now leading for the industry. To achieve transformative change in how cancer is treated and researched over the next decade, we recognize that we'll need to think beyond real-world data while keeping it at the core of what we do. So we've set out to expand on the foundation we've already established in the industry, including harnessing the emerging blurring of lines between approaches to observational and clinical research. That's led us to this idea, in discipline, of integrated evidence, an area some of you may be familiar with and an idea that Flatiron is uniquely well positioned to contribute to because of our position, as Steph described it, at the intersection of research and care, which is allowing us to stretch integrated evidence in new ways.
So we're going to spend a few minutes just talking about what integrated evidence is and why it's critical for the future path of the industry. Let me start with what the benefits are if we get it right. This is the promise of integrated evidence, which starts to address head-on where Carolyn and Steph started today talking about the mounting challenges with traditional approaches to evidence. And no one in the industry would argue against the importance of these benefits, or at least no one has argued with us yet. In fact, just about every CEO in the industry has shared some version of twice the approvals at half the R&D costs. It's hard to believe this could be achieved with the same set of tools that we have today.
Let's dream for a moment. A decade from now, what if we could accrue for your study in a quarter of the time because consented patients have the even greater chance of receiving the experimental treatment? Or because we've expanded the pool of eligible patients by bringing clinical trial opportunities to the communities where these patients live? Or what if we could dramatically improve the probability of technical success across your early development portfolio through an arsenal of new evidence tools? Finally, what if we could earn accelerated approval and be confident that you could seamlessly and more efficiently embed your post-marketing evidence needs into everyday care for patients and sites?
This is why at Flatiron, we fully embrace integrated evidence. But we also recognize that we can't do this alone, so I'm going to shift gears for a minute to talk about what integrated evidence is. We're going to start with a definition. Of course, with any well-written definition, it appears deceptively simple. Unlike single-source evidence, you can think of integrated evidence as the result of carefully, thoughtfully bringing together multiple sources of data where we end up with a new thing. Importantly, that's achieved with research methods that are tailored for integrated evidence and, done right, integrated evidence will provide insights that are more robust, reliable and useful that wouldn't have been possible using any of the component source data alone.
As many of us know, that's easier said than done. It's difficult enough generating evidence from a single source of data, as I'm showing here. When your end goal is robust evidence from a variety of data coming together ... In other words, when generating integrated evidence ... The problem space explodes. Now, we must think critically about the appropriate research methods for combining these additional data sources that come from routinely captured data and the need for adapted or, in some cases, entirely new analytic techniques. For mortality data, which I'm showing here on the left, at Flatiron this is a composite endpoint that comes from several different data sources. Even something as simple as date of death needs careful consideration in terms of how it's developed and used.
This is much more than linking data sources together. We have to do the work to assess how successful those operations were, if there's a need for adapted or, in some cases, entirely new analytic techniques. The modality is just one of the lenses through which we can look at data. We could imagine a use case where the routinely collected data isn't sufficient, as I'm showing here, and with advancements in tech and statistics, that we'll begin to talk more about today, we could enrich this evidence through intentional data collection. In this case, directly through the electronic health record.
Being able to combine these different types of data present new possibilities. Although integrated evidence represents a nascent discipline, there are lots of examples where we're already taking this approach at Flatiron, like our composite mortality endpoint, as I mentioned a minute ago, which is embedded through all of our offerings at Flatiron today as just one example. And as we further expand the aperture of integrated evidence to include more kinds of information, we also expand the range of potential for evidence that wasn't possible previously. Think of imaging-based real-world endpoints using a patient's radiographic images as an example; an area that Flatiron is heavily investing in today. And it's this expanded aperture that allows us to unlock new use cases that were previously out of reach for more limited forms of evidence.
At the same time that we're opening up our eyes to even more kinds of data, we're also collaborating with our industry partners to define what good looks like in terms of applications of these data. And there are really three key considerations in creating integrated evidence. You can think of integrated evidence as a discipline for how we can generate, combine and analyze data. And I’ll provide more color for each of these in turn.
These don't necessarily need to follow this order, but let me start with generate. This, of course, relates to how we generate data. In our case, captured through the electronic health record as part of routine clinical care or increasingly, prospectively by intentional data capture workflows embedded directly in the EHR in support of novel prospective study designs. This also relates to how others generate data; the data we source to combine with ours like genomic data from Foundation Medicine or Medicare claims, for example. Even beyond that, we sometimes generate data from our own data using machine learning models to impute new information, all of which requires careful consideration.
Next, combine. This is the work we do to successfully connect and combine data sources with similar or different modalities. Here we think about enriching patient information through linking to radiographic images, for example, or developing a composite endpoint like our mortality variable. It's here where we think about the basic operations of linking and pooling but, critically, also the methods and validation work to assess the performance of the combined results. Given this carefully combined data, we'll next consider the analysis; the research methods. With integrated evidence, we can't just apply our standard analytic tool kit. In some cases, we can adapt existing statistical methods but in others, entirely novel methods may be required. We see this as an important area of collaboration with our biopharma partners, the data science community and other key stakeholders like FDA, NICE in the UK and others that we're directly partnered with.
Let's make this a bit more tangible. I'm going to walk through an example end-to-end how we take an integrated evidence approach to our mortality endpoint. I'll walk through the story of our composite mortality variable because we think it does a nice job of bringing the discipline of integrated evidence to life. When we limit ourselves to just a single data source ... say, just looking at health records ... we risk portraying a misleading picture of survival outcomes. So misleading, in fact, that these data alone are not fit for purpose when it comes to anything that depends on survival measures.
Roughly 35% of actual deaths are not captured in the EHR structured fields, so imagine as part of a global HTA submission, you made the choice to assess effectiveness of your new medicine by contextualizing results of your single arm trial against a real-world cohort not realizing that the real-world data has an artificially inflated survival rate. You'd be misled to believe that your medicine performs no better than the standard of care or even may appear to perform worse. So through the lens of integrated evidence, we can start to think about the work that needs to be done to unlock use cases like this. In fact, we've worked on several just like this one. On this Kaplan-Meier plot, we're showing the same cohort of patients who plotted based on different sources of mortality data. And at the bottom, we have what's generally considered the gold standard for mortality; the National Death Index. And above, we have the EHR with 35% missingness. It's important to note that the National Death Index suffers from recency and access limitations, so it's most useful as a benchmark tool. As I mentioned a moment ago, creating integrated evidence requires us to think carefully about how disparate data sources are generated. We need to consider their strengths and weaknesses like recency, sensitivity, and inherent biases. And when you do that, we're led to discover facts like the missingness I just shared about structured EHR data; that it overestimates survival by a significant degree. So, we might just then say, "Well, there's other sources of data. Let's just link those to fill the gaps." But if we're doing this right, we have to pause and, again, assess these other sources carefully.
Consider the Social Security Death Index. It seems like a perfectly viable solution to augment EHR data until you look more closely and learn about how these data are generated. The sensitivity of the Social Security Death Index has actually been declining for the past five years for some bureaucratic reasons. If we just jump to blindly link these data without being curious about the policies, procedures, constraints and other biases of how the data regenerated, we'd be setting ourselves up for failure. With this learning, to even further increase the performance of our data, we integrated data from funeral home and obituary websites, which brings us to our next topic: how the data are combined.
In our mortality story, we're combining data to add more information for patients in our network. We can simply link these data sets deterministically with a common patient identifier. In the case of mortality, we ran into a scenario where data may disagree between sources. We needed to develop and test multiple approaches to resolve these conflicts and assess the performance of each of them in turn. In this case, our approach was to sequentially incorporate each potential source and then test a number of different sources and sequences until we hit a performance target relative to the benchmark. And for this story, the result is pretty great. Our composite mortality performs nearly as well as the gold standard for this kind of information, so having done the work to inspect each respective source, how it was generated and characterize the idiosyncrasies of each one and having carefully combined them, we're left with a bit of a new thing. Something that's more than just one plus one equals two.
Using this composite data for research requires special analytic considerations and techniques. For example, we can take the correlation into account based on the performance assessment of our composite mortality endpoint when conducting a comparative analysis. And this work following our integrated evidence approach has earned us the right to say things like this: that Flatiron's mortality variable is fit for long-term survival estimates.
As you'll hear from Somnath in a moment, this approach to develop integrated evidence is a few levels beyond the simple operations of just linking data sets together. It provides a thoughtful map for how we can achieve integrated evidence with the promise to transform observational and prospective research.
With that, I'll pass it off to Somnath, someone who’s enthusiasm for the science I find inspiring, so I hope you will, too. Thanks.
Somnath Sarkar: Thank you, Shane. Hello, everyone. I'm honored to be here with you all today. I've been part of drug development and real-world evidence research for quite some time now and I see first-hand the expanding opportunity of real-world evidence in drug development. At the same time, there are some specific scientific challenges that we need to continue to address and that is where I believe integrated evidence generation will come in.
As you know, real-world evidence, by definition, is generated from real-world data through research methods. In my mind, integrated evidence generation will also go through a very similar, however perhaps somewhat nuanced, path. For example, when we pull data from multiple sources, statistical methods may be required to address certain challenges such as heterogeneity data. Similarly, we require specialized tools and statistical analytics tools, rather, when we work with combining data from prospective and retrospective sources. And we need to be transparent about these approaches via writing protocol, statistical analysis plans and findings need to be shared in peer-reviewed publications. As we just heard from Shane, we have been on the path of generating integrated evidence for quite some time now. Development of our mortality variable being an excellent example of integrated evidence generation early in Flatiron history. For example, again, staying on survival analysis for a second, I'm so excited to share today that recently we published an assessment of our statistical bias called Delayed Entry Bias, which happens when a real-world cohort is overrepresented by long surviving patients. Impact of such bias is critical to evaluate for survival analyses when you are using clinical genomics data.
Examples of such use cases could be in early development studies as well as supporting market access decisions. These types of scientific research are critical in advancing drug development use cases that you see in the slide here. However, you may be now wondering how, exactly, an integrated evidence generation approach is going to enhance research in the future. We must continue the discussion on widening the aperture of integrated evidence in clinical research.
As we know, studies are either observational, which could be prospective or retrospective in nature, or interventional, which are generally called prospective clinical trials. We believe an integrated evidence-based approach enabled by technology will advance observational research as well as help develop novel interventional designs such as pragmatic designs. And this is exactly what Flatiron is developing. We are developing a technology enabled integrated evidence generation platform that could maximize the synergy between real-world data and interventional research.
The goal here is to bridge the divide between routine care and clinical research. This integrated platform allows us to aggregate data collected both routinely and intentionally. We believe this platform will also provide us with an opportunity to achieve novel clinical trial designs. You will hear more about this platform and future opportunities coming from this platform in the context of clinical research during a future ResearchX episode. In particular, as we know, randomized clinical trials have been around for a long time and are a major part of clinical research. Unfortunately, RCTs have not evolved much and remain costly, slow and less inclusive. Today I will share two examples of novel interventional study approaches ... or design approaches ... that are based on an integrated evidence generation approach.
First we'll present an example of our real-world control arm and see how this type of study design could be enhanced using an integrated evidence-based approach. Later I will share an example of our novel hybrid trial design, which seeks to incorporate data from the real world, with the objective of bringing in speed and efficiency to a randomized trial. Also, as you'll see both these novel designs have a goal to match real world patients with clinical trial patients. With the integrated evidence platform that I just shared in the last slide, we could consensually collect limited and additional data on real world patients to increase completeness of cohort for matching. I will touch on this later in my presentation.
I have been involved in both these areas of research for a while now, and I'm so excited to see the recent advancements made and even more excited to share a couple of examples with you today.
First, our example of a real-world control arm. We have seen the tremendous potential of real-world control arms used in various stages of clinical development. Let's think of a scenario, say you are Dr. Louis, a pharma early development scientist. You have a novel, small molecule or an antibody that you believe is going to change treatment for late stage DLBCL patients. These patients, as we know, have a short life expectancy. You want to get the drug to patients as soon as possible. So, you conduct a single arm Phase II study. Or maybe you had difficulty randomizing patients in a control. So now, you have to contextualize outcomes of your single arm trial with some external data. You hope to improve probability of success for go/no go to the next stage of development or even to file for an accelerated approval. You would like to use a real-world database comparator arm called a real-world control because, you know, unlike other comparators, real-world data is contemporaneous.
Also sometimes, there is simply no published data available to compare to. We have seen real-world controls from Flatiron data used for these types of purposes. Now we ask a question to ourselves, what if we could increase the probability of regulatory and technical success of a single arm clinical study by using a Flatiron real-world control arm? So in that context, we'll next discuss a few scientific challenges faced in control arm development and how an integrated evidence based approach could help us make some progress. One key challenge is comparability of endpoints. As we noted, mortality data is well validated. However, for single arm studies, overall survival is not accepted as a primary endpoint. Primarily because, without a randomized control, it is difficult to make any meaningful conclusion from survival outcome. Response rate is used as the primary endpoint in a single arm study. Response rate in a clinical trial is based on tumor measurements from images and has a well-documented scientific approach. Essentially, it provides an answer if a patient responded or not in a single arm study. In the real world setting, Flatiron developed a variable to evaluate response to treatment that is based on electronic health records. We call it real-world response. And used as the endpoint in our real-world control arm. So now, we have got two different endpoints, the imaging-based endpoint response in the single arm study and the real-world response in the control arm.
How do we assess one against the other? Which, I think would be needed before one can use them in a comparative analysis in studies such as real-world control comparison.
The plot on the right here shows the regression analysis of response data from clinical studies and relevant real-world cohorts. We see a high correlation between real-world response and tumor-based response at a study level. This is an excellent and very important observation. However, just showing a correlation between two disparate endpoints across multiple studies may not be sufficient. We need to be able to study the concordance at the patient level and understand reasons for disagreements before we can utilize these endpoints in a comparative analysis.
Now, let's discuss how an integrated evidence-based approach could help in this situation. To understand how real-world response based on clinical assessment compares with response based on imaging from a trial, we need imaging data in addition to the clinical trial data for every individual patient. Flatiron has undertaken a large initiative of combining data from radiology images with EHR data. And from that available combined data, we recently performed a study called "RUBIES" in advanced non-small cell lung cancer patients. This study evaluated concordance between real-world response and tumor measurement-based response at an individual patient level.
As we can see, this is the type of hard science of integrated evidence generation that we need to think about. Also, conducting such a study is a very large undertaking. Let's come to the results. As we can see in the table, the overall agreement rate between these endpoints is robust and may provide us with an adjustment factor that the researcher then could use for contextualizing a response rate from the trial. Moreover, we are able to get more granular characterization of agreement as well as disagreements with the green and the red box shown in the table below. Finally, integrating clinical data with imaging data allows us to get one step more granular, to understand the reasons for discordance. This would not have been possible without the "generate, combine and analyze" approach of integrated evidence.
Now, let's move to another key challenging area in cancer research and in real-world control arm development, namely unmeasured confounding, and the bias it may bring in causal inference. Remember Dr. Louis? She is interested in assessing treatment benefit on outcome. However, there are factors such as age, race, ethnicity, etc, which could impact patients' access to treatment and thereby confounding the benefit assessment of treatment. Now, let's consider age, which is available in the EHR and can be used to perform adjusted analysis. However, in many cases we encounter patient characteristics such as socioeconomic status, or SES we'll call it, that may impact both treatment as well as outcome that are not readily available and given and measured. It is well known that the socioeconomic status of neighborhoods influences access of healthy foods, spaces for physical activity, internet, and transportation.
It is also well known that cancer patients with lower SES have often faced a lack of access to physicians or healthcare resources. However, an unresolved question is to what extent socioeconomic disparity would impact outcomes. This unknown association could lead to unmeasured confounding in a study and could be a major source of bias by analyzing patient data, especially when comparing populations from different heterogeneous data sources. So, before continuing on the SES variable, on this slide I've shared a few examples of patients' characteristics, which are not readily available from the EHR. However, with the use of artificial intelligence and machine learning approaches, we can in fact extract this information to be used for addressing confounding bias. Some of these variables are listed on the slide.
Coming back to socioeconomic status, which is a patient's characteristic, again. That cannot be extracted from EHR using these techniques, requiring a very different approach. So, how do we address this important confounding or mediating factor? This goes to our approach of generating, combining and analyzing. We used a validated scientific approach based on US census data from the American community survey. This is collected every five years. This survey data is also used in National Cancer Institute surveys. This information such as income, education and poverty level for households are available at an area level, not an individual level. And this is to protect patients' privacy. We combine this data to develop social economic status at a patient's neighborhood level.
This plot here is for multiple myeloma patients. The survival plot sheds light on questions we are trying to answer. We see, in multiple myeloma patients with lower SES, a poorer survival, that's on the slide I've put below. There is approximately 11 months of survival advantage from most versus least affluent patients. We used multiple myeloma as an example because this inquiry was triggered by a regulatory interaction. Now coming back to our real-world comparison, this SES data would be generated at a neighborhood level for both the single arm trial patients and the real-world control arm. And then, we can use either a matching approach or a co-variant adjusted approach in a causal inference analysis.
Now, I would like to share another very interesting finding with you. Very exciting, I feel. On your left column, we have the percentage of US population in various SES levels and on the right, we have the same distribution for Flatiron patients. This comparison is performed at the neighborhood level, as I mentioned earlier. As we can see, SES distribution of Flatiron patients are closely representative of the general US population. From this analysis, we see that oncologists in the Flatiron network treat patients coming from broad socioeconomic status, which could be addressing another important challenge that we have encountered, which is called representativeness of Flatiron data. Which is critical in realizing the goal to conduct more inclusive research.
We have been answering this very important question of representativeness of Flatiron data in various ways. We have made various comparisons to US-based cancer patient populations, such as SEER and NPCR. We have published those results. You just saw our recent finding with SES data and Flatiron made significant investments in developing a global footprint to help access local data. You may have seen our recent announcement with NCCHE in Japan, and we are excited with the prospect of answering similar questions with ex-US cohorts. These, I believe, would very much help address the question of representativeness.
Next we'll move to another case study demonstrating an integrated evidence generation approach. However, before I do that, I would like to make a note that I have only shared a few selected challenges and potential solutions through an integrated evidence-based approach in the context of real-world control arm development. There are many other examples of using an integrated evidence-based approach in addressing various other known challenges in observational research.
Let me share with you a novel hybrid control trial design that intends to blur the line between interventional and observational research. In many clinical development programs, being randomized to a control arm, is an undesirable outcome for cancer patients. Hybrid control arm design uses a real-world cohort and an adaptive design approach to reduce the number of patients randomized to the control arm. The design creates an internal statistical benchmark that allows us to validate the level of information that could be borrowed or combined with the control arm of the study.
On the next slide. We are going to deep dive a bit and share our progress. We have conducted extensive simulation studies on this design. Let me share a few findings with you. Overall, we found that as differences in patients' characteristics between cohorts increases, which in turn would increase the bias in outcomes, such as survival, information borrowed from real-world control arm would decrease. Another key learning is statistical power improves by bringing in external data, which is not surprising. However, on the other hand Type I error, or false positive error rate, could increase as well. However, there is good news that we found through the simulation which is that inflation of false positive rate had an upper bound. This would be an important finding to discuss when these designs get discussed with the agency.
As one can imagine, performing our own simulation studies is an excellent start. But one needs to undertake validation studies before this type of novel approach could be used in a prospective clinical trial. And we have started such validation evaluation with our biopharma partners. One of the problems that may arise is that the real-world cohort lacks sufficient information to match with the trial cohort. It could be missing data for a particular important co-variant. This is where we return to the idea of our integrated evidence generation platform that I mentioned earlier. With a platform like this, we could consensually collect limited additional data on real-world patients to increase the relative completeness of the cohort for matching. And this statement applies just as much to our real-world control design, as it does to the hybrid designs. This could be just one of the many benefits of this platform that we discussed.
We have seen high interest and enthusiasm from both sponsors, as well as the agency, in regard to novel hybrid designs. In partnership with pharma, FDA recently launched a new program called Complex Innovative Trial Design, or CIDs. As we can see from this slide, FDA's stated objective of this program is to facilitate the use of CIDs in late stage development. One of the three cases or use cases that was highlighted in the CID program is a hybrid arm design in DLBCL. This is a tremendous advancement from a regulatory standpoint of accepting novel trial designs, such as hybrid. However, we need to note that this particular hybrid design is to borrow information from previously controlled clinical trials. This sets a high bar for hybrid designs, if we plan to borrow information from real-world data, which means, we have work to do. I do believe that our research and development and collaboration with pharma partners will get us there.
As we have just seen, matching real world patients with trial patients and improving trial accrual is a significant progress, as it could bring speed and efficiency in randomized clinical trials. And this particular hybrid design could be a stepping stone toward a future where trial inclusion is broader and trials can be conducted more often in the community setting, meeting patients where they are. We believe the aperture of integrated evidence generation will further expand via Flatiron's integrated evidence generation platform, and bring the possibility of conducting various types of pragmatic designs in the future.
Lastly, I believe the success of integrated evidence generation will require joining the best practices of various different fields and disciplines. In particular, it will require collaboration across multiple stakeholders and disciplines, such as clinical science, clinical operations, biostatistics, epidemiology, technology innovation, and regulatory sciences.
I want to thank you all for attending ResearchX and before we move to Q&A, I would like to pass it back to Shane. Thank you.
Shane Woods: Thanks Somnath. So, we're going to open it up for Q&A momentarily and thanks for all the great questions that are coming in.
First, a couple final thoughts from me. Just a handful of examples we've talked through today, show we can collectively deliver in these three areas for the industry and importantly for patients. And at the end of the day, this is really about an industry moving in a new direction.
We're seeing that the leading life science companies are already starting to invest towards integrated evidence. There's been a shift in leadership focus, evolving development strategies, and new capabilities emerging. We haven't seen any of our biopharma partners unlock the full potential of integrated evidence yet. In part because adoption and change like this takes time and iteration, but also because the data and approaches needed aren't there yet.
So it's important that we're creating the pre- competitive space to come together, like your engagement and participation today. We see this as an opportunity for Flatiron to really lean in with our industry partners, to embrace and enable integrated evidence, and ultimately to meet our collective commitment to patients, which means we've got work to do. Thanks everyone. I'm going to pass it to my colleague, Alex, who's going to moderate the Q&A.
Alex Gorstan: Thanks, Shane. And thanks Steph and Somnath. Even though I get to work with you all every day, I still really enjoy hearing you all talk about this great work. Okay. We've got some good questions for you all. Let's start here. So Shane and Steph, it looks like this one is a good one for you two. Okay. The question is, we've already started to adopt integrated evidence planning as part of our lifecycle evidence needs for each asset at my company. How do you see companies like Flatiron playing a role in our planning? Shane, you want to kick this one off?
Shane Woods: Yeah. Yeah. Happy to start. It's really an important question, and one we've given some thought to. This has been an observation, I guess, at Flatiron as well. We partner with a lot of biopharma companies. What we're really observing within biopharma is this expansive view of approaches to evidence and their planning activities. That's the part I take away from the question around integrated evidence planning, kind of looking across the lifecycle of a molecule, and thinking about new and expanded evidence that can be used in those settings.
Flatiron's role, and you've heard a bunch from us around this today, is around stretching approaches to integrated evidence by broadening the types of evidence used. And then that can be considered from a planning perspective by biopharma. I guess I see this as biopharma, by and large doing a lot of the planning, Flatiron and others in the industry that are contributing to these identified evidence needs and then maybe finally there has been this culmination between biopharma and Flatiron and actually defining a lot of the scientific methods.
This is a great example of where we can come together and collaborate deeply. I know I'm describing these as separate activities, and they are, largely, but they do require a heavy level of coordination. I just want to stress that point. It can't be about each of us going off and doing our own things. This has to be about coordination between these different steps of planning and actually generating the evidence.
Alex Gorstan: Awesome
Stephanie Reisinger: Yeah. Thanks, Shane. I want to just add a couple things. That was great. Just taking a step back, from my perspective, Flatiron can really implement this idea of integrated evidence in unique ways. As we said today, it's not new, but what we can do is a little bit unique. With our point of care integrations, with our access to the healthcare providers and patients, and with our direct partnerships with FDA and the other regulatory agencies and some of the key stakeholders.
And so because of this unique position in the ecosystem, that allows us to help our biopharma partners manage their products, with evidence generated across the product lifecycle from R&D, through commercialization. And then one more thing, as Somnath shared today, it also makes us a unique partner. Really uniquely positioned to be a thought partner, as our industry continues to push forward and embrace the possibilities of real-world evidence and integrated evidence. Thanks, Alex.
Alex Gorstan: Yeah, you got it. Okay. Another good question. Shane, I'm going to pick on you again, and maybe Somnath can add some color here. How does Flatiron tend to address collection of curated real-world data from imaging, for example, radiology and pathology?
Shane Woods: Yeah. Yeah. Happy to start. I definitely will look to Somnath on the more technical side, but let me just start with the high level in terms of what we're doing with imaging today. I want to tie this back to the framework a little bit of: “generate, combine and analyze”. I think this story really starts with generate. Flatiron's investing pretty heavily in radiographic images. Now, the challenges, the images aren't actually captured in the electronic health record.
A bunch of this investment that we're making is actually on the operational side, just to collect the images themselves and basically get them ready for use. Where we started using images, I guess first, has been around endpoints, progression, response, just thinking about solid tumors for a second, as an example. We started there, developing those endpoints based on clinician notes and radiology reports, but we recognized that we want to go further.
So the radiographic images are a big part of that, because we're able to leverage those to glean more information, similar to how we would maybe see in a clinical trial with RECIST, for example. I think the other part of this is radiomics maybe, if I'm interpreting the question right, which is obviously a growing space and another application for images. Maybe I'll kick it over to Somnath, and get his perspective on those two areas.
Somnath Sarkar: Thanks, Shane. Yes. Happy to jump in here a little bit. I think it's a great question. I was just reading a paper in JCO Clinical Cancer Informatics a month or so ago. I think the radiomics as well as cancer pathology are identified as two areas where computational medicine is converging with physician medicine. So I think this is a really new and novel area for cancer research in general. And as you said, I think Flatiron is definitely invested heavily in the scans and the images, and reading them for future advancement.
For example, looking at the scans and figuring out what kind of tumor volume measurements we can figure out. And that could be a really important covariant, which I was describing in my presentation earlier that could help us understand further about these unmeasured confounding and so on and so forth. Similarly, the pathology, which comes from digital, digitizing that and reading that is also in our thoughts and in our future goals, but more to come I'm sure. And then we can communicate as we get to some of those areas. Thank you.
Alex Gorstan: Great. Thanks, Somnath. There are a lot of fantastic questions coming in, and unfortunately I think we really have time for one more to address live with the group here. Somnath, maybe this is a good one for you here. So the question is, or the comment and question is, very exciting to see the stratification across socioeconomic status quintiles. In what other ways is the representativeness of Flatiron's real-world data assessed?
Somnath Sarkar: Oh, yes. So thank you. So that's also a really important question. The reason it is important is because we do different real-world evidence-based studies. One question that has come up is how do you contextualize the data? Hence the importance of investigating the overall representativeness of the population. However, it's a challenging and important area. We have done, as I have noted earlier, that comparison with SEER and NPCR and a few other datasets.
One of the key challenges that we see is that not everything is collected everywhere similarly. For example, Flatiron data could have endpoints that SEER wouldn't have. And also they're registries, and ours is more EHR. So fundamentally there are some differences. But having said that, we do believe that we started the process of looking into various different datasets and trying to compare and contrast, and be transparent about when we are actually presenting data and so on. I'll just stop there for now. Thank you.
Alex Gorstan: Okay. Awesome. Thank you for those thoughtful responses and thanks to everyone who joined to take time out of your day for this important discussion. So that wraps it up for our first episode. As a reminder, we've got more episodes taking place over the coming weeks.
Next up, we'll be going deeper on the topic of multimodal data, which is a concept within the sphere of integrated evidence. We're really looking forward to having you all join us. And finally, since I mentioned before, we weren't able to get to all of your questions. Please know the lines of communication remain open even after we end this episode, please feel free to get in touch with us at rwe@flatiron.com. Thanks. See you next time.
