Real-World Evidence: Data Quality, Use Cases & More

Emerging Technology

A major hurdle in surfacing real-word data comes from interpreting the complex clinical context and nuances within a patient’s health record. Due to their expertise, trained clinicians are best suited for this. However, as researchers seek data on rarer cohorts, statistical techniques like machine learning are helping to scale data curation efforts by humans.

These same techniques are also emerging as valuable tools for researchers as they explore novel approaches to increase data quality and completeness, and to identify meaningful patterns in large sets of data. Together, these tools are proving essential for realizing the promise of real-world data.

Flatiron team members discuss the evolving role of machine learning and natural language processing (NLP) in real-world evidence along with appropriate applications for these technologies. Learn more about machine learning and AI.

Model-assisted cohort selection with bias analysis for generating large-scale cohorts from the EHR for oncology research

Article

A team at Flatiron developed a technique that combines machine learning and natural language processing with human review to build large-scale research cohorts without sacrificing quality. See how it was applied to the selection of metastatic breast cancer patients.

Data Quality

As researchers begin to draw on real-world data to answer challenging research questions, the importance of the underlying data quality comes into focus. Unlike data sourced from clinical trials, real-world data are typically recorded in support of a patient’s treatment plan rather than according to a particular research protocol. This means that key insights may lie scattered across annotated radiology reports or physicians’ notes

Without careful effort to process and verify these data, real-world data may be subject to gaps and biases, rendering the data unfit for use. To succeed with real-world data, researchers must first understand the quality challenges, impacts, and what measures must be taken to assure fit-for-use quality in the context of their research questions.

How do missing deaths in real-world data impact overall survival analyses?

Research Summary

Carrigan, G., et al. An evaluation of the impact of [...] missing deaths on overall survival analyses of advanced non-small cell lung cancer patients conducted in an electronic health records database. Pharmacoepidemiol Drug Saf. 2019;28(5):572-581 view less

The quality of real-world evidence depends on the robustness and reliability of endpoints such as overall survival. So how does missingness of information impact OS?

Read our summary

Endpoints

With the advent of clinically deep, longitudinal real-world data, the prospect of establishing regulatory-grade real-world endpoints is a promising next step in the journey for researchers seeking to translate real-world data into real-world evidence.

The challenges of developing reliable, scalable and meaningful endpoints from real-world data are many. As researchers consider the applicability of real-world endpoints for their research questions, it is important to understand the data sources that are used to create these variables and the processes that are followed to evaluate whether they are considered fit for purpose.

Learn ways in which incomplete survival data can lead experts astray as well as how Flatiron has overcome this challenge by going beyond the EHR to develop a composite mortality variable. Watch the webinar.

Generating real-world tumor burden endpoints from EHR real-world data

RESEARCH SUMMARY

Griffith, S. D., et al. Generating real-world tumor burden endpoints [...] from electronic health record data: Comparison of RECIST, radiology-anchored, and clinician-anchored approaches for abstracting real-world progression in non-small cell lung cancer. Adv Ther. 2019;36(8):2122-2136. view less

This study evaluates different methods to collect outcomes from clinical reports available in the EHR to better understand disease progression, including comparisons to RECIST.

Read our summary

Endpoints: Real-world progression and its association with OS

RESEARCH SUMMARY

Griffith, S. D., et al. Characterizing the feasibility and performance of real-world tumor [...] progression end points and their association with overall survival in a large advanced non-small cell lung cancer dataset. JCO Clinical Cancer Informatics. view less

If real-world evidence is to become a key component of our clinical data portfolio, developing a framework to characterize progression and progression-free time/time to progression based on EHR-derived data sources is critical.

Read our summary

Discover how real-world endpoints are enabling new approaches and how Flatiron is partnering with regulatory and industry stakeholders to develop and evaluate them. Learn about how real-world endpoints are derived in oncology.

How rapid adoption of IO impacted outcomes in the real world

Research Summary

Khozin, S., et al. Real-world progression, treatment, and [...] survival outcomes during rapid adoption of immunotherapy for advanced non-small cell lung cancer. Cancer. 2019;125(22):4019-4032. view less

In this study, researchers from the FDA and Flatiron Health applied their method to assessing disease progression in real-world cancer datasets to a large cohort of patients with advanced NSCLC treated with PD-1/PD-L1 inhibitors: nivolumab, pembrolizumab, and atezolizumab.

Read our summary

Sean Khozin, formerly of the FDA’s Oncology Center of Excellence, discusses the challenges as well as the progress in generating reliable endpoints from real-world datasets at the 2018 Flatiron Research Summit. Learn more about his perspective.

Assessing Flatiron's composite real-world mortality endpoint

RESEARCH SUMMARY

Curtis, M. D., et al. Development and validation of a high‐quality [...] composite real‐world mortality endpoint Health Serv Res. 2018. ;53(6):4460-4476. view less

In the absence of a national mortality dataset that is recent, accessible and linkable, we’ve developed a composite mortality variable, amalgamated from multiple data sources, and benchmarked it to understand its quality.

Read our summary

An exploratory analysis of real-world end points for assessing outcomes among immunotherapy-treated patients with advanced non-small-cell lung cancer

Journal Article

Berger, M. L., et al. (2016). Future Oncology, 12(10), 1261–1274. view less

Researchers from six organizations collaborated to develop a common protocol that aligned definitions and methods for real-world data drawn from administrative claims, cancer registries and/or electronic health records.

Read journal article

Show 8 more in Endpoints

Methodology

The approaches for generating evidence from clinical trials are well established. Randomized experiments simplify causal inference. Data quality and completeness are supported by the prospective nature of data collection with tailored data capture systems. Patient outcomes are evaluated according to widely accepted measurement protocols to provide meaningful and consistent study endpoints.

These affordances are not granted to studies of observational data. The use of retrospective real-world data is potentially challenged by its lack of randomization in addition to its quality characteristics. Yet, as biostatisticians advance the development of specialized analytical techniques and incrementally demonstrate progress, the promise of real-world evidence for decision-making comes further into focus.

Comparative effectiveness from a single-arm trial and real-world data: alectinib versus ceritinib

Journal Article

Davies, J., et al (2018). Journal of Comparative Effectiveness Research, 7(9), 855-865. view less

In the absence of comparative evidence from RCTs, external controls can be used to bridge the gap using direct adjusted comparisons, which may lead to systematic variation or bias. Innovation in methods of gathering and assessing comparative evidence is needed.

Read journal article

Rare populations: RWE and the future of precision medicine

RESEARCH SUMMARY

Miksad, R. A., et al. Small but mighty: The use [...] of real‐world evidence to inform precision medicine. Clin Pharmacol Ther. 2019;106(1):87-90. view less

This study seeks to define quantitative and qualitative methodologic and analytical tenets by which small-cohort RWE can contribute mightily to precision medicine research and treatment.

Read our summary

Regulatory & Policy

As real-world evidence (RWE) plays an increasing role in healthcare decisions, regulators and health technology assessment groups are working to optimize their use of RWE. Driven by both legal mandates and a desire to improve access to prescription drugs and other medical products, their expertise continues to evolve as they, along with the rest of the scientific community, gain further experience in the assessment of RWE.

For researchers seeking to utilize RWE, it is critical to remain informed of the guidance and precedence these groups establish through published research, partnerships, guidance documents, and in feedback on submissions.

Chana Weinstock, MD, of the FDA discusses the use of real-world evidence in oncology drug approvals, the role of external control arms and how COVID-19 is impacting regulatory decision making. Watch her talk now.

Flatiron’s former CMO & CSO, Amy Abernethy, and Dr. Rick Pazdur of the FDA discuss the agency’s engagement with real-world evidence, the path forward for RWE, and the future of cancer care at the 2018 Flatiron Research Summit. Hear Rick Pazdur's perspective.

Using real-world evidence for label expansion: Use of palbociclib in male breast cancer patients

Research Summary

Bartlett, C.H., et al. Real-world evidence of male breast cancer [...] (BC) patients treated with palbociclib (PAL) in combination with endocrine therapy (ET). Poster presented at: 2019 Annual Meeting of the American Society of Clinical Oncology (ASCO); June, 2019; Chicago, IL. view less

The advent of CDK inhibitors such as palbociclib has dramatically changed the way breast cancer is treated. However, the original approvals did not include male patients, an extremely rare patient population.

Read our summary

Applications

The uses of real-world data are as varied as the sources from which the data are derived. The same goes for the setting in which they are applied – from translational/early clinical development through post-market applications. Objectively, the most valuable applications are those that accelerate the development of and access to lifesaving therapies.

Researchers across the world are pushing boundaries, establishing new precedents and sharing learnings as they start to see more of the value of real-world evidence contributing directly to many drug development activities.

Michael Taylor, PharmD, PhD, of Roche/Genentech explores how his team used a clinico-genomic database in support of R&D and discusses a study looking to prospectively collect and link real-world clinico-genomic, imaging, and outcomes data. Watch now.

Razelle Kurzrock, MD, shares her insights on the evolution of precision oncology from the perspective of a practicing physician in a rare tumor clinic. Listen to her talk.

Simon Papillon-Cavanagh, PhD, of Bristol Myers Squibb presents his work on using real-world data to understand the prognostic role of genetic mutations. Watch the recording and access his slides.

Comparative Effectiveness with RWD?

RESEARCH SUMMARY

Ramsey S.D., et al. Using electronic health record data to [...] identify comparator populations for comparative effectiveness research. Journal of Medical Economics. 2020. view less

In comparative effectiveness research, real world use of treatments does not always match practice guidelines or clinical trial inclusion criteria, which means choosing comparator populations involves a tradeoff between matching to replicate “optimized” care similar to a trial-defined population versus “realized” care that is more representative of real-world practice.

This article delves into how to determine the appropriate population when conducting comparative effectiveness research.

Read our summary

Do FDA alerts actually change practice patterns? FDA, Penn and Flatiron use real-world evidence to investigate

Research Summary

Parikh, R. B., et al. Association Between FDA Label Restriction and [...] Immunotherapy and Chemotherapy Use in Bladder Cancer. JAMA. 2019;322(12), 1209-1211. view less

This paper examines the impact of an FDA alert and label change on clinical practice using real-world data obtained from electronic health records.

Read our summary

Organ dysfunction in the real world: Adv Melanoma + immune checkpoint inhibitors

RESEARCH SUMMARY

Spillane, S., et al. Organ dysfunction in advanced melanoma [...] patients treated with immune checkpoint inhibitors. Oncologist. 2020. view less

Researchers from the National Cancer Institute and Flatiron Health used real-world data to examine treatment patterns, toxicity and clinical outcomes of immune checkpoint-inhibitor treated patients with advanced melanoma and organ dysfunction.

Read our summary

A closer look at immune checkpoint inhibitors and end-of-life care

RESEARCH SUMMARY

Riaz, F., et al. The adoption of immune checkpoint inhibitors and [...] patterns of care at the end of life. J Oncol Pract. 2020. view less

To further understand the association between FDA approvals of immune checkpoint inhibitors and treatment patterns at the end of life, researchers utilized real-world data to examine treatment patterns at the EOL for patients with advanced melanoma, non-small cell lung cancer or microsatellite stable colon cancer.

Read our summary

How rapid adoption of IO impacted outcomes in the real world

RESEARCH SUMMARY

In this study, researchers from the FDA and Flatiron Health applied their method for assessing disease progression in real-world cancer datasets to a large cohort of patients with advanced NSCLC treated with PD-1/PD-L1 inhibitors: nivolumab, pembrolizumab, and atezolizumab.

Read our summary

Affordable Care Act (ACA) Medicaid expansion impact on racial disparities in time to cancer treatment

Abstract

Adamson, B. J. S., et al. (2019) Annual Meeting of the American Society of Clinical Oncology (ASCO). May 31 - June 04, 2019. view less

This abstract sought to look at the correlation between Medicaid expansion under the Affordable Care Act and racial disparities in timely cancer treatment using data collected from EHRs.

Read journal article

One year later: Using RWE to look at PDL1 therapies following approval

RESEARCH SUMMARY

Khozin, S., et al. Real‐world outcomes of patients with metastatic [...] non‐small cell lung cancer treated with programmed cell death protein 1 inhibitors in the year following US regulatory approval. The Oncologist. 24(5):648-656. 2019. view less

Co-authored with the FDA, this study analyzed a large and recent real-world cohort to estimate real-world survival following treatment with PD-1 inhibitors, nivolumab and pembrolizumab, in the first year following regulatory approval for NSCLC.

Read our summary

Learn about real-world data and its applications in oncology

Emerging Technology

The Role of Machine Learning & NLP in Real-World Evidence

Video

Model-assisted cohort selection with bias analysis for generating large-scale cohorts from the EHR for oncology research

Article

Are you a current Flatiron data user looking for dataset documentation and methodological guidance?

Data Quality

Real-world data and the case of the missing deaths

Article

The Promise of RWE & A Regulatory-Grade Quality Checklist

Video

How do missing deaths in real-world data impact overall survival analyses?

Research Summary

Leveraging Real-World Evidence for Regulatory Use

Video

Endpoints

Real-World Data vs the Gold Standard: Enhanced Validation Analysis of a Composite Real-World Mortality Endpoint

Video

Generating real-world tumor burden endpoints from EHR real-world data

RESEARCH SUMMARY

Endpoints: Real-world progression and its association with OS

RESEARCH SUMMARY

An Intro to Real-World Endpoints in Oncology

Video

The Life Science Perspective on Real-World Endpoints

Video

How rapid adoption of IO impacted outcomes in the real world

Research Summary

Regulatory Perspective on Real-World Endpoints with Sean Khozin

Video

Assessing Flatiron's composite real-world mortality endpoint

RESEARCH SUMMARY

Real-World Endpoints Discussion With Industry and FDA

Video

An exploratory analysis of real-world end points for assessing outcomes among immunotherapy-treated patients with advanced non-small-cell lung cancer

Journal Article

Methodology

Comparative effectiveness from a single-arm trial and real-world data: alectinib versus ceritinib

Journal Article

Rare populations: RWE and the future of precision medicine

RESEARCH SUMMARY

Regulatory & Policy

Getting Real: Real World Evidence in Oncology Drug Approvals

ResearchXNEW

The Role of Real-World Evidence in Regulatory Decision Making

Video

Kadcyla (T-DM1) in HER2+ mBC: Label Update on LVEF following a post approval commitment – based on RWD

ResearchXNEW

Integrating Clinical & Genomics Real-World Data to Support New Drug Application

ResearchXNEW

Postmarket Real World Data Perspectives: IBRANCE (palbociclib) Males Case

ResearchXNEW

Using real-world evidence for label expansion: Use of palbociclib in male breast cancer patients

Research Summary

Regulatory Perspective on Real-World Endpoints with Sean Khozin

Video

Applications

Use of a Clinico-Genomic Database (CGDB) for R&D

ResearchXNEW

Impacting Complex Genomic/Immune Landscapes: Precision Medicine in the Real-World Cancer Clinic

ResearchXNEW

STK11 and KEAP1 mutations as prognostic biomarkers in an observational real-world lung adenocarcinoma cohort

ResearchXNEW

Comparative Effectiveness with RWD?

RESEARCH SUMMARY

Do FDA alerts actually change practice patterns? FDA, Penn and Flatiron use real-world evidence to investigate

Research Summary

Organ dysfunction in the real world: Adv Melanoma + immune checkpoint inhibitors

RESEARCH SUMMARY

A closer look at immune checkpoint inhibitors and end-of-life care

RESEARCH SUMMARY

How rapid adoption of IO impacted outcomes in the real world

RESEARCH SUMMARY

Affordable Care Act (ACA) Medicaid expansion impact on racial disparities in time to cancer treatment

Abstract

One year later: Using RWE to look at PDL1 therapies following approval

RESEARCH SUMMARY

Sign up for the Research Newsletter