Emerging Technology

Just over ten years ago, a pivotal piece of legislation, the HITECH act, was created to motivate the implementation of electronic health records (EHR) and supporting technology. Today, with EHR adoption at over 90 percent, we are facing the new challenge of generating real-world data at the quality, completeness and accuracy necessary for the most valuable applications across the healthcare ecosystem.

A major hurdle in surfacing real-word data comes from interpreting the complex clinical context and nuances within a patient’s health record. Due to their expertise, trained clinicians are best suited for this. However, as researchers seek data on rarer cohorts, statistical techniques like machine learning are helping to scale data curation efforts by humans. These same techniques are also emerging as valuable tools for researchers as they explore novel approaches to increase data quality and completeness, and to identify meaningful patterns in large sets of data. Together, these tools are proving essential for realizing the promise of real-world data.

Learn more about emerging technology in RWE

Machine Learning and Real-World Evidence

Article

Model-assisted cohort selection with bias analysis for generating large-scale cohorts from the EHR for oncology research

A team at Flatiron developed a technique that combines machine learning and natural language processing with human review to build large-scale research cohorts without sacrificing quality. See how it was applied to the selection of metastatic breast cancer patients.

Video

The Role of Machine Learning & NLP in Real-World Evidence
Exclusive Content the-role-of-machine-learning-and-natural-language-processing-in-real-world-evidence 45:37

Flatiron team members discuss the evolving role of machine learning and natural language processing (NLP) in real-world evidence along with appropriate applications for these technologies. Learn more about machine learning and AI.

Journal Article

Opportunities and challenges in leveraging electronic health record data in oncology
Berger, M. L., et al. (2016). Future Oncology, 12(10), 1261–1274.

Two case studies help to show both the opportunities and challenges with extracting structured and unstructured data from EHRs.

Read Journal Article
Are you a current Flatiron data user looking for dataset documentation and methodological guidance?
Visit the Customer Knowledge Center

Data Quality

As researchers begin to draw on real-world data to answer challenging research questions, the importance of the underlying data quality comes into focus. Unlike data sourced from clinical trials, real-world data are typically recorded in support of a patient’s treatment plan rather than according to a particular research protocol. This means that key insights may lie scattered across annotated radiology reports or physicians’ notes.

Without careful effort to process and verify these data, real-world data may be subject to gaps and biases, rendering the data unfit for use. To succeed with real-world data, researchers must first understand the quality challenges, impacts, and what measures must be taken to assure fit-for-use quality in the context of their research questions.

Learn more about real-world data quality

Real-world data and the case of the missing deaths

Article

Real-world data and the case of the missing deaths

Researchers working with EHR-derived real-world data (RWD) must be able to trust that the data they’re analyzing are telling an accurate and reliable story. What many researchers don’t realize, however, is that approximately 35 percent of actual deaths are missing from structured EHR fields. See how a team at Flatiron is tackling this problem.

Video

The Promise of RWE & A Regulatory-Grade Quality Checklist
the-promise-of-real-world-evidence 14:59

Flatiron Co-Founder Zach Weinberg opens the 2018 Flatiron Research Summit sharing his perspectives on the promise of real-world evidence. Watch his talk and see the quality checklist.

Video

Leveraging Real-World Evidence for Regulatory Use
leveraging-real-world-endpoints-for-regulatory-use 54:01

Watch thought leaders throughout the industry discuss current and future uses of regulatory-grade RWE at the 2018 Flatiron Research Summit. Learn more about regulatory applications.

Endpoints

With the advent of clinically deep, longitudinal real-world data, the prospect of establishing regulatory-grade real-world endpoints is a promising next step in the journey for researchers seeking to translate real-world data into real-world evidence.

The challenges of developing reliable, scalable and meaningful endpoints from real-world data are many. As researchers consider the applicability of real-world endpoints for their research questions, it is important to understand the data sources that are used to create these variables and the processes that are followed to evaluate whether they are considered fit for purpose.

Learn more about real-world endpoints

Video

Real-World Data vs the Gold Standard: Enhanced Validation Analysis of a Composite Real-World Mortality Endpoint
Exclusive Content Real world data vs the gold standard 45:32

Learn ways in which incomplete survival data can lead experts astray as well as how Flatiron has overcome this challenge by going beyond the EHR to develop a composite mortality variable. Watch the webinar.

Journal Article

Generating real-world tumor burden endpoints from electronic health record data: Comparison of RECIST, radiology-anchored, and clinician-anchored approaches for abstracting real-world progression in non-small cell lung cancer.
Griffith, S. D., et al. (2019). Advances in Therapy, 36(8), 2122-2136.

This study evaluates different methods to collect outcomes from clinical reports available in the EHR to better understand disease progression, including comparisons to RECIST.

Read Journal Article

Journal Article

Characterizing the feasibility and performance of real-world tumor progression end points and their association with overall survival in a large advanced non–small-cell lung cancer data set
Griffith, S. D., et al. (2019). JCO Clinical Cancer Informatics.

If real-world evidence is to become a key component of our clinical data portfolio, developing a framework to characterize progression and progression-free time/time to progression based on EHR-derived data sources is critical.

Read Journal Article

Video

An Intro to Real-World Endpoints in Oncology
Exclusive Content real-world-endpoints-oncology 07:10

Discover how real-world endpoints are enabling new approaches and how Flatiron is partnering with regulatory and industry stakeholders to develop and evaluate them. Learn about how real-world endpoints are derived in oncology.

Video

The Life Science Perspective on Real-World Endpoints
Exclusive Content life-science-real-world-endpoints 08:43

Hear from Mitch Higashi, PhD, Vice President, Health Economics and Outcomes Research at Bristol-Myers Squibb, give his perspective on real-world endpoints. Watch his presentation.

Journal Article

Real‐world progression, treatment, and survival outcomes during rapid adoption of immunotherapy for advanced non–small cell lung cancer
Khozin, S., et al. (2019). Cancer. 125(22), 4019-4032.

In this study, researchers from the FDA and Flatiron Health applied their method to assessing disease progression in real-world cancer datasets to a large cohort of patients with advanced NSCLC treated with PD-1/PD-L1 inhibitors: nivolumab, pembrolizumab, and atezolizumab.

Read Journal Article
Exclusive Content regulatory-real-world-endpoints 13:49

Video

Regulatory Perspective on Real-World Endpoints with Sean Khozin

Sean Khozin, formerly of the FDA’s Oncology Center of Excellence, discusses the challenges as well as the progress in generating reliable endpoints from real-world datasets at the 2018 Flatiron Research Summit. Learn more about his perspective.

Journal Article

Development and validation of a high-quality composite real-world mortality endpoint
Curtis, M. D., et al. (2018). Health Services Research, 53(6), 4460–4476.

In the absence of a national mortality dataset that is recent, accessible and linkable, we’ve developed a composite mortality variable, amalgamated from multiple data sources, and benchmarked it to understand its quality.

Read Journal Article

Video

Real-World Endpoints Discussion With Industry and FDA
Exclusive Content real-world-endpoints-panel 14:01

A panel of experts convene to discuss the state of real-world endpoints and balancing scientific rigor. Hear how key stakeholders are thinking about real-world endpoints.

Journal Article

An exploratory analysis of real-world end points for assessing outcomes among immunotherapy-treated patients with advanced non-small-cell lung cancer
Stewart, M., et. al. (2019). JCO Clinical Cancer Informatics, 3(3), 1–15.

Researchers from six organizations collaborated to develop a common protocol that aligned definitions and methods for real-world data drawn from administrative claims, cancer registries and/or electronic health records.

Read Journal Article

Methodology

The approaches for generating evidence from clinical trials are well established. Randomized experiments simplify causal inference. Data quality and completeness are supported by the prospective nature of data collection with tailored data capture systems. Patient outcomes are evaluated according to widely accepted measurement protocols to provide meaningful and consistent study endpoints.

These affordances are not granted to studies of observational data. The use of retrospective real-world data is potentially challenged by its lack of randomization in addition to its quality characteristics. Yet, as biostatisticians advance the development of specialized analytical techniques and incrementally demonstrate progress, the promise of real-world evidence for decision-making comes further into focus.

Learn more about real-world evidence methodology

Journal Article

Comparative effectiveness from a single-arm trial and real-world data: alectinib versus ceritinib
Davies, J., et al (2018). Journal of Comparative Effectiveness Research, 7(9), 855-865.

In the absence of comparative evidence from RCTs, external controls can be used to bridge the gap using direct adjusted comparisons, which may lead to systematic variation or bias. Innovation in methods of gathering and assessing comparative evidence is needed.

Read Journal Article

Journal Article

Small but mighty: The use of real‐world evidence to inform precision medicine
Miksad, R. A., et al. (2019). Clinical Pharmacology & Therapeutics, 106(1), 87-90.

This study seeks to define quantitative and qualitative methodologic and analytical tenets by which small-cohort RWE can contribute mightily to precision medicine research and treatment.

Read Journal Article

Regulatory & Policy

As real-world evidence (RWE) plays an increasing role in healthcare decisions, regulators and health technology assessment groups are working to optimize their use of RWE. Driven by both legal mandates and a desire to improve access to prescription drugs and other medical products, their expertise continues to evolve as they, along with the rest of the scientific community, gain further experience in the assessment of RWE.

For researchers seeking to utilize RWE, it is critical to remain informed of the guidance and precedence these groups establish through published research, partnerships, guidance documents, and in feedback on submissions.

Learn more about the use of RWE for regulatory decision-making

real-world-evidence-in-regulatory-decision-making 53:50

Video

The Role of Real-World Evidence in Regulatory Decision Making

Flatiron’s former CMO & CSO, Amy Abernethy, and Dr. Rick Pazdur of the FDA discuss the agency’s engagement with real-world evidence, the path forward for RWE, and the future of cancer care at the 2018 Flatiron Research Summit. Hear Rick Pazdur's perspective.

Blog

Do FDA alerts impact practice patterns?
FDA-alert-effectiveness

What's the impact of an FDA alert and label change on clinical practice? We leveraged EHR-derived real-world data to evaluate an alerts effectiveness in changing the direction of immunotherapy and chemo prescribing trends. Read more on the Flatiron blog.

Video

Regulatory Perspective on Real-World Endpoints with Sean Khozin
Exclusive Content regulatory-real-world-endpoints 13:49

Sean Khozin, formerly of the FDA’s Oncology Center of Excellence, discusses the challenges as well as the progress in generating reliable endpoints from real-world datasets at the 2018 Flatiron Research Summit. Learn more about his perspective.

Applications

The uses of real-world data are as varied as the sources from which the data are derived. The same goes for the setting in which they are applied – from translational/early clinical development through post-market applications. Objectively, the most valuable applications are those that accelerate the development of and access to lifesaving therapies.

Researchers across the world are pushing boundaries, establishing new precedents and sharing learnings as they start to see more of the value of real-world evidence contributing directly to many drug development activities.

Learn more about applications of real-world evidence

Journal Article

Organ dysfunction in advanced melanoma patients treated with immune checkpoint inhibitors
Spillane, S., et al. (2020). The Oncologist.

Researchers from the National Cancer Institute and Flatiron Health used real-world data to examine treatment patterns, toxicity and clinical outcomes of immune checkpoint-inhibitor treated patients with advanced melanoma and organ dysfunction.

Read Journal Article

Journal Article

Adoption of immune checkpoint inhibitors and patterns of care at the end of life
Riaz, F., et al. (2020). JCO Oncology Practice.

To further understand the association between FDA approvals of immune checkpoint inhibitors and treatment patterns at the end of life, researchers utilized real-world data to examine treatment patterns at the EOL for patients with advanced melanoma, non-small cell lung cancer or microsatellite stable colon cancer.

Read Journal Article

Journal Article

Association between FDA label restriction and immunotherapy and chemotherapy use in bladder cancer
Parikh, R. B., et al (2019). JAMA, 322(12), 1209-1211.

This paper examines the impact of an FDA alert and label change on clinical practice using real-world data obtained from electronic health records.

Read Journal Article

Journal Article

Real‐world progression, treatment, and survival outcomes during rapid adoption of immunotherapy for advanced non–small cell lung cancer
Khozin, S., et al. (2019). Cancer, 125(22), 4019-4032.

In this study, researchers from the FDA and Flatiron Health applied their method for assessing disease progression in real-world cancer datasets to a large cohort of patients with advanced NSCLC treated with PD-1/PD-L1 inhibitors: nivolumab, pembrolizumab, and atezolizumab.

Read Journal Article

Journal Article

Affordable Care Act (ACA) Medicaid expansion impact on racial disparities in time to cancer treatment
Adamson, B. J. S., et al. (2019) Annual Meeting of the American Society of Clinical Oncology (ASCO). May 31 - June 04, 2019.

This abstract sought to look at the correlation between Medicaid expansion under the Affordable Care Act and racial disparities in timely cancer treatment using data collected from EHRs.

Read Journal Article

Journal Article

Real-world outcomes of patients with metastatic non-small cell lung cancer treated with programmed cell death protein 1 inhibitors in the year following U.S. regulatory approval
Khozin, S., et al (2019). The Oncologist, 24(5), 648-656.

Co-authored with the FDA, this study analyzed a large and recent real-world cohort to estimate real-world survival following treatment with PD-1 inhibitors, nivolumab and pembrolizumab, in the first year following regulatory approval for NSCLC.

Read Journal Article