Popular Searches:


    How we think about carrying evidence across country borders

    Last updated: April 11, 2022

    • Brennan Beal headshot

      Brennan Beal, PharmD, MS

      Quantitative Scientist, Flatiron Health

    The rapid emergence, and subsequent uptake of oncology real-world data (RWD) has ushered in a new age of cancer treatment — one where we are beginning to support real-time decision-making and empowering clinicians to be more efficient than ever before; however, in this new age we’ve found ourselves asking an age-old question: does the data we are using to help make these decisions truly reflect the patient in front of us? The question is more pressing when called to answer for the well-being of an entire population. Can the experiences of US patients tell a reliable story applicable to patients from other countries?

    For health technology assessments (HTA), the question of data relevancy is at the forefront of decision-making. In these cases, decision makers are called to decide how well a given drug may perform in the real world, relative to the next best thing, and what a reasonable cost is for the added benefit for an entire population. So, ensuring that the population of interest is reflected in the data at hand is a crucial part of their process.

    To illustrate the task, and it’s accompanying challenges, consider the fictional country of Transportugal, who has been presented with real-world evidence (RWE) from the US to support a label expansion for a product in their country. Though the US study was conducted with sufficient rigor, Transportugal’s subject matter expert is quick to point out some immediate differences between their country and the US, listed in Table 1, which cast doubts on the external validity of the study’s results.

    Table 1: Differences in Cancer Care between the US and Transportugal*

    United States


    Drug of interest is administered 35% in community, 65% in academic centers.

    Drug of interest is administered 100% in a community setting

    3 drugs available for treatment, with the drug of interest accounting for 30% of market share

    2 drugs available for treatment, with drug of interest accounting for 50% of market share

    Patients usually have limited access to supportive care

    Provides supportive care for all cancer patients

    *Transportugal is a fictional country used for exemplification

    From the differences in Table 1, we can begin to understand why making decisions with evidence external from the population of interest is difficult. Don’t we expect differences in the site of administration to affect outcomes? And if the drug in question only accounts for 30% of the market share in the US, compared to the 50% in Transportugal, aren’t the cohorts fundamentally different?

    In reality, the list of difficulties go on:

    • Do the guidelines and treatment pathways differ from country-to-country?
    • Are the same backbone therapies available to both groups of patients?
    • Are the frequency of visits and diagnostic assessments comparable?

    Questions, such as those provided above surrounding the relevance of US RWD to ex-US countries, are questions of uncertainty in the transportability (Figure 1) of inferences. Can the insights generated from the US transport to another country and allow for expedited decision-making and promote health equity both within the US and beyond?

    Illustration of NSCLC EDM

    Figure 1. Illustration of the Flatiron NSCLC patients as a sample of the broader US NSCLC population; Relevance of Flatiron RWE to the TRANSPORTUGAL NSCLC population requires assumption of transportability.

    Though there is debate in the literature, we can likely never say for sure whether a given inference is transportable. We can never know with certainty whether the group of patients for which we intend to make a decision will have the same outcomes in another setting, such as Transportugal,* as they do in the US even with the same technologies1. A treatment provided to those with extensive caregiver support may realize more health gains than the same treatment provided to someone forced to provide for themselves; however, the data will view them equally. That is to say that in the data, those two patients are equal (eg, binary). In the data, the treatment was taken, or it was not, which makes transporting the true weighted effects difficult.

    This is to say that with the results in-hand, one may find themselves pressed to retrofit them to a given target population. So then, our hope lies in the prespecification of our analysis with the goal of a representative population in mind. To help with this, we at Flatiron have created a list of considerations for the representativeness of US data for HTA decision-making (Table 2).

    Using this framework, we can rethink how the study was pre-specified to prevent any undue anxieties thereafter. At a bare minimum, we can become more transparent about possible limitations to create an environment of trust, rather than the skepticism demonstrated by our fictional subject matter expert. For example, we probably could have identified differences in market share from the beginning of the analysis and knowing this may have changed how we constructed our cohort — to ensure those with access in transportugal are represented in the US EHR data. Or, we may have considered differences in site of administration and opted for a hierarchical model to handle these discrepancies.

    Ultimately, there are a variety of ways that a target population may vary from that represented in any given dataset. The hope is that with the help of the considerations below, we can construct cohorts that are fit-for-purpose, identify populations similar to ex-US populations, and at the very least, be cognizant of the limitations that exist as we continue to push for a future where the full potential of RWD can be realized.

    Table 2: Considerations for the Representativeness of US EHR Data for HTA Use Cases

      Patient Characteristic Differences
    Baseline demographics Demographics may encompass a set of effect-modifying variables - differences in the prevalent and incident population should be considered
    Prevalence of disease The baseline prevalence of a given disease may affect the transportability of some elements based on the mathematical association with relevant endpoints
    Preference for modifiable risk-factors Preferences, and thus the prevalence, for modifiable risk factors (smoking, obesity, etc.) within a given population may modify the transportability of outcomes between countries if these risk-factors are known effect modifiers.
    Biomarker prevalence For cancers with a diverse genetic etiology, there may exist significant treatment effect heterogeneity. Therapies indicated for those cancers may lack transportability in populations with a widely different biomarker makeup. Further, because biomarker testing rates may differ between populations, those selected into the cohort may also differ and affect the transportability of outcomes.
      Treatment Differences
    Access to a given treatment A prevalent population may not be represented in the EHR data given restriction in access based on socio-economic or variability in payer preferences for a given product. Thus, patients selecting into a given cohort could vary and impact observed outcomes.
    Access to supportive care Supportive care is known to improve outcomes for patients in many settings; however, access to supportive care varies within and between countries.
    Market share of the pharmaceutical(s) of interest and competitors Environments with a large diversity of available technologies for a specific indication require contextualization for who selects into a cohort treated with a specific technology.
    Market share of backbone therapies used concomitantly with a therapy of interest Even in situations where the market share for a technology of interest is the same, concomitant therapies of interest (eg. high versus low dose dexamethasone) may differ. If these therapies are effect-modifying, the distribution of them in the given data will affect transportability of the outcomes.
    Guideline differences between jurisdictions / localities Because the approved label /reimbursement criteria for a given therapy may vary, the way a product is used between countries may also sometimes differ, which may present itself in what is known as the compound treatment problem. Further, labels may also influence the preceding drugs that patients have been exposed to, complicating the question of transportability.
      Setting Differences
    Treatment site variation It is thought that outcomes between academic/research institutions may be different than those observed for community practices based on available resources. Thus, if countries differ in the distribution of these treatment sites for a given cohort of interest, outcomes would also be expected to differ.
    Differences in time-to-treatment initiation within a disease’s natural history Time-to-treatment initiation may vary dramatically between countries (driven by localitied procedures to confirm diagnosis and/or healthcare system capacity) and therefore change a particular risk set and influence outcomes
    Disease assessment frequency Disease assessment frequency can provide erroneous conclusions about metrics such as progression free survival or other outcomes that rely on monitoring schedules, and thus the time at which observations can be made.
    Preference for end of life care In later lines of therapy, the risk-set a country chooses to treat may be different from that of another country based on differences in preferences for hospice. So, countries that tend to treat more aggressively may treat a sicker risk-set than that of a country that is more likely to choose for alternative end-of-life remedies


    Notes: This table is intended to be a dynamic, living tool that will change over time. As Flatiron Health develops more learnings from experience with HTA use cases, and transportability nuances that arise, this tool will continue to be updated.