And I think the Food and Drug Administration and other parts of the government actually realized right away
that they were sitting on essentially a treasure trove of information, because this was an entirely new
disease and one that required an immediate response.
This is The Real World, in Practice. A podcast by Flatiron Health. I'm Alex Gorstan. Today, we sit down
with Nicole Mahoney, senior director of regulatory policy at Flatiron and our chief medical officers for
our provider and research teams. That's Bobby Green and Mike Vasconcelles. At the center of their
discussion is the realization that COVID-19 is having a major impact on the practice of oncology. This
is leading to downstream impacts on researchers who rely on real-world data. And as policymakers
scrambled to leverage data against the growing pandemic, major health policy challenges are coming into
Now in the discussion today, we're going to hear a lot about real-world data. And if you're not already
familiar with the term, real-world data refers to information gathered routinely outside of clinical
trials. And in our case, we're really talking about detailed information gathered at the point of care.
Information that's proving increasingly valuable in the fight against cancer.
Okay. Let's dive in. Here's Bobby kicking things off. And just for background, in addition to being the
chief medical officer for our provider team, Bobby is also a practicing community oncologist. Okay
There is a sense in a strange way that COVID has almost been this sort of seminal moment for real-world
evidence. And so maybe just to start, what can real-world evidence do to really help elucidate insights
As I mentioned before, Nicole is our senior director of regulatory health policy at Flatiron. She came to us
from Merck where she was global regulatory policy director.
So when we think about what real-world evidence can do, I think it's important to say first and foremost, it
can and is filling critical gaps in what we understand about this disease. So I think of things through the
regulatory policy lens and early, very early on in the pandemic, FDA worked with a group of stakeholders,
pharmaceutical companies, real-world data providers, and others, academics to think about what questions can
we ask. And there was really an acute need at the time to understand the natural history of the disease. So
that's been a real help to have real-world evidence address those things. So that means, who's getting
COVID? Who requires ventilation? What are the outcomes for these patients and what treatments are they being
exposed to? And are any of those treatments working? Do they appear to be working? And so the information
can also be used in a whole variety of ways, including the management and allocation of resources.
This is Mike speaking here. He's our chief medical officer on the research side of the business. Also a
practicing oncologist, Mike spent lot of his career leading the development of oncology therapeutics across
a number of well known life science companies.
So Nicole, what's your view of just the immediacy of the access to the data? In terms of those questions
that are being asked? I mean, to me that's really one of the potential real opportunities that a real-world
data set can provide.
That's absolutely correct. And I think the Food and Drug Administration and other parts of the government
actually realized right away that they were sitting on essentially a treasure trove of information. Because
this was an entirely new disease and one that required an immediate response. And I think that real-world
evidence can fill the gaps because it can be collected in essentially real time and shared out in real time.
Or at least it has the potential to do that.
I was just going to say Nicole, the flip side to that is, on the one hand we have electronic health records.
The data is easier to pull out than it would have been 10 years ago or 15 years ago. But at the same time,
the sort of world we envisioned with interoperability where large hospital systems all have EHRs that talk
to each other and so you could easily analyze this data at scale, without a ton of work on the backend. We
haven't really gotten to that point yet right?
Okay. Here's a good spot to make sure we're all on the same page here. Interoperability. So this is
basically the idea that different health information systems should be able to work together and share
information seamlessly. It's easy to imagine a world where primary care physicians, specialists, and even
mobile apps on a patient's phone share the same picture of the care being delivered. And it seems like
that's the way things should work. But as you heard from Bobby, we're not there yet. But still there were
major opportunities right in front of us early on in the pandemic. Opportunities for real-world data to
influence patient care. The off label use of drugs like Hydroxychloroquine, and Dexamethasone being great
examples. Real-world data could enable physicians to make decisions faster, differently, or more
confidently, but like most things in healthcare there are both benefits and limitations to consider. Let's
hear Nicole speak to some of those nuances.
While there is a lot of promise in real-world evidence, I think this has laid bare a lot of the limitations
to the way our current healthcare data ecosystem is actually structured. Even for things like public health
and getting reporting out from different organizations. There are a whole set of technological difficulties
in pulling information from different healthcare providers or healthcare centers. But there are, at the same
time policy issues. Issues that make it difficult or unclear on how and when information can be shared, for
example. So I think you're absolutely right. There are multiple layers to the issues that you've raised.
One thing that may be a shock to no one is that the healthcare sector is slow to adopt new technology. Now
you might be thinking, if there ever was a time to accelerate health policy objectives like
interoperability, now would be the time. Is that possible given the industry's track record?
My 2 cents, I think there are certain things that have been accelerated by this. The perfect example, being
telehealth. A lot of the restrictions and billing around telehealth was made easier. Which was an initiative
by CMS, which was incredibly important, which I think we all hope will continue. And I think helped in care
delivery across the countries, especially in the cancer space. I will take the cynics view that I'm not sure
what can actually truly speed up interoperability beyond the relatively slow pace that's been going on. Just
because I think it's a huge and complicated problem to tackle, which requires more than CMS changing billing
codes for a certain thing. Just my perspective, but I'm skeptical that it will. I hope I'm wrong, but I'm
skeptical that it will.
And Bobby building on the rapid adoption of caring for patients in unique ways, with a telehealth approach,
immediately had to figure out how to incorporate those approaches to care in your practice. And I'm just
curious, do you see that waning as the acute need to provide access in that way starts to dissipate over the
coming... hopefully in the coming months or years?
There was definitely a novelty component to it. And I think... I mean Mike you'll remember probably the
first day you got your beeper in medical school, you were really excited. And then within a week you
wanted to throw it out the window because you're like "This isn't as cool as I thought." I think there
was a component to telehealth that was like that. The first day you're in clinic doing telehealth,
you're like, "Wow, I can do a video chat with patients. This works really well." I think that there have
been technical and logistical things that have made it a little harder. I think some of the telehealth
companies just weren't ready for the huge flood of people who were going to use it.
So the platform, at least that I've used sometimes gets a little bit clunky, hard to connect, patients
have difficulty connecting. So it's certainly not a panacea, but that being said, there's clearly been a
critical role during COVID. And to specifically answer your question, I think it's going to persist.
There are certainly lots of things that I'm sure you've seen in clinic and that we all see as clinicians
of examples when patients travel to the office for care. And at the end of the day, they probably didn't
need to come in. There are certainly opportunities when people need to come in for treatment, that's
different. When people critically need labs that you need to be able to look at that day, that's
different. There's some times an assessment that you can only do with a physical exam and seeing someone
in person. But there's also a lot of stuff that you can do remotely. And I think still deliver really
good care. And I hope that persists, because in general it makes life easier for patients. And also,
especially if you're in rural areas, areas where it's difficult to get transportation, it certainly
becomes even more critically important.
So, it's safe to say we've seen some modifications to everyday treatment practice. Now, when one of the
tried and true use cases of real-world data is to better understand treatment patterns under normal
real-world circumstances, what are we to make of these data when normal is out the window? Here's Mike.
I think it's fair to say, at this point in time, we're really just working really to understand the impact
of the way in which changes in care delivery are affecting the way in which we can properly ask research
questions, and it goes really beyond sort of a shift or an addition to telehealth visits. We know, for
example, kind of to your point, that access to the clinic in a variety of ways, whether it has to do with
monitoring of care or delivery of medicines to treat cancer, all of that has had an impact because of the
pandemic. That rolls through the research questions that we can ask.
When I think about answering important questions, I think about drug development. I think that it's, for
certain, the COVID pandemic has accelerated a conversation about real-world evidence and accelerated the
thinking around what can be done with it, what appropriate uses are, what type of questions can be answered.
We're even getting into a granular level because data organizations are coming to the table, working through
experiments, trying to answer questions collectively. We're really accelerating the conversation. The jury
is still out, as Bobby was mentioning, on whether or not we'll come to solutions during this. It can be
applied when things kind of go back to more of a normal state. So what works for a case where there's an
acute need for information might not work under more normal conditions.
Depending on where you live in the country, COVID-19 may be impacting life in different ways. When we think
back to mid-March, the country effectively slowed to a standstill. Everything from haircuts to elective
procedures were just canceled. How does that affect routine procedures? The yearly mammograms, for instance,
that just didn't happen?
One of the sort of interesting, outstanding questions, in some ways we've done a natural experiment, right?
We've said, just as an example, what happens if you stop doing mammograms on people for a couple of months,
because elective testing shuts down? And there was a period of time when mammograms weren't getting done,
there's been a period of time where colonoscopies haven't gotten done. And I sort of wonder what are the
opportunities there for real-world evidence to answer questions about one, what the negative impact is. And
there's also a flip side, right? There's probably things that we routinely do in care that we actually don't
need to do and are there things that we stopped doing that aren't going to result in harm?
There is a pretty pervasive view that the impact will be negative, but did you suggest that there may be
silver linings that could be gleaned or at least better understood from this sort of natural experiment?
Yeah, I mean, so let's say, as an example, we saw that not doing mammograms on women between the ages of 40
and 50 actually had no impact, but the only impact we saw was on women over the age of 50. There's a lot of
screening, preventative routine health care that we sort of take for granted that we need to do, but you
could argue that the evidence on impact at least on mortality is questionable. I mean, Mike as you know, we
have lots of people follow up with us in clinic at certain intervals without any data to support it, just
because a bunch of experts say that we should, or this is how we trained 25 years ago. I had my colon cancer
patients come in every X number of months and I did this on them. And who knows what the real impact is
because they're just aren't data.
These types of questions definitely aren't lost on policymakers because we've seen over the past month and a
half, two months, we've seen FDA come up with a new list of questions about exactly what you're describing.
What are the ramifications for different patterns of treatments or missed screening techniques, this is what
policymakers are really interested in. On the flip side, our life sciences partners are asking, "What can we
maybe eliminate from the information that we collect for research purposes? What is extraneous information
that may not be necessary to answer critical clinical questions?" And the interesting thing is this whole
pandemic is really accelerating conversations around that too. We've seen drug development change
essentially overnight, and in response, the policy makers are putting out guidance in a fast and furious
way, continuously updating it almost in real time as they recognize that they can exhibit some flexibility
on certain things. So it'll be very fascinating to see how this plays out over the next couple of years.
So Nicole, just for our listeners, I mean, on the latter point, that's more so directed in the clinical
research realm, correct? Clinical trials and kind of classic development questions.
Exactly, and I think there's a way to bridge what's happening in clinical care to the way questions are
answered and the ramifications for drug development. And I think that's what policy makers are trying to
make the bridge between what is going on in the real world. And then what are the impacts for drug
Well, I think that's a really important point that we talk a lot about the generalizability and the
representativeness of real-world data in the context of research questions and also the kind of
comparability of those cohorts to say clinical trial cohorts.
Generalizability and representativeness are two terms we should pause on for a second. Generalizability
refers to the idea that things that we learn by studying a smaller group are also true for the population at
large. And representativeness, that speaks to the characteristics of the smaller group, that is, is the
makeup of the smaller group, things like race, age, socioeconomic status, is the mix of the traits that you
see in a smaller group, similar to the mix of traits that you see in the real world? For the best kinds of
studies, you've selected a sample that's representative of the real world so that you can reach conclusions
that are generalizable to the broader population.
I think, however, any of us who've been involved in clinical research intuitively know and understand that
those populations that we enroll into clinical trials are not representative of the patients we care for day
over day. And so the extent to which sort of the pragmatic realities of dealing with all of the issues that
the pandemic has presented with respect to sort of the prosecution of clinical trials may, at the end,
suggest that we are over-engineering the way in which we think about patients enrolled in those studies,
we're collecting too much information, we're too prescriptive with respect to eligibility criteria. And in
fact, if we designed clinical studies that were more pragmatic to the patients that we care for, that we'll
end up with those cohorts over time that are more reasonably comparable if we look at a real-world
experience compared to a clinical trials experience. So there is a lot that we're in the midst of learning
that will sort of play out in the time that comes ahead of us.
This is a really interesting idea. Has the quest for scientific rigor led us to a place where we're
collecting too much information or being too prescriptive about trial protocols and is this pandemic leading
us to some realizations about that?
And it feels like, I mean, Mike, you know this space much better than I do, but just like in routine
clinical care, there are things that we do that are important, but there are also things that we do just
because we've done them before, and we don't know how they're important. In sort of the routine of clinical
trials, there are things that we do because it's important for patient safety, but then there may be all
these other things that we do just because we've always done them and do we really need to? An example that
sort of comes to mind is, in the midst of the pandemic, the FDA relaxed some of the requirements on, if
you're on a clinical trial, you don't necessarily have to go to the site where you enrolled on the trial to
get an assessment, you could go to a clinic closer to home to get lab work done or something like that
because they didn't want people traveling. Couldn't that have happened sort of without the pandemic in a lot
of circumstances, right?
That's what the hope is that some of these lessons learns will give the experience and confidence to change
the way we approach clinical trials for drug development.
Nicole, you posted an article internally, I think just this morning, talking about how sort of in the midst
of everything that's going on, and work from home, and clinical trial accrual, which in some cases,
unfortunately, has dropped off, that we can talk about, the FDA has continued to review oncology drugs and
push through at breakneck speed on getting drugs approved. Just curious on your perspective there on what's
been happening at the FDA with oncology drug approvals.
I think it's been remarkable. I don't think they've missed a beat, and I think they've pioneered a few
different things, such as a virtual ODAC meeting, an advisory committee meeting. And I think that's a
testament to their commitment to making sure that everyone is working towards getting patient access to
effective therapies, when possible. We've certainly seen that in our collaboration with the FDA. We have a
research collaboration with them. Not only are we keeping up on that collaboration and the joint work that
we do, but we're talking about answering a couple of new projects. And so it's been fascinating. I think
that they're running a tight ship. It's been very efficient. And as you saw on the article that I posted,
the oncology folks are really being singled out, along with the antivirals group, as really over-performing
and not missing any deadlines during this time period.
Yeah. I just want to echo that observation. I mean, it's nice to see the metrics coming forward, but sort of
anecdotally, over the last few months, I've been candidly, repeatedly impressed on both fronts that you
alluded to, Nicole.
Yeah. I think in that space, and then you can sort of look in clinical care, both in the cancer
community and the medical community in general, people sort of stepping up and realizing people still
need care throughout this, whether it's people in intensive care units on ventilators with COVID or just
cancer patients getting cared for in a community or academic practices across the country.
I think the one area where there's been, my perspective, I'm curious your guys, a lot of criticism has
been specifically around the ability and probably worse in the U.S. than other places, the ability to
quickly run randomized clinical trials to answer important questions about COVID. Rob Califf, the former
FDA commissioner has been out a lot saying it's just been a really lost opportunity that we can't, in
these very large hospital systems where we have patients, run randomized trials. As you guys know, there
were so many observational trials of hydroxychloroquine, thousands of patients treated off label, not
that many patients treated on clinical trials. The Decadron study, which showed some benefit to
dexamethazone was not run in the U.S. You know, I think Rob Califf would argue that we should have an
infrastructure that allows us to do sort of lightweight, sort of pragmatic like trials in a crisis like
this, and we just don't have it. Curious your take on that.
It's another opportunity in a way, right? I think probably the criticism has some validity, but then the
question is how do you seize on that or build on that, to take advantage of ecosystems either in large
academic medical centers or through other sort of consortiums that are built around research questions,
hopefully not for the next pandemic, but for other sort of emerging sort of critical issues that inevitably
will confront us in all kinds of ways.
Yeah. I think it's been very interesting. I listened to a podcast this morning where someone was talking
about vaccine development and said, we're in the phase where the troops are storming the beach with all
these different solutions. And it seems like that's what's happening in the U.S. So we have tons and tons of
clinical trials getting off the ground. That's a good thing, you want multiple shots on goal. On the other
hand, we probably don't have the central organizing force in the way that maybe some other countries have to
kind of deploy all the resources in a more systematic way. So opportunity's there, but it also highlights
some limitations, that we discussed before about our infrastructure for conducting studies.
Yeah. It feels like it is an opportunity. I mean, there was just, you guys may have seen a STAT article by a
Matthew Herper I think yesterday or the day before, and Dr. Califf was quoted. I just wrote it down here
saying, "If the goal was to optimize the likelihood of figuring out the best treatment options, the system
is off course." So I think we do have some work to do there.
Okay. Shifting gears a little bit. There's no doubt that clinics have been affected by COVID. We released
data in the cancer letter back in May that brought this to light. Now we're six months into the pandemic.
What's the new normal like for a clinician?
I would say there are still challenges. It's not fun walking around an office all day wearing a mask,
but you do it because that's the way to do it. I would say, and part of this is, from data that we've
seen across our network and part of it is my own anecdotal experience still practicing, to a certain
extent things have gotten remarkably back to normal. We've seen early on in March. No one really knew
what to do, but practices seem to be pushing forward and coping pretty well.
One of the things that's sort of been a nice surprise is we've not seen lots of clinicians get sick.
We've not seen, to the best of my knowledge, lots of staff in the offices get sick or patients get
infected from coming into the office. And again, I think clinics across the country are doing a good job
of universal mask wearing, not, generally, keeping guests out. Yeah. I mean, I think, both in community
and academics and hospital systems, we've seen things get back to some semblance of normal, which is a
good thing, so.
How are you balancing delivery of care with anything structural around social distancing? I mean, are you
trying to manage that proactively and in any sort of way with respect to scheduling or volume of people in
the clinic at any one time, or are you just really relying on masks and delivering care as you normally
would six months ago.
So it's both. It's trying to keep people out of the waiting room and limiting visits, so the waiting
room, which in pre-pandemic times can often be crowded. It's not crowded anymore. There's plenty of
space in there. I mean, as you know, oncology patients tend to come in with lots of people with them and
limiting that has, I think been impactful. Making sure that there's enough spacing in the infusion room.
I think a lot of infusion rooms are set up that way to begin with, others are not, but practices have
made attempts to increase the social distancing there.
And there's a nontrivial percentage of patients who are still getting telehealth, which is dropping the
volume of people coming into the office. I will say there are parts of a telehealth visit these days
that feel more intimate than sitting in an exam room where both you and your patient are wearing masks
and trying to maintain social distance as well. In fact, I've even had a couple patients where I've
thought the physical exam isn't critical, where I've said, come in, get your labs, and then we can have
a nice conversation over telehealth because it's honestly a much easier conversation. It's amazing how
as you're talking to patients, when you're both wearing masks, like not having the visual cues of
knowing if someone's smiling or not smiling is just some basic things like that are really, really hard,
even in person.
I'm wondering how, as the only non-clinician here, how are practices using real-world data to inform their
setups or what they do, or are they doing that?
I would say, the areas where people have used real-world data has mostly been right now, Nicole, from an
operational standpoint. And in aside from to the extent that real-world evidence has contributed to
understanding how we treat COVID patients, I think there's been observational data that has shown that
treating cancer patients during the pandemic has not been adverse, that you can safely get patients
chemotherapy, and bad things aren't happening because of that. So I think that sort of informed the
Other than that, I think it's mostly been around operational insights to practices. Like one of the
things we've done is provided volume data, both to our individual practices and benchmarking so they can
understand is the volume drop that they're seeing different than everywhere else, would help them
operationally prepare. Mike, I don't know if you've seen anything different across academic centers or
with anyone that you've interacted with.
No, I think it's similar, but I do wonder if we've seen real benefit derive from some of the network based
sharing of data in the context of this situation. This goes to the future state, we're talking about the
future state as it relates to an individual patient's care. But does your practice management organization,
Bobby, are they yet clamoring for that network-wide perspective going forward on any kind of regular basis
to help them manage their practices? Or is it-
So I think one of the first and most notable things that happened at the beginning of the pandemic was
people were craving understanding what their colleagues across the country was doing because there was panic
and this was completely uncharted territory. So I think there was really this need to want to network and
understand what was going around elsewhere, not just in your own practice. And I think that ultimately this
only accelerates the need to do that, not just in the midst of a pandemic, but in general, I think there's
So you do think it'll translate into some something that could be longitudinally beneficial to practices?
I do. I think so. And I think it's... I mean the community oncology in general does do a pretty good job of
networking and keeping in touch and sharing best practices. And I think that this serves to prove the value
of that. And to the extent that organizations like ours and others can make it easier to share information
and get insights from that. I think that's critically important.
And listen, if there's a world where it was very easy to share, we're continuing to treat patients with XYZ
and people are doing fine, we're continuing to do this and people were doing fine, that information
ultimately is, as you say, Mike, looking at a future state would be super valuable, I think.
I'm just curious as we think about the pandemic and delivery of health care, Nicole, what keeps you up at
So that's a great question. And I think what keeps me up at night regarding real-world evidence in terms
of the information we're producing is the issue about trust and credibility. And I think that keeps
other organizations up at night too. And I say that because there's this huge need for information and
for getting information quickly. And the flip side of putting out research papers to answer questions
very quickly is that maybe they're not thoroughly vetted. And I think we're at a critical juncture where
if we want to increase the credibility and trust in real-world data, we have to make sure that as
community we're publishing reliable results. And I think what keeps me up at night is we may not have
best practices down that everyone agrees with, but the good news is I see that coming.
So since the pandemic started, the Reagan-Udall Foundation, on behalf of FD1A, has put together what
they call a COVID evidence accelerator. And it's a huge multi-stakeholder group of data organizations,
drug companies, academics, all talking every single week about how we can use real-world data to answer
critical questions in the time of COVID. And just last week, there was a whole discussion about best
practices to increase trust in real-world data. And I think that we'll have to follow through on those,
but that is what keeps me up.
Bobby, what keeps you up?
Other than that bars are open and people are still going to bars, and it will play out and we will
eventually learn what the impact was. But I do worry that things that happened specifically around delay in
certain screening or people delaying in seeking care in March and April, not just for cancer related things,
but for all things that we're going to find out that, that had a big impact on mortality and outcomes that
we are going to recognize down the road. And it's just going to become part of the impact in lives loss due
to COVID. I mean, you've seen the data. The number of appendectomies went down, the number of heart attacks
went down, the number of strokes went down. Well, they didn't go down, people just didn't get them taken
care of. And ultimately what's going to be the impact of that. How about you, Mike?
I think what I'd add to that is just the challenge that it looks like we're going to continue to face, in
the US in comparison to other parts of the world, that I think really speaks to some really interesting
cultural and maybe structural issues, we alluded to some of this earlier with respect to the nimbleness with
which we were able to respond cohesively to critical clinical research questions, maybe impeded by some of
the structures that exist here. And I think we're seeing that play out in terms of the way in which we're
managing through the pandemic in a way the individualism and entrepreneurial- ism, that's the bedrock of our
society and our culture think probably having a more complex effect as it relates to cohesive response to a
very pervasive issue. And we're seeing that play out in all kinds of ways. And we're going to struggle
because of it over the next bit of time, but we'll get through it. I have no doubt about that. We'll get
Well said. Thanks, Mike.
Thanks for listening. To learn more at the intersection of real-world data, oncology and health policy,