HEPV Real-World Evidence Forum

RWE in the Context of Payer Decision Making

April 20, 2021 | 1 pm – 4 pm ET, Virtual

On April 20, 2021, MDIC’s Health Economics and Patient Value (HEPV) Initiative and FDA hosted a virtual public forum on real-world evidence (RWE) in the context of payer decision-making. More than 200 participants including representatives of FDA, CMS, the device and diagnostics industry, patient advocates, and other organizations working with real-world data (RWD) and RWE attended the Public Forum.

Participants learned about CMS-FDA collaboration on real-world data (RWD), examples of RWE used by the FDA in regulatory decisions, the perspective of CMS and other stakeholders on RWE, and several examples of the use and possible use of RWE in the development of medical technology. This summary provides a brief overview of each presentation and discussion, with links to the video of all presentations and discussions and slides from most presentations.

How to View the RWE Public Forum

View the presentations and discussions on our full Forum video or follow the links provided below for individual sessions.

View or download the slides.

Download the agenda

Pamela Goldberg, MBA

President and CEO, MDIC

“This is clearly an important topic for all of us in today’s environment,” says Pamela Goldberg, MBA, President and CEO of MDIC, in welcoming participants to the Forum. Goldberg presented an overview of MDIC. A public-private partnership created to advance regulatory science of medical devices for patient benefit, MDIC brings together government agencies, the device and diagnostics industry, and patients. “Our ultimate goal is to solve problems on behalf of patients,” says Goldberg.

MDIC has:

  • 63 participating member organizations
  • 300+ subject matter experts involved in working groups
  • 70+ resources available to download in a digital resource library
  • 20+ active working groups and committees

HEPV is the newest of MDIC’s four initiatives. The other initiatives are Clinical Science, Data Science and Technology, and NEST Coordinating Center (NESTcc). NESTcc aims to accelerate the development of new and safe health technologies, including by leveraging RWE.

Click here to watch this presentation starting at 0:05.

Jo Carol Hiatt, MD, MBA

Vice President, Health Economics and Patient Value, MDIC

“MDIC’s Health Economics and Patient Value initiative builds on an area I’m passionate about: bringing technologies to patients as quickly as we can with careful review of the opportunities to gather and use RWE to advance coverage and regulatory decisions,” says Hiatt. HEPV also focuses on incorporating patient perspectives and risk tolerance in coverage and payment decisions.

Two working groups provide guidance on evidence requirements that would be a resource for payers:

  • Patient Perspective Research (PPR) Working Group, chaired by Jeff Farkas
  • Real-World Evidence (RWE) Working Group, chaired by Barry Liden

The PPR Working Group works with payers, health technology assessment organizations, and others to enhance the understanding and incorporation of patient risk tolerances and preferences into clinical trial design and selection of opportunities.

The RWE Working Group, the sponsor of this Forum, is working to enhance the use, acceptability, and incorporation of RWE. “Clinical trials have gaps in what they can accomplish. Things change once a technology is in a different environment with different populations and different providers,” says Hiatt.

View the slides. 

Amy Abernethy, MD, PhD

Principal Deputy Commissioner

Acting Chief Information Officer

U.S. Food and Drug Administration

“Modern RWD has tremendous potential to accelerate the generation of high-quality and actionable evidence for FDA and CMS decision-making and promoting access to important new technologies,” says Abernethy. CMS and FDA have been working together to align their thinking on RWD and RWE and to promote rapid access and evidence development. For example, they used outcomes data from the TVT (Transcatheter Valve Therapy) Registry in clinical care, coverage, and regulatory decision-making.

Rapid development of RWD capabilities in the private technology sector may lead to RWD approaches that provide evidence that is equal to or perhaps better than evidence developed using current approaches. The COVID-19 Evidence Accelerator, developed at the FDA’s request in partnership with the technology sector and data providers, provided tremendous insights and knowledge about the value of RWD developed through sustained collaboration with public and private experts and stakeholders.

Defining how best to leverage RWD to inform regulatory and payment decisions will require sustained dialogue involving CMS and FDA, as well as product developers, clinicians, patients, payers, and other stakeholders. Abernethy noted that the MDIC RWE Public Forum is an important step in the necessary dialogue.

“We at FDA very much look forward to continuing the dialogue between CMS and FDA to align our thinking on how best to generate additional evidence, particularly for breakthrough and other high-value technologies,” says Abernethy.

Click here to watch this presentation starting at 17:28.

Lee A. Fleisher, MD

Chief Medical Officer and Director

CMS Center for Clinical Standards & Quality

 

Fleisher agreed with Abernethy that CMS and FDA are working together to determine how RWD and RWE can be used to obtain FDA market approval and CMS coverage. “It’s important for us to have clarity about what we need to make decisions,” says Fleisher, who noted that

RWE will help both FDA and CMS better serve patients.

“FDA and CMS will find a successful path forward through RWE and real-world decisions. We are working closely to find the right balance of not compromising evidentiary standards while investigating new, less burdensome ways of collecting evidence for FDA market approval and CMS coverage,” says Fleisher.

Along with clinical data, claims data, databases such as the Surveillance, Epidemiology, and End Results (SEER) Program for cancer statistics, and in the future, data from personal devices, may be used in decision-making about safety, efficacy, and reasonable necessity.

“Our challenge today is to leverage lessons learned through coverage with evidence development and not ignore the technological advancements to find easier ways to utilize RWD for both CMS and FDA,” says Fleisher.

Click here to watch this presentation starting at 23:20.

Felipe Aguel, PhD

Deputy Director

Office of Clinical Evidence and Analysis

Office of Product Evaluation and Quality

Center for Devices and Radiological Health, FDA

 

In March 2021, FDA published Leveraging Real World Evidence in Regulatory Submissions of Medical Devices, with about 90 examples of regulatory decisions made by CDRH based on RWE from FY2012-2019. “We wanted to encourage expanded use of RWE for regulatory decisions and provide a single place for internal and external stakeholders to see and learn from past experience,” says Aguel.

The report covers a variety of submission types (i.e., 510(k), De Novo, PMA, and HDE), data sources (e.g., registries, medical records, claims, and device-generated data), and purposes (e.g., new authorization, indication expansion, and post-market study). Examples include the full continuum of clinical and device areas including:

  • Three examples of digital health devices, which validated the use of RWD from Software as a Medical Device
  • Two examples using patient-generated data
  • Nine examples using device-generated data
  • One example of a clinical trial embedded in a registry

“Building on learnings from this effort and others, CDRH will continue to evaluate how RWD and RWE can be used to efficiently inform regulatory decision-making,” says Aguel.

Relevance, reliability, and data quality are critical components of CDRH review of submissions that rely on RWE. The degree of relevance is largely dependent on the context of use:

  • Informing clinical study design
  • Substantial equivalence
  • Pre-market benefit/risk determinations
  • Post-market surveillance

Poor quality data are not likely to be usable for decision-making, regardless of relevance. Well-defined standards for RWD quality are critical. FDA is working with MDIC’s NESTcc and other stakeholders to develop metrics, maturity models, and standards for data quality.

Click here to watch this presentation starting at 30:10.

View the slides.

Read the FDA report Leveraging Real World Evidence in Regulatory Submissions of Medical Devices.

Steven Farmer, MD, PhD

Chief Strategy Officer

Coverage & Analysis Group

Centers for Medicare & Medicaid Services

 

“CMS views innovation as holding tremendous promise to improve beneficiary health and aims to bring proven technologies to patients as efficiently as possible,” says Farmer. Along with the CMS view on innovation, Farmer presented the center’s value mission of evidence-based coverage, statutory authorities, and an example of how CMS considered RWE in making a National Coverage Decision on Transcatheter Edge-to Edge Repair (TEER) for functional mitral regurgitation.

Evidence about devices can change long after FDA market authorization as more is learned about them and as ancillary treatments are developed. RWE can close at least some of the gaps between FDA’s safe and effective requirements for market authorization and CMS’s reasonable and necessary requirements for coverage. However, the RWE must meet rigorous methodologic standards.

In evaluating TEER for functional mitral regurgitation, CMS reviewed two randomized controlled studies (RCTs), which had conflicting results, 25 observational studies, 3 meta-analyses, and expert guidelines and consensus documents. The centers also participated in a National Institutes of Health expert meeting on the treatment of functional mitral regurgitation. RWE provided insight into the discrepant RCT findings. CMS decided to cover TEER under the Coverage with Evidence Development paradigm and required that all patients be entered into a registry to determine which patients benefit most and the long-term outcomes.

“CMS looks forward to working with FDA and other stakeholders to identify RWE applications that close the evidence gaps while preserving the quality of evidence that CMS requires,” says Farmer.

Click here to watch this presentation starting at 48:00.

John Whitney, MD

Vice President, Medical and Reimbursement Policy
Anthem, Inc.

Anthem, a large private payer with Medicare Advantage plans, Medicaid plans, and commercial group plans in 14 states, doesn’t differentiate between RWE and other evidence. The company bases coverage decisions on medical necessity and looks for:

  • Credible scientific evidence
  • Peer-reviewed literature
  • Proof of improvements in the net health outcome
  • Items and services that are at least as good as any established alternative
  • Improvement outside an investigational setting

“RWE has a lot of opportunity to enhance some of the scenarios we encounter where a device has been FDA-approved or cleared and the evidence still has gaps,” says Whitney. Currently, however, published evidence about RWE is often lacking or has bias or unsound methodology. It is often difficult to determine whether the evidence shows clinically-meaningful outcomes.

The Anthem subsidiary that publishes RWE treats these research questions like RCTs, with committee review of topics, suggestions, development of formal hypotheses, and appropriate research methods. To move the use of RWE in devices forward, “We need a common language with strong and well-defined standards about what good looks like. That will help us identify clinically-meaningful outcomes,” says Whitney.

Click here to watch this presentation starting at 1:12:00.

Jo Carol Hiatt, MD, MBA

Steve Farmer, MD, PhD

Sebastian G. Schneeweiss, ScD, MD, ScD, MS

Chief, Division of Epidemiology

Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine
Brigham and Women’s Hospital

Professor of Medicine and Epidemiology

Harvard Medical School

Bernard Black, MA, JD

Nicholas J. Chabraja Professor of Finance

Northwestern University, Pritzker School of Law and Kellogg School of Management

This technology example used a hypothetical implanted artificial pancreas from ACME Medical Devices. After Hiatt described the device, evidence from randomized controlled trials (RCTs), and FDA labeling, Farmer highlighted CMS’s perspective and gap analysis on the device. Then Schneeweiss and Black presented the possible application of RWE to close the evidence gap and address some of CMS’s issues. A summary of each part of the hypothetical technology example is presented below.

Click here to watch this presentation starting at 1:30:00.

Part 1: ACME Device Company Implanted Artificial Pancreas

Jo Carol Hiatt, MD, MBA

Approximately 1.6 million Americans have Type 1 Diabetes and require insulin, which is challenging to manage. The hypothetical implanted artificial pancreas represents a major advance in diabetes management and quality of life. It addresses lifestyle factors, improves care, and simplifies insulin management.

Two multicenter RCTs showed that the artificial pancreas performed extremely well compared to self-administered insulin. FDA approved the hypothetical artificial pancreas for people age 16 and over based on these RCTs.

Click here to watch this presentation starting at 1:30:00.

View the slides.

Part 2: CMS Gap Analysis

Steve Farmer, MD, PhD

Key CMS considerations when reviewing a new device for coverage are whether the device belongs to an applicable benefit category and meets CMS’ reasonable and necessary definition, based on a thorough review of the available evidence. If the device is reasonable and necessary, it may be eligible for a national or local coverage determination. If it does not meet the reasonable and necessary threshold but is promising, CMS identifies the evidence gap and has covered some of these devices through the Coverage with Evidence Development pathway.

The hypothetical implanted artificial pancreas has a benefit category. CMS did an environmental scan of the two pivotal clinical trials and other published evidence and determined that the evidence did not meet the reasonable and necessary threshold. Important gaps to be closed before CMS could make a decision to cover the device were the need to:

  • Demonstrate hard outcomes and patient preferences
  • Demonstrate device durability and health outcomes for at least 5 years
  • Provide data that are more generalizable to Medicare beneficiaries

Click here to watch this presentation starting at 1:38:00.

Part 3: From Real-World Data to Real-World Effectiveness in Medical Device Research

Sebastian G. Schneeweiss, MD, ScD, MS

“Well-reported RWE studies can be reproducible and valid,” says Schneeweiss, who highlighted work underway at Harvard Medical School to build evidence for and support reproducibility and validity in RWE and presented a proposed study design to fulfill CMS’s gap analysis on the hypothetical artificial pancreas.

In Schneeweiss’ proposed artificial pancreas CMS gap analysis study, the study population in the RCT is people aged 65+ with Type 1 difficult-to-control diabetes. In the RWE study, people aged 65+ with Type 1 and 2 difficult-to-control diabetes would be included. Identifying people with difficult-to-control diabetes based on claims data would be tricky. While it is possible to use claims data to identify the exposure (artificial pancreas) and comparison (insulin) groups for the RWE study, insulin dosing data are not available. Major adverse cardiovascular events (MACE), including cardiovascular death, end stage renal disease or declining renal function, and device failure and removal are all captured in claims data. Self-efficacy is not captured in claims data and an interview sub-study before and after device implantation would be necessary for the RWE study.

Analysis would start the day after implantation and continue until an event, death, or five years of follow up for both the RCT and the RWE study. RCT data would be analyzed using intent-to-treat analysis and Cox proportional hazards regression analysis. Quality checks would be protocol registration and audit trails for both the RCT and the RWE study. For the RWE study, a standard software platform would be used for transparency.

Study design issues are the main challenge to conducting valid non-interventional RWE studies such as a CMS gap analysis for the hypothetical artificial pancreas. The risk of errors can be reduced by contemplating a trial with the key characteristics related to causal inference.

Click here to watch this presentation starting at 1:44:00.

View the slides. 

Part 4: Real-World Evidence for Medical Device Safety and Efficacy

Bernard Black, MA, JD

“Studying RWE requires big datasets going beyond registries to claims data and EHR data and including a control group of people who didn’t get the device,” says Black. He noted that the U.S. is “terrible at data consistency across time and datasets and at linking datasets such as claims and EHRs.” Linked, quality datasets should be developed as healthcare research infrastructure, with leadership from CMS, FDA, and CDC. Government datasets should be free or very low cost to academic researchers.

If good data existed about the hypothetical artificial pancreas, a regression discontinuity design could be used to conduct a real-world observational study that would link claims data, EHR data, mortality data, and demographic data. CMS might say it wants a big trial with everyone age 68 to 73 to be eligible and BMI between 25 and 30. Results will be valid beyond the ages and BMIs studied, but not too far outside of these variables. Crude randomization could be done using the first letter of the last names. Researchers can also use crossover design and analyze time variation.

Black recommended that device manufacturers design their observational study up front, using the appropriate study design for the specific device. Causal inference researchers can help design the study. Device manufacturers should register the study and allow the study to be replicated by independent researchers chosen by FDA or CMS.

Click here to watch this presentation starting at 2:06:00.

Part 5: Technology Examples Discussion

Bernard Black, MA, JD

Steve Farmer, MD, PhD

Jo Carol Hiatt, MD, MBA

Sebastian Schneeweiss, MD, ScD, MS

 

During this discussion, the speakers commented on the presentations about the hypothetical artificial pancreas and answered questions from the audience. Topics covered included:

  • The applicability of a RCT in the real world
  • The role of RWE in device development
  • Causal inference designs in peer-reviewed literature
  • The importance of replicating and publishing RWE studies

Click here to watch this presentation starting at 2:20:00.

 

Panel/Roundtable Discussion

Steve Farmer, MD, PhD

Jo Carol Hiatt, MD, MBA

Eleanor Perfetto, PhD

Interim CEO
National Health Council and Executive Vice President for Strategic Initiatives

Flora Sandra Simmi, MPH

Senior Vice President, NESTcc

Sean Tunis, MD, MSc

Senior Advisor, FDA

During this panel discussion, the panelists discussed the use of RWE in the development of devices. Topics covered included:

  • Data standardization and quality
  • The role of MDIC’s NESTcc in providing access to RWD from multiple health systems and data sources
  • The importance of including the patient perspective about the disease and the treatment when developing and interpreting RWE studies
  • RWD for therapeutic devices compared to diagnostics
  • Use of RWE for coverage decisions on breakthrough technologies after FDA approval

 

Click here to watch this presentation starting at 2:31:00.

View the slides.

Jo Carol Hiatt, MD, MBA

There is tremendous opportunity to continue to improve RWE data quality and methods and align decision-making between FDA and CMS. “We are hoping to continue to sponsor this kind of conversation going forward to help CMS and other payers be as familiar as possible with the various data sources that may be relevant to the decisions they may be making,” says Hiatt.

“What Data Can’t Do” by Hanna Fry (The New Yorker, March 22, 2021) had intriguing comments relevant to this Forum:

“We’re left to find a happy medium between two extremes exposed by the philosopher of science Abraham Kaplan: ‘If you can measure it, that ain’t it,’ and by Frank Knight, founder of Chicago School of Economics, ‘If you can’t measure it, measure it anyway.’”

She also noted a comment by environmental economist James Gustave Speth:

“We tend to get what we measure so we should measure what we want.”

Click here to watch this presentation starting at 2:51:00.