Investing In The Science Of Science: What Medicare Can Teach The NIH About Experimentation

The National Institutes of Health (NIH) today stands at a crossroads. While the agency continues to receive bipartisan support and steady funding increases, the NIH has long faced structural challenges related to the process of grantmaking, the pace of research, and the work of translating discoveries into clinical practice. The COVID-19 pandemic further spotlighted the NIH’s highs and lows. On the one hand, the agency shepherded critical science underpinning diagnostics, therapeutics, and vaccines. Yet, at the same time, the NIH lacked sufficient speed when it came to addressing scientific questions (for example, for Long COVID) and frequently encountered barriers related to data sharing and study redundancy with regards to clinical trials. Coming on the heels of a global pandemic and in the midst of the first national search for a director in more than a decade, the present moment offers a unique opportunity to transform the NIH to meet 21st-century public health challenges.

While many barriers in science policy are often attributed to funding shortages, the challenges plaguing the NIH have more to do with execution rather than expenditures. Indeed, despite the NIH’s dedication to evidence generation for public good, very few of the agency’s core processes are actually evidence-based and were instead developed over time based on internal experience. As a result, today, we simply don’t know the answers to countless fundamental questions about how an agency as influential as the NIH should be operating to most effectively achieve its mission. For example, which is superior: person-based or project-based funding? What is the ideal time frame for grants? What is the best way to incentivize high-risk high-reward research? Despite profound implications, these questions are largely unexplored. If the NIH were able to try out more innovative ideas and test them, it might be able to substantially accelerate the pace of scientific innovation.

This problem isn’t unique to the NIH; a similar conclusion—that we have so much more to learn about the best way to fund and nurture science—was reached by a recent international panel on scientific funding commissioned by the Canadian government. Answering these core science policy questions will require the NIH to apply the same rigor it applies to traditional scientific research.

How, then, can the NIH “turn the scientific method on [itself]”? Historically, the agency has experimented by using its Common Fund to pilot new funding streams, such as the High-Risk, High-Reward Research Program. However, this mechanism is likely not applicable to performance improvement initiatives, as the laws governing the agency are clear that NIH dollars should go toward scientific research, not necessarily research on science. Furthermore, one-off pilot projects are likely insufficient to compel the agency to embrace a full-scale transformation of its grantmaking and program evaluation practices.

Instead, the NIH needs a mandate to reinvent itself, and the authority to use the agency itself as a laboratory for experimentation.

Consider The Center For Medicare And Medicaid Innovation

Although this concept might seem foreign to science policy makers, there is actually a rich tradition of using experimentation to drive performance improvement in US government. The most prominent example today is the Center for Medicare and Medicaid Innovation (the Innovation Center), which was established by the Affordable Care Act. The law empowered the Innovation Center to waive traditional health care regulations to conduct experiments on the health care system. The law also appropriated the agency with $10 billion in renewable funding each decade to design, implement, and evaluate payment and delivery reforms. Experiments that meet specific criteria regarding the cost and quality of care can then be endorsed by the secretary of Health and Human Services for permanent implementation within our national health care policies and programs.

Between 2010 and 2020, the Innovation Center launched 54 demonstration models encompassing almost one million clinicians and 26 million patients nationwide. Such experiments varied in their design, from randomized controlled trials to prospective studies. For example, the Million Hearts Cardiovascular Disease Risk Reduction Model enrolled nearly 400,000 patients in a randomized trial evaluating the effects of a new risk assessment tool for cardiovascular disease. Nearly one in five dialysis centers in the United States participated in the Innovation Center’s Comprehensive End-Stage Renal Disease Care Model, which incentivized care coordination and achieved more than $200 million in savings while reducing hospitalizations by 3 percent over a five-year period.

As another example, consider Medicare’s Comprehensive Care for Joint Replacement Model, which required hospitals in randomly selected metropolitan statistical areas to participate in a specialized payment model for hip and knee replacement surgeries. Although not every Innovation Center experiment was successful, several have delivered substantial savings (for example, accountable care organizations) and improvements in patients’ health outcomes (for example, the Medicare Diabetes Prevention Program), while demonstration models with null findings have helped inform subsequent experiments and reforms.

Notably, the Innovation Center’s demonstrations also yielded insights to support process improvements in health policy and health services research. For example, as demonstration models expanded, the voluntary nature of participation raised concerns about the risk of selection bias, leading policy makers to explore the use of mandatory models. Beyond program operations, the advent of demonstration models also generated a substantial trove of data, which are available for independent analysis by academics and outside entities, whose evaluations have in turn helped to spur discussion on critical aspects about model design (for example, benchmarking). Furthermore, data sharing and performance measurement in turn has helped promote process improvements at the provider level, particularly around the use of health information technology. Taken together, the Innovation Center’s work has not only catalyzed innovation in health care delivery but has also fostered innovation in how academics, providers, policy makers, and other key stakeholder groups approach the challenge of health care transformation in the first place.

Lessons For The Future Of The NIH

We believe Medicare provides a model for how the NIH could approach an effort to improve the science of science. Here are two key lessons from the Innovation Center’s experience for policy makers to consider.

First, experimentation in government will be more successful with a statutory mandate that is at least somewhat insulated from the political winds. The Innovation Center had the authority to experiment outside the bounds of traditional regulations; had funding that was insulated from the politics of the annual budget process; and had a clear process for translating the outcomes of experiments into changes in health care policy. For an analogous “Center for NIH Innovation” to experiment with new approaches to science funding (for example, peer-review processes), Congress should give the agency new authorities to waive normal processes. For example, while federal law requires the NIH to conduct peer review, an innovation center could experiment with giving the NIH program officers more authority to override peer review, or to award grants that have only one strong vote of support, or to conduct limited lotteries as a tiebreaker.

Likewise, given how the outcomes of science research—such as changes in health care spending and outcomes—often require many years before they can be measured, Congress should provide protected funding over a long period. As well, the NIH would need to develop clear protocols for how the results from a “successful” science policy experiment are disseminated and adopted into standard practice across the agency, mimicking Medicare’s own process for model certification.

Second, meaningful experiments require rigorous designs and support for evaluation. The Innovation Center experience illustrated the feasibility of conducting well-powered, randomized experiments in government practice and also offered lessons for how to design such models (for example, minimizing model interactions and redundancy). Furthermore, the Innovation Center invested heavily in developing processes for rapid cycle evaluation and supporting independent evaluations to broaden the evidence base for health care reform.

Similarly, effective NIH experiments will require thoughtful designs. For instance, the ability to test experiments in science funding across different institutes and centers could offer insight into whether grantmaking models are generalizable or better suited to specific use cases. When considering evaluation mechanisms for NIH experiments, officials will need to develop criteria that define the outcomes that matter. Although challenging for a field where progress is often measured in decades, the NIH could consider defining specific objectives, such as the proportion of grants going to younger or more diverse investigators, or the translation of pre-clinical research into Phase 1 or Phase 2 trials, or the relative failure rate of purportedly “high-risk” research projects.

Realizing this vision for the NIH will require action from Congress. Although the director’s office does have dedicated staff for program evaluation and portfolio analysis, it lacks both the authority and resources to advance system-level experiments in NIH operations. Other contemporary NIH reforms, such as the recently created Advanced Research Projects Agency for Health (ARPA-H), do not resolve the structural challenges within the agency (and indeed, some policy makers have proposed separating ARPA-H’s personnel and processes entirely from the NIH).

The selection of a new NIH director uniquely positions the agency to assess opportunities for bold action and creative thinking. Although the NIH currently funds many innovative projects in fields ranging from cancer biology to climate change, we believe that investing in the science of science itself would serve as a force multiplier for the nation’s premier institution for biomedical innovation. By applying the lessons of Medicare, policy makers can provide the NIH with a toolkit for reinventing itself and help maximize the returns on investments in scientific research for the US people.

Leave a Comment

Your email address will not be published. Required fields are marked *