USAID has long supported research designed to identify interventions that are effective at achieving social and behavioral change (SBC) in global health and development, and has funded programming based on the findings from these research efforts. In a recent request for information (RFI), USAID asked the SBC community for advice on how USAID could both improve the quality of the evidence on which its SBC programming is based, and foster local leadership in producing and implementing SBC evidence and programming responsive to local priorities.
The Social Science Research Council recently submitted a response to an RFI from the National Institutes of Health’s Common Fund, detailing how the NIH could improve the reliability of evidence in behavioral research by borrowing the idea of master protocols from the field of oncology. Master protocols are coordinated multisite trials followed by meta-analysis designed to assess both the internal and external validity of interventions across populations. Here, we share a related submission to the USAID’s RFI, suggesting that master protocols could also help USAID achieve its goals of improving the quality of SBC evidence while simultaneously ensuring responsiveness to local SBC priorities and centering locally led research teams.
The Challenge: Improving the Evidence Base for Social and Behavioral Change (SBC) Programs
A significant challenge to providing evidence-based, effective, and scaled social and behavioral change (SBC) programming is the unreliability of the existing SBC evidence base. Virtually all SBC interventions are evaluated through single-shot randomized controlled trials, on the model of clinical trials of biomedical products. But single-shot trials of SBC interventions may be less reliable than findings from similarly-sized clinical trials of biomedical products.
First, SBC interventions may produce smaller and/or more variable effects than the effects produced by biomedical products. As a result, findings from single-shot trials of SBC interventions may be unreliable: True but small signals may be swamped by noise; large signals may be misleading. Second, it may be challenging to ensure fidelity to SBC intervention designs across settings, leading to overconfidence in the external validity of findings from single-shot trials of SBC interventions. Third, SBC interventions may be more likely to have interactive effects with contextual features. Single-shot trials will fail to identify these interactive effects across diverse contexts. Finally, academic journals in the social and behavioral sciences may be less likely to publish replications than to publish original studies, potentially leading to publication bias in favor of single-shot trials of SBC interventions reporting possibly unreliable estimated effects.1Isaiah Andrews and Maximilian Kasy, “Identification of and Correction for Publication Bias,” American Economic Review 109, no. 8 (August 2019): 2766–94. Empirically, we know that many single-shot trials of SBC interventions fail to replicate.2Open Science Collaboration, “Estimating the Reproducibility of Psychological Science,” Science 349, no. 6251 (2015).
Recent efforts by private philanthropic organizations have sought to address this challenge by funding multiple simultaneous randomized controlled trials of SBC health interventions. During 2021 and 2022, for example, a coalition of private philanthropies contributed a total of $12.5 million to enable the Social Science Research Council to organize the Mercury Project research consortium to evaluate SBC interventions designed to increase vaccination uptake in the United States, Africa, Asia, and Latin America/the Caribbean. Contributing philanthropic organizations included the Rockefeller Foundation, the Robert Wood Johnson Foundation, the Bill and Melinda Gates Foundation, the Alfred P. Sloan Foundation, and Craig Newmark Philanthropies. The National Science Foundation committed up to $12.5 million in additional matching funds. Currently 15 teams are working to evaluate a portfolio of SBC interventions in a variety of settings, with three additional teams to be announced shortly.
The Mercury Project consortium is organized around a common research framework that groups interventions by type and defines common outcomes. For example, intervention types include those that reduce information search, decision, and logistical costs, and those that increase the social/peer benefits of vaccination. These intervention types go well beyond the communication-based interventions that are often overrepresented in SBC programming. Common outcomes in the Mercury Project research framework include accurate knowledge of disease risks and vaccine safety and verified vaccination uptake.
However, the trials in the Mercury Project consortium are not closely coordinated replications of a single well-defined set of SBC interventions. Instead, the Mercury Project trials are better understood as one-shot trials of different SBC interventions in different settings.
In order for SBC health programming to be informed by reliable, credible, and reproducible scientific knowledge about SBC health interventions, with both internal and external validity, trials of SBC interventions need to be rigorously replicated in multiple settings. Because of the difficulty of publishing ex post replications of trials of SBC interventions, multiple-site simultaneous replication should be built into study designs ex ante.
The Solution: Master Protocols for SBC Research
A significant facilitator of sustainable, evidence-based, effective, and scaled SBC programming would be support from USAID for “master protocol” designs to evaluate SBC interventions. Master protocol designs, an emergent framework for multisite research in biomedical contexts, provide a transformative opportunity to build sustainable, evidence-based, effective, and scaled SBC programming.
Master protocol trial designs provide for closely coordinated clinical trials and meta-analyses designed to efficiently evaluate the efficacy of interventions in multiple populations and settings.3“Master Protocols: Efficient Clinical Trial Design Strategies to Expedite Development of Oncology Drugs and Biologics Guidance for Industry,” US Food & Drug Administration, March 2022. The use of master protocol designs has recently grown rapidly in the field of oncology, but the design has yet to be applied to SBC research agendas.4Jay J. H. Park et al., “Systematic Review of Basket Trials, Umbrella Trials, and Platform Trials: A Landscape Analysis of Master Protocols,” Trials 20, no. 572 (2019).
Master protocol designs provide a transformative opportunity to dramatically increase the efficacy and sustainability of SBC programming in global health and development settings. For example, because SBC interventions may produce relatively small and/or variable effects, findings from single-shot trials of SBC interventions may be unreliable. Master protocols ensuring the coordinated replication and meta-analysis of multiple trials of SBC interventions in diverse contexts and settings could efficiently identify those SBC interventions that consistently replicate (or fail to replicate), leading to more effective and sustainable SBC programming.
Because some SBC intervention designs may be challenging to implement with fidelity in diverse settings, positive findings from single-shot trials may lead to overconfidence in external validity. Master protocols could efficiently identify those SBC interventions that fail to replicate because of infidelity in implementation across multiple sites, again leading to more effective and sustainable SBC programming.
SBC interventions may be likely to have interactive effects with contextual features, but single-shot trials will fail to identify these interactive effects across diverse contexts. Master protocols could efficiently identify whether and how these interactive effects exist, contributing to more effective and sustainable SBC programming.
Finally, because master protocols build in simultaneous replication from a project’s inception, both meta-analyses and findings from the subtrials in SBC master protocol designs could be published simultaneously, avoiding the publication bias affecting ex post replications. The result would be SBC programming better informed by both local and general findings.
Master Protocol Designs for SBC Research on Vaccination Uptake
A key global priority is to identify SBC interventions that can ensure high and equitable vaccination rates, consistent with USAID’s stated commitment “to advancing equitable immunization to save lives and protect children and communities from outbreaks of vaccine-preventable diseases.” Finding SBC interventions that can produce high vaccination rates will only become more important going forward, given the potential for novel vector-borne diseases to emerge from climate change–induced disruption of ecosystems,5Colin J. Carlson et al., “Climate Change Increases Cross-Species Viral Transmission Risk,” Nature 607 (2022): 555–562. human expansion into animal habitats,6Rory Gibb et al., “Zoonotic Host Diversity Increases in Human-Dominated Ecosystems,” Nature 584 (2020): 398–402. and global migration.
A key technical gap in finding SBC interventions that can ensure high and equitable vaccination rates is that the existing evidence base is derived largely from single-shot trials, undermining the potential for more effective and sustainable SBC programming to support high vaccination rates. This presents an opportunity for USAID to support a pilot master protocol design for SBC interventions designed to increase vaccination uptake. For example, findings from the first 18 SSRC Mercury Project trials of SBC interventions to increase vaccination uptake will be published in 2023 and 2024. Other trials of SBC interventions designed to increase vaccination uptake could be sourced from the research literature. The most promising interventions could be selected for inclusion in a master protocol design launched in 2024/2025, providing for simultaneous replication trials in multiple populations and settings followed by meta-analysis. The master protocol could be led by a coordinating organization that would ensure fidelity to intervention and study designs across settings and conduct meta-analysis of coordinated findings.
An investment by USAID in a master protocol design to evaluate SBC interventions targeted at increasing vaccination rates would result in replicated and policy-relevant findings for the SBC interventions evaluated in the pilot master protocol, leading to evidence-based, effective, and scaled SBC programming that ensures high and equitable vaccination rates. An investment by USAID in a master protocol design for SBC vaccination programming would also significantly advance the field of SBC research.
Such an investment would be transformative, with high potential to dramatically affect SBC research and programming over the next decade by establishing master protocols as a benchmark standard. The investment would be catalytic, resulting in multiple high-impact deliverables within five years, including validated protocols for designing, managing, and analyzing the results from master protocol designs in SBC research. Because the behavioral drivers of vaccination uptake likely share commonalities across domains, the investment would be synergistic, with findings in one domain (e.g., HPV) leading to knowledge advances in other domains (e.g., childhood immunization). The investment would be cross-cutting across the missions of multiple partner countries and be relevant for multiple diseases/conditions. Finally, the investment would be unique, with no other organization likely to fund such a research agenda.
Master Protocols, Local Priorities, and Locally-Led SBC Research
A priority for USAID is to improve not only the quality of the SBC evidence base, but also the extent to which SBC research projects are responsive to local priorities and led by local investigators.
Too often in global health and development research, project teams consist of mostly North American and European researchers, who define research questions and lead SBC research projects. However, this approach is not likely to lead to long-term strengthening of local research and implementing organizations. Further, there is compelling evidence that when a research team includes local research partners and is responsive to local priorities, evidence uptake is more likely.7Michael Kremer et al., “Is Development Economics a Good Investment? Evidence on Scaling Rate and Social Returns from USAID’s Innovation Fund” (working paper, USAID, March 2021).
A master protocol design to produce SBC evidence about vaccination uptake could easily be designed to be responsive to local priorities. Many countries strive to achieve high, equitable, and timely vaccine coverage; subgroups across countries may also face similar barriers to getting vaccinated, including barriers amenable to SBC intervention. A global master protocol for vaccination uptake research could be built around countries with similar vaccine priorities. Countries with budgets that reflect this priority could opt in to be part of a master protocol RFP focused on uptake of specific vaccines, defined by canvassing the existing literature to identify the most promising SBC interventions relevant for uptake of those vaccines. Research teams working on the same master protocol could be coordinated through a consortium that creates opportunities, incentives, and efficiencies for knowledge generation across sites and teams. In this way, country priorities could be addressed while also allowing for more generalizable and effective SBC programming.
Further, a master protocol’s coordinating organization could ensure that research teams participating in the master protocol are led by local investigators, following the approach used by the Social Science Research Council for the Mercury Project. That approach (a) prioritized SBC proposals originating from the country of study; (b) in some cases suggested the addition of external research partners to locally led SBC research teams; and (c) supported PIs and implementing organizations local to the country of study with monthly virtual research workshops and annual in-person convenings.
Master protocols are a promising strategy to improve the quality of the evidence on which USAID SBC programming is based, and to foster local leadership in producing and implementing SBC evidence and programming responsive to local priorities. We urge USAID to consider supporting master protocol designs in its social and behavioral change research and implementation work.
