Reflecting on the benefits of evaluating a health care delivery program: A recap from the HCDI @ 8 Convening

This spring, the United States Health Care Delivery Initiative (HCDI) hosted its second convening, HCDI @ 8, during which researchers and implementing organizations shared their experiences evaluating programs using randomized trials. As we look to the future of health care delivery, we reflect on these discussions, highlighting why an organization may want to evaluate their program with a randomized evaluation. 

Choosing to evaluate a social program can be a significant decision for an implementing organization. Randomized evaluations reveal the efficacy of a program. At the outset, neither the researcher nor the implementing partner know whether the evaluation will find that the program has the intended impact. Evaluations take time and resources to plan, execute, and analyze, with few guarantees. This then begs the question, why would an implementing organization choose to evaluate their program?

Why evaluate your program?

Sharing their experiences during a panel discussion, panelists highlighted several reasons why they were motivated to undertake an evaluation. To make decisions about effectively using finite resources, the University of Illinois Urbana-Champaign worked with David Molitor and his co-authors to test the impact of a workplace wellness program for employees and its impact on health outcomes. Through evaluation, they were able to find that workplace wellness programs do not improve employee health outcomes and therefore chose not to invest in a program of their own. 

Allison Sesso of RIP Medical Debt, an organization that buys and forgives medical debt, joined as Executive Director during the planning stages of the evaluation of their program, and was very pleased that she had. To Sesso, the primary goal of RIP Medical Debt is to do what is in the best interest of the people they are serving and to ensure that their services are actually helping people.

Additionally, evaluation gives credibility to the program and can aid in attracting donors and new board members. Evaluation shows that “you’re willing to take the leap and…[to look] at yourself in the mirror.” Sesso also said that she is able to leverage the media attention given towards the study as an opportunity to promote the program and its policy implications.

How can partnerships benefit evaluations and programs?

Strong partnerships are vital to conducting a successful randomized evaluation of a program or policy. Partner organizations bring key insights on program operations and the people they serve, while researchers bring technical expertise. Conversations between researchers and on the ground staff make for stronger, more meaningful evaluations that answer the questions that will best help policymakers and practitioners make key policy decisions. J-PAL helps to facilitate the collaboration between the research team and implementing team and provides technical support. 

Allison Hess of Food Fresh Farmacy, which offers a food as medicine approach to supporting people with diabetes who are experiencing food insecurity, echoed the sentiment that J-PAL, which connected her organization with researchers, was vital in providing technical support and securing funding. Hess admitted that she was initially hesitant about going ahead with an evaluation given the small team and limited resources of Food Fresh Farmacy and the challenges of working in health care delivery during the Covid-19 pandemic. Nevertheless, she found that the researchers and J-PAL were “instrumental early on with really laying out the expectations, what our responsibilities were going to be...and also help to secure funding.” Dr. John Bulger, also of Food Fresh Farmacy, noted that collaboration with the researchers and J-PAL “added things to the program that we would have never done” and that “no matter what the result ends up being, [the organization] will be much better for it and [the] patients will be better for it.”

How can evaluations influence policy and decision-making?

Randomized evaluations are important first steps in shaping policy. As an example of this, Dr. Fatima Cody Stanford highlighted a study she completed with HCDI Co-Chair Dr. Marcella Alsan on public health messages about Covid-19 and the effect on Black and Latinx individuals’ knowledge and information-seeking behavior. The randomized evaluation revealed that for Black participants, viewing a message from a Black physician increased information-seeking behavior. These results build on Dr. Alsan and her co-authors’ patient-provider race concordance study which found that Black men who were treated by a Black doctor were more likely to take-up preventive services.

The number of Black physicians in the United States remains low, with only a 4 percent increase in the last 120 years. These studies highlight the importance of diversifying the physician workforce, as well as possible messaging solutions in the interim. Since the completion of these studies, the American Medical Association has strengthened efforts to increase diversity in medical schools and residencies and cited the patient-provider race concordance study in explaining this policy change.

Evaluating a program can be a positive experience that can empower your organization to influence decision-making, but it can be difficult to know where to start and for an implementing organization to find the right researcher and vice versa. J-PAL offers matchmaking services to pair researchers with practitioners, supports study design and implementation, and holds requests for proposals to apply for research funds. If you are interested in learning more about how we could support an evaluation of your program, please reach out to HCDI Initiative Manager Hannah Reuter at [email protected].