Four Ways Foundations Can Increase the Value of Evaluation Beyond Their Foundation: Lessons from the Gates Foundation Networks for School Improvement Design Charrette
/by Clare Nolan and Christina Garcia
Funders wield tremendous power when they commission evaluations of their work. They decide what questions get answered, which data sources and designs are favored, and what reporting and engagement look like. Traditional RFP processes reinforce this power dynamic by concentrating decision-making in the hands of funders and having evaluators develop approaches in silos without feedback from others who can benefit from their work.
But what might it look like to design and commission evaluation differently? This was the question the Bill & Melinda Gates Foundation posed when it first approached Engage R+D and our partner, Christina Garcia Consulting, in January 2019. Over the ensuring months, we facilitated a design charrette that engaged more than 50 people in collaboratively designing an evaluation of the foundation’s largest K-12 investment, Networks for School Improvement (NSI).
A design charrette is a collaborative planning technique commonly used in the urban planning and design fields. It involves consulting with professionals alongside stakeholders who will be impacted by a given project or product with the goal of creating feasible and sustainable solutions.
In this new case study, we describe the design charrette process in detail. Drawing on this experience, here are four ideas for how funders can increase the value of evaluation for grantees and others with the potential to benefit from the knowledge they generate.
Invite internal and outside stakeholders into evaluation design.
Most evaluation RFPs are developed in-house by a small number of foundation staff and/or their consultants. The design charrette upended this traditional approach by inviting outside researchers and evaluators to sit alongside staff and intermediary partners. Together, they identified guiding questions for the evaluation and possible design approaches and vetted different evaluation proposals. This process promoted new thinking and helped all stakeholders better understand the tradeoffs at play in the design of the NSI evaluation. Involving internal and external stakeholders upfront increased buy-in for the resulting evaluation.
Tune into the grantee experience to maximize benefit and reduce burden.
Evaluations are often designed and implemented from the perspective of funders and evaluators who are distant from the day-to-day realities of grantees. This contributes to an undue burden that is difficult to overcome given the power dynamics at play. The design charrette process brought grantees in early on so they could describe their experiences implementing the work and how the evaluation could be useful to them. Grantees also provided feedback on emerging evaluation approaches before they were selected for funding, giving them increased power in the grantee-funder-evaluator triad. Post-charrette, our team facilitated a journey mapping process to help foundation staff, selected evaluators, and intermediary partners understand the whole of what grantees were being asked to do by whom and when. This helped to avoid pain points and catalyze new thinking about how the timing of evaluation activities could be shifted to better support grantees’ work.
Break out of formulaic thinking to avoid unintended consequences.
Breaking out of familiar evaluation paradigms is hard, but possible, with the application of creative techniques. One of the most talked-about exercises from the charrette involved asking participants to imagine ways to create “the most data burden and useless results for grantees, evaluators, and the foundation.” This powerful exercise—known as Triz—enabled participants to raise serious concerns in a lighthearted atmosphere. As one grantee described, “Some of it felt a little fun and light and silly, and then some of it felt really serious and like something that needed to be put into the room to make sure that we were keeping these unintended consequences front-of-mind.” Similarly, one of the NSI intermediary partners shared, “I was surprised by how much clearer my own thinking got... Because it’s this playful exercise, I got to just say [my worst-case scenario] exactly how I actually think it.”
Promote collaboration over competition to achieve better results.
Overall, charette participants felt that the co-design approach produced better results than the typical RFP process. One of the academic participants characterized it as a “golden opportunity” to try something different and reflected, “It was really a big bet by the foundation on this complex initiative, which did not seem well-suited for traditional business-as-usual evaluation approaches.” One of the intermediary partners agreed and pointed out that the process accomplished more than just evaluation design: “The end result is something that more people can buy into and have clarity around, in addition to making sure that the questions are the right questions.”
At Engage R+D, we believe funders can broaden the benefits of evaluation by engaging more stakeholders in their design. We hope that by sharing this story, more folks will consider applying participatory approaches to their work. We believe doing so results in higher-quality evaluations that produce more meaningful insights—not just for foundations but also for others who can benefit from this knowledge.
