HomeOur Strategy Monitoring & Evaluation

Monitoring & Evaluation

Overview

To achieve the greatest possible impact, The Rockefeller Foundation engages in evidence-based philanthropy—measuring the progress of each initiative against specific goals and outcomes to ensure the strategy is delivering results. This approach requires deciding which metrics to track, and designing systems to accurately measure and report on desired improvements in poor or vulnerable people’s lives. The Rockefeller Foundation has therefore focused on building and refining our monitoring and evaluation (M&E) capabilities as a key part of our strategic model.

One key aspect of the Foundation’s approach to M&E is that we not only compile information on the outcomes of our initiatives, but also utilize this information as an essential element in our management processes so that we can take action based on what we are learning. By tracking the performance of individual pieces of work in this way, we can course correct as necessary and ensure transparency and accountability in how our work is designed and implemented. The analysis of progress is also used more broadly to manage the Foundation’s overall portfolio for maximum impact, to derive evidence-based lessons learned from our work and share them externally, and to advance the practice of monitoring and evaluation in the field of social change.

“By tracking the performance of individual pieces of work, we can course correct as necessary, and ensure transparency and accountability in how our work is designed and implemented.”

Here’s how monitoring and evaluation is typically embedded into the lifecycle of an initiative at the Foundation:

  • Early in the development stage, an initiative team partners with the Foundation’s Monitoring & Evaluation (M&E) team to scan the existing evidence in a given field to see what has and hasn’t worked. This scan is then codified in a synthesis review, which provides a critical base of learning that the initiative team can build on.
  • Late in the development stage, once the concept and strategy for an initiative are complete and the initiative plan is being designed, the M&E team works closely with initiative team members, grantees and partners to draft an initial measurement approach and identify key indicators that could be used to track the work if it advances into execution.
  • Once an initiative enters the execution phase, that measurement approach is more fully developed and validated. Often this involves gathering key stakeholders together to clarify a common vision of the problem being addressed, the desired outcomes, and indicators for success. The concept of “shared outcomes” is central to this process, with all grantees and partners understanding the value of their distinct work as well as the role they play in the broader initiative, including interdependencies with other actors. If a grantee’s capacity for M&E is limited, the Foundation will support their ability to develop data management and other capabilities. Often this involves enlisting specialized M&E organizations as partners to help grantees set up monitoring systems and provide support in analyzing monitoring data.
  • The Foundation then continues to review and utilize key indicators of the initiative’s progress on a regular basis, particularly through:
    • Quarterly team and management reviews of monitoring dashboards for individual initiatives, so that we can address in a timely way what is and is not working, relative to the plan, and adapt our management of the work accordingly.
    • Annual strategy reviews, in which the initiative team meets with Foundation leadership to assess longer-term progress and make high-level strategic course corrections as needed, with M&E-based evidence informing the strategy review.
    • Semi-annual or annual forums that bring together the Foundation’s initiative teams with grantees and partners to review progress, share evidence-based lessons and challenges, and identify where changes are needed.
  • Initiative indicators typically range from management indicators that track shorter-term activity and indicate whether the work is progressing at the anticipated pace, quality, and volume to output, outcome, or impact data, which show how well that work is producing desired results. These indicators include a balance of quantitative and qualitative data, while the dashboards highlight key drivers and barriers to help illuminate the story behind the data. In the Foundation’s Smart Power for Rural Development initiative, management indicators currently being tracked include the number of energy supply companies and telecom companies engaged, as well as significant engagements with government and investors. Output and outcome indicators track progress in creating a thriving mini-grid sector (for example, the number of plants built, plant performance, and perception of effectiveness by energy supply companies); expanding economic opportunities (such as the number of micro-enterprises connected); how regulatory issues are being addressed (for example, the number of mini-grid friendly policies, frameworks, or regulations that have been advanced); and success in attracting interest from outside investors (such as the amount of dollars secured). Early impact data measure benefits for poor or vulnerable populations in terms of inclusive economic growth at a household and locality level, as well as benefits for health and safety, education, and gender-specific impact.
  • Each initiative also undergoes a mid-term evaluation about halfway through its execution time frame—typically at about year three or four—to guide mid-course improvements to strategy and implementation. These evaluations are conducted by independent, external teams of M&E specialists, and include a review of interim data and documents as well as interviews with representatives of grantee organizations. For example, the mid-term evaluation for the Asian Cities Climate Change Resilience Network (ACCCRN) was conducted in 2011, three years into the initiative, by Verulam Associates, a U.K.-based international development consulting firm with associate companies in India and Bangladesh. The evaluation found that the intervention was generally effective and timely, and also made 14 targeted recommendations to inform changes that were then implemented in the subsequent years of the initiative.
  • Similarly, initiatives undergo an independent final evaluation at the conclusion of their execution phase to assess the initiative’s results, including sustainability and lessons learned that may also benefit the field. The evaluation for the Rebuild By Design competition, for example, was conducted by the Urban Institute in Washington, D.C., at the end of the competition’s year-long team selection, research and community engagement process. The evaluation found that the initiative was successful as an innovative strategy for reaching resilience goals, while noting certain shortfalls in the initial planning process. These final evaluations are publicly disseminated to ensure that the lessons learned can be leveraged for the benefit of the broader field.
  • The Rockefeller Foundation also looks at progress at the Foundation level, through regular management reviews across our portfolio of work. These reviews enable us to identify implications for resource allocation as well as for pipeline planning in terms of the amount and type of new work that should be explored.
  • In addition to supporting our own staff and grantees in building M&E skills and experience, The Rockefeller Foundation supports the development of global M&E capabilities by funding regional M&E networks and institutions that train, coach, and mentor evaluators, and by partnering with evaluators from other regions. For example, we support the African Evaluation Association (AfrEA), a pan-Africa umbrella organization comprising more than 25 national M&E associations in Africa, as well as The Centers for Learning on Evaluation and Results, located in Africa and Asia.

The Foundation also supports the development of content and new tools for global M&E. The Foundation has funded research efforts on key trends and topics, such as Emerging Opportunities: Monitoring and Evaluation in a Tech-Enabled World, a report published in 2014 by the monitoring and evaluation consultancy firm ITAD, and Assessing Impact Investing: Five Doorways for Evaluators, another 2014 report issued by E.T. Jackson & Associates. We have also supported GlobalGiving’s Storytelling project, which gathers local feedback from people in developing countries, and BetterEvaluation, an online interactive community of evaluation practices that provides advice, support, and good practice examples to evaluators in developing and developed countries.

"The Rockefeller Foundation is committed to evaluation practices that are rigorous, innovative, inclusive of stakeholders’ voices, and appropriate to the contexts in which the foundation works."

Nancy MacPherson, Rockefeller Foundation Managing Director, Evaluation

The Evolution of Monitoring & Evaluation

  • The Rockefeller Foundation’s focus on developing methods to effectively monitor and evaluate the progress and outcomes of our initiatives dates back to our strategic refresh in the mid-2000s. The challenge put forth at that time—to develop a strategic model that would multiply the Foundation’s impact—included the implicit requirement that metrics be developed to measure that impact.
  • In 2006 the Foundation incorporated this evaluation requirement into our new strategic model, an approach that focused on targeted initiatives designed to produce measureable results within a prescribed time frame. That same year, we launched the Alliance for a Green Revolution in Africa (AGRA)—the first large-scale initiative under the new strategic model, which included support for a program monitoring and evaluation capability.
  • Over the next several years, the Foundation refined the process of putting assessment parameters in place during the development phase of initiatives. These measures could be quantitative, qualitative, or a mix of both, depending on the initiative, and ideally could be used to appraise the initiative’s execution and its results—both critical elements of impact. During this time, discussions on the effective monitoring and evaluation of initiatives continued to take place at the executive leadership and Board levels. The Foundation formally established an in-house M&E function in 2008. A year later, we brought the evaluation function under the organizational umbrella of a new in-house strategic planning group, creating the position of vice president for strategy and evaluation.
  • Beginning in 2008, as we continued to refine our strategy for gathering and analyzing data and systemizing grantees’ reporting and data collection, the Foundation began contracting with outside groups to conduct peer-review mid-course and post-completion evaluations of specific initiatives.
  • In 2010, the Foundation augmented our internal evaluation process by establishing a series of meetings known as “Strategy Soaks,” in which individual initiative teams meet periodically with the Foundation’s Executive Team and other relevant staff to review data informing the progress of their initiative and discuss possible strategy adjustments.

More recently, recognizing that the Foundation’s work is focused on dynamic spaces in which opportunities and risks evolve rapidly, we enhanced our short-term monitoring capabilities by instituting quarterly dashboards of key monitoring indicators, which are reviewed in quarterly meetings with initiative teams and the Foundation’s management team.

Monitoring & Evaluation in Action

0/0

Transforming Health Systems Initiative Mid-Term Evaluation—Conducted in 2011

Why this Initiative

While health spending has increased dramatically around the world, access to affordable, quality services has lagged. At the same time, national health systems in developing countries often lack the capacity to meet daily health needs, as well as anticipate, prepare for, and recover from health shocks down the line.

  • A doctor consults with patients. (Photo credit: Jonas Bendiksen)

The Solution

Launched in 2009, The Rockefeller Foundation’s Transforming Health Systems (THS) initiative was a five-year, $100 million program. Its aim was to catalyze innovations and actions to create broader access to affordable, quality health services in developing countries, and enhance resilience in these countries’ health systems and populations. The initiative’s activities included a global effort to promote universal health coverage, as well as a strong commitment to the use of mobile-based electronic health (e-health) tools to facilitate medical care.

Employing Monitoring & Evaluation to Increase Impact

In 2011, halfway through the scheduled time frame of our Transforming Health Systems initiative, the Foundation commissioned an independent, mid-term evaluation of the initiative’s strategy and impact to date, in order to identify potential opportunities for mid-course corrections. The external evaluation team reviewed all relevant strategic plans, reports, and initiative activities, debriefed current and past members of The Rockefeller Foundation’s initiative team as well as our senior leadership, and interviewed some 40 partners and grantees. The evaluation team also conducted a panel discussion that included the Foundation’s initiative team and outside experts, and surveyed a number of global health leaders.

The resulting report included detailed breakdowns on milestones achieved, cost-effectiveness, and perceived impact in each of the initiative’s activity areas. Overall, the evaluation confirmed that the initiative was well-matched to The Rockefeller Foundation’s strengths and had clear value, and that it was exerting substantial global influence. However, the evaluation team also suggested that the initiative’s activities could benefit from a greater degree of focus, and expressed concern that it was “trying to do too much with too little capacity and resources,” especially in regard to the initiative’s in-country work in its four target nations: Bangladesh, Ghana, Rwanda, and Vietnam.

Other potential areas for improvement cited by the external evaluation team included “an opportunity and a desire for greater coordination and communication among grantees”; “a need for greater/simpler clarity in how the work of THS… ties together into something that will synergistically affect health systems”; “room for improved team dynamics”; and “an opportunity for greater clarity around initiative-wide capacity strengthening efforts.”

Based on these and other observations, the external evaluators presented a number of recommendations for mid-course corrections. These included:

    • Increasing the involvement of the initiative’s local grantees, and enhancing communication with all stakeholders around the initiative’s strategy and learnings
    • Defining the initiative’s target end points and policy goals more clearly
    • Defining the initiative’s in-country strategy in its four target nations more clearly, including how the work in each country is supporting the global efforts of THS
    • Clarifying the role and benefits of the e-health tools the initiative was employing
    • Continuing to build on the momentum of the Joint Learning Network for Universal Health Coverage (JLN)—the global consortium for knowledge sharing around universal health coverage that THS helped establish

As a result of this evaluation, and after consultation with the Board of Trustees, the Foundation decided to devote much of its attention in the THS initiative to promoting universal health coverage on the global agenda. It was also decided that THS would focus on strengthening the promising Joint Learning Network in order to maximize its sustainability in the future. Finally, the Board of Trustees agreed that THS would consolidate and complete work in the four THS focus countries, focusing on areas of greatest dynamism and traction in each country. Accordingly, an additional $15 million was appropriated toward these goals and the life of the initiative was extended through 2015.

In retrospect, some of the mid-term evaluation’s recommendations were more valuable in shaping the design of future initiatives than in changing the course of THS itself. The THS initiative was able to achieve a number of positive final outcomes—including its successful push for the United Nations resolution, passed unanimously in 2012, urging all member nations to develop affordable universal health coverage systems, as well as its founding of the JLN, which now thrives independently. The initiative also succeeded in including universal health coverage as a goal in the UN-approved Sustainable Development Goals in 2015. However, the Foundation ultimately agreed with the mid-term evaluation’s concerns that the stated ambitions of the initiative exceeded what could be accomplished with the available capacity and resources. This understanding played an important role in guiding The Rockefeller Foundation’s subsequent strategy, including the decision to implement our 100 Resilient Cities initiative on a robust global scale, with an appropriately high level of funding support.

Learn More About Our Work in Health

Asian Cities Climate Change Resilience Network (ACCCRN) Midterm Evaluation—Conducted in 2011

Why This Initiative

One-third of the natural disasters that occurred in the world over the last 30 years have taken place in Asia. Asia’s cities are at particular risk. In addition to being among the world’s most vulnerable urban areas to flooding, their populations are growing rapidly: By 2025, 21 of the world’s projected 39 megacities (cities with populations of 10 million people or more) will be in Asia. The poorest residents of these urban areas are most likely to live in informal settlements in hazard-prone areas, putting them at greatest risk.

  • With its low elevation, rapidly urbanizing population, and central role in global rice production, flooding alone could easily create catastrophic social and economic disruptions in the region.

The Solution

In 2008, as a $42 million six-year component of our broader Climate Change initiative, The Rockefeller Foundation launched the Asian Cities Climate Change Resilience Network (ACCCRN) initiative. ACCCRN established and supports a self-governing regional network of institutions and experts that work with vulnerable Asian cities, helping them strengthen their capacity to withstand and recover from natural disasters and other effects of climate change. Its activities include funding specific projects, enhancing coordination within individual city governments, and promoting information sharing among participating city and national governments. The network also generates knowledge and platforms to help national governments, multilateral donors, and the private sector prioritize investments that advance urban climate change resilience across the continent.

Employing Monitoring & Evaluation to Increase Impact

In 2011, three years into the initiative’s six-year timeframe, The Rockefeller Foundation commissioned an independent mid-term evaluation to assess ACCCRN’s relevance and rationale, report on its progress, and give recommendations on mid-course corrections. At the time of this evaluation, ACCCRN was engaged with its original ten participating cities in Vietnam, India, Indonesia, and Thailand, with the intent of scaling out the initiative to additional cities in later years.

ACCCRN was the first Foundation initiative in which the short-term monitoring function and mid-term and final evaluations were carried out by the same grantee—Verulam Associates Ltd., a consulting firm with a long history of working in Asia. The mid-term assessment, which included interviews with the Foundation’s initiative team and in-country site visits with grantees and other stakeholders, found that significant progress had been made in terms of planning and executing small-scale resilience projects in the selected cities. At the same time, the evaluation team noted certain aspects around the initiative team’s interactions with its grantees and target populations that it felt could be strengthened.

These observations and related recommendations—some of which mirrored steps already being taken by The Rockefeller Foundation’s ACCCRN team—centered on the initiative team’s relationship with its grantees and the populations they served. In particular, the evaluators felt ACCCRN was being limited by a “hub and spoke” structure in which The Rockefeller Foundation dealt bilaterally with all of its various grantees, with a resulting lack of communication and interaction among the grantees themselves. The evaluating team also suggested that the initiative would benefit from focusing more on the most committed individuals in each of its target cities, who were already doing the majority of the work in advancing the initiative, and cultivating them as project “champions.”

In addition, the evaluation recommended shifting the project’s emphasis from building a literal network—which carried a risk of redundancy, since a number of city-based networks already existed in the region—to connecting key individuals and city organizations with those existing networks. The evaluation also noted that in order to retain relevance, ACCCRN’s projects moving forward had to be of sufficient size and impact to attract citywide interest and gain national and international attention. The external evaluation team also cited the need to develop a clear strategy for scaling out the program to additional cities, in preparation for the Foundation’s exit from direct oversight of the initiative.

Finally, the evaluation team advised engaging ACCCRN’s grantees more fully in the initiative’s ongoing monitoring process, and urged the project to disseminate documentary evidence of its activities more freely, rather than rely on the eventual publication of peer-reviewed articles to communicate the program’s accomplishments.

The evaluation also presented some general recommendations on how The Rockefeller Foundation selects and tasks our grantees in alignment with our strategic model’s emphasis on measurable results. “While grants architecture has been modified to better suit a results-oriented way of working,” it noted, “… there remain aspects of older granting behavior with grantees often charged with fairly low-level activity-based deliverables, and initiative-wide synergies and objectives not always explicitly included in grant agreements and management processes.”

The observations contained in this important mid-term evaluation influenced both the ongoing execution of ACCCRN and The Rockefeller Foundation’s strategic operation as a whole. As noted above, the initiative team had already arrived at some similar conclusions, most notably around the limitations of the project’s “hub and spoke” structure. At the time the mid-term evaluation was completed, the team had already proposed a new Strategy and Alignment Group (SAG), made up of Foundation staff and core grantees, which would serve as a forum for improved communication and coordination among ACCCRN participants. The evaluation team applauded this development and urged the Foundation to implement the SAG concept as quickly as possible.

The ACCCRN initiative team acted on many of the evaluation’s other recommendations as well. Today, ACCCRN has expanded its work to more than 50 new cities and an additional country, Bangladesh, with a primary focus on supporting individual practitioners in these cities while also building partnerships with institutions and country networks. The Rockefeller Foundation has provided funding for the international nonprofit Mercy Corps to administer ACCCRN, though the Foundation continues to provide financial support as a network partner. City-led intervention projects carried out to date include establishing early warning systems for climate-related disease outbreaks, developing new workplace policies to address climate change’s health effects on workers, providing access to storm- and flood-resistant credit and housing, devising new reservoir management protocols, testing innovative cooling methods for buildings, and restoring mangrove populations.

The mid-term evaluation’s findings have also impacted subsequent Foundation efforts. For example, our 100 Resilient Cities initiative, launched in 2013, formalizes the “champion” concept by funding a Chief Resilience Officer position in the municipal government of each participating city. In addition, the Foundation’s initiatives now put greater emphasis on integrating grantees into the monitoring and evaluation process, incorporating overall project goals more fully into their grant arrangements, facilitating communication and real-time learning among grantees, and transmitting the story of each initiative’s accomplishments to the public.

Learn More About ACCCRN

Smart Power for Rural Development Quarterly Monitoring—Launched in 2015

Why This Initiative

An estimated 1.3 billion people around the world lack access to electricity. In rural India alone, nearly 300 million people live without even basic lighting, a situation that significantly hinders their opportunity for economic development.

  • Installing the final module on a solar panel.

The Solution

The Rockefeller Foundation’s Smart Power for Rural Development (SPRD) initiative is bringing electricity to rural areas through a series of mini-grids, each connected to a local power plant that is powered by solar photovoltaic cells, biomass energy, or other renewable energy sources. The initiative is now focused on India, where The Rockefeller Foundation has committed $75 million in funding to SPRD, with the goal of electrifying 1,000 rural villages in the districts of Bihar and Uttar Pradesh as a demonstration of the viability of this model for further rapid scale-up. In three years, the initiative’s intended outcomes include: supporting the operational capacity of regional energy service companies (ESCOs) to establish and maintain mini-grids—work that includes assisting with financing and linking them with telecom companies who serve as anchor customers; helping local enterprises and residents tap into and benefit from the new electricity sources; promoting national and state-level policies that support this new rural electrification model; and encouraging investment in the emerging mini-grid sector.

Using Monitoring & Evaluation to Increase Impact

To ensure that the initiative remains on course, progress in all outcome areas is being carefully monitored. The team has broken down each outcome area into a number of component parts, or sub-outcomes, each of which is being tracked on a quarterly basis. Some of the data is being collected by Smart Power India (SPI), the main implementation organization driving toward the 1,000-village goal, which uses the key implementation metrics to help inform its day-to-day business. To understand the implications of the work at a household level, the Foundation has engaged Sambodhi, an India-based research and consulting firm. Using an innovative metric of village-level GDP, Sambodhi is monitoring how the initiative is impacting individuals and spurring economic growth in its target rural communities. It is also assessing the initiative’s impact on the growth of existing local enterprises and establishment of new enterprises. Finally, the consultancy Dalberg is conducting interviews to understand perspectives of key stakeholders, including ESCOs, policymakers, and funders (both those who are engaged with the SPRD model and those who are not) and is also aggregating and analyzing the SPI implementation data and Sambodhi impact data.

Smart Power for Rural Development Model

Every quarter, the status of each outcome component is summarized in an initiative-level dashboard, with Dalberg updating the aggregated data and the initiative team analyzing why progress was—or was not—made. The team also outlines its perspective on the dashboard’s implications. All of this is then discussed with the Foundation’s management team in a quarterly review meeting. In the outcome area related to scaled operational capacity, for example, sub-outcomes being tracked include: the degree to which the mini-grid sector is being recognized and validated; statistics on the development and efficient operation of additional, viable ESCOs; the status of SPI as a well-established organization; the confidence level of regional telecom companies in SPRD; and how well partnerships and innovations are driving down costs related to the electrification model. For each sub-outcome, both quantitative and qualitative assessments may be included where appropriate, including drivers of and barriers to success, emergent opportunities and risks, and implications for the work in the months ahead.

  • Installing the final module on a solar panel.

Since the initiative’s launch, this robust quarterly monitoring process has helped the initiative team and its partners maintain a steady focus on a wide range of initiative elements, and to identify and address critical issues as they arise. Early dashboard results, for example, highlighted the need to better understand the resource needs of regional ESCOs, to closely track and validate the business case, and to address challenges related to legal and regulatory barriers. To date, thanks in part to this monitoring, the initiative has made significant progress in all outcome areas: A sizeable and growing number of villages are now being supplied with electricity through micro-grids, and local enterprises are also using these new electricity sources to support their operations. Plant business metrics appear to be improving, as well. On the policy front, the initiative team has been working closely with the governments of both Bihar and Uttar Pradesh on issues such as subsidies and grid interactivity, and to support the development of state-level action plans. In addition, the initiative has elicited interest in its electrification model from a range of potential funders, including private-sector investors, non-governmental organizations and government agencies, and several African governments have expressed interest in replicating the model.

Learn More About Smart Power for Rural Development

Related Blog Posts and Reports

See All »

How We Work