MVP End-line Evaluation

6307064084_3bf79aeb8f_b

Monitoring and Evaluation systems are critical for real-time tracking of activities and costs, setting priorities, and measuring progress towards goals and program effectiveness. The year 2015 marks the end of the ten-year Millennium Villages Project, see below for The Lancet peer-reviewed final evaluation protocol (submitted March 2015, accepted June 2015).

Protocol and accompanying technical paper

The Millennium Villages Project: A protocol for the final evaluation
Shira Mitchell, Andrew Gelman, Rebecca Ross, Uyen Kim Huynh, Lucy McClellan, Matthew Harris, Sehrish Bari, Joyce Chen, Seth Ohemeng-Dapaah, Patricia Namakula, Sonia Ehrlich Sachs, Cheryl Palm, Jeffrey D. Sachs

Click here for the protocol summary in The Lancet.

Abstract
The Millennium Villages Project is a ten-year integrated rural development project implemented in ten sub-Saharan African sites. This protocol documents the final evaluation of the project, including its basic principles, site selection, and a five-part evaluation plan. The evaluation will include an adequacy assessment, impact evaluation, cost assessment, process evaluation, and description of systems design and tools. We describe data collection and analysis for each component, as well as our plan for transparency and study limitations. Taken together, this evaluation is designed to assess the Millennium Village Project’s model for achieving the Millennium Development Goals in rural sub-Saharan Africa.

Full text can be viewed here.
Accompanying technical paper can be viewed here.

Timeline

Slide1 copy

FAQs

Note: Many technical issues regarding data and statistical methodology can be found in the technical paper.

What has been the timing of the evaluation protocol’s preparation? [+]

The MVP is now in the 10th and final year of the project. During 2012-2014, the MVP team met with the Independent Expert Group (IEG) to discuss evaluation options. During 2014-2015, the MVP team formulated a strategy based on the IEG recommendations, extensive discussions with experts outside of the MVP (see acknowledgements in The Lancet protocol), and internal deliberations. The protocol was submitted to The Lancet in March 2015, went through a peer-review process, and was accepted in June 2015. Data collection has begun in all ten sites; see The Lancet protocol for details.

Why is the MVP evaluating only 10 of the 14 MVP sites? What happened with the other 4 sites? [+]

From The Lancet protocol: “Four sites were not scaled up for various reasons and therefore did not attain the economies of scale needed to sustain full operations: Toya, Mali; Ikaram, Nigeria; Dertu, Kenya; and Gumulira, Malawi. Ikaram’s United Nations Development Programme (UNDP) funding ended in 2010 and was therefore discontinued as an MV site. Gumulira began with funding from a donor grant and was taken over by another non-governmental organization (NGO) in 2011. Dertu and Toya remained small and were caught in regional conflict.”

Given evaluation resource constraints, the MVP chose to focus on the ten sites that were scaled up. These were the original ten MV sites in each of the ten countries, and they remained in operation throughout the project’s ten-year period. We estimate the treatment effect for areas where the MVP would have scaled up to 25,000 inhabitants without disruption. Our comparison villages must come from these areas (to match the treatment areas). We’ve attempted to address this with our matching and regression adjustment. To select appropriate comparison villages, our matching (and regression adjustments) must include pre-treatment variables that are predictive of selection to be a Millennium Village and scale-up.

To extrapolate beyond these ten sites, we must decide whether an external area would have been chosen to be a Millennium Village and be scaled up to 25,000 inhabitants. Additionally, we do not advocate extrapolating to geopolitical and agroecological areas that are quite different from those of the ten sites we study. External validity is challenging in all social policy studies, so extrapolation of the evaluation results to new areas will always require careful thought. A major strength of the project lies in its implementation in ten different countries and wide range of agroecological zones.

How does the MVP define project start date of a site? [+]

The MVP evaluation team consulted the project leadership and site teams to determine the earliest date at each site that the MVP started operations.

How does the MVP define “treatment”? [+]

The planning and implementation of interventions has varied across sites based on local context at each site, which explains the variety of interventions implemented at varying times. At the end of the project, a book will document each site’s field operations and experiences over the course of the MVP.

Is the MVP evaluation using country census data? [+]

From The Lancet protocol : “Census data for our ten countries is not readily accessible at a fine enough (i.e. roughly MV cluster size) geographic scale with which to identify potential comparisons. Furthermore, boundary data is required in order to map administrative units. For selection of comparisons we can only use census data prior to the project baseline (2005), for which boundary data at a fine enough geographic scale is often unavailable. We have been working to resolve this issue. Due to time and resource constraints, census data were not acquired and processed in time for selection of comparisons.” Nonetheless, we hope to continue to work with site teams to procure census data, and hope to fit small area models using these data. See the technical paper for details.

Is a ten-kilometer buffer sufficient to guarantee against spillovers from the MV to the comparison areas? [+]

From The Lancet protocol: “There are three possible types of interferences between the treatment areas and candidate comparison areas, sometimes referred to as “externalities”, which may interfere with the SUTVA, discussed above. The first is a spread of services to nearby areas, reduction of infection risk, and use of services within the MV cluster by temporary or sustained migrants. Second, government spending within the district containing the MV may shift from the MV to other areas in the district. Third, areas outside the cluster may imitate the MVP interventions or adopt policies such as bed-net and fertilizer distribution.” The ten-kilometer buffer zone works to minimize the first type of externality. There is no guarantee that there is no spread of services beyond ten kilometers. The other types of spillovers are indeed harder to control on a purely geographical basis. We will attempt to analyze them qualitatively, especially regarding policy changes at the district and national levels.

What is the sampling and enumeration strategy? [+]

We use a two-stage sampling process to select households within the villages and then individuals within the sampled households. From The Lancet protocol: “The target sample size for each module is determined using a combination of logistics, and relative importance of different vulnerable populations and intervention beneficiaries. A demographic census in sampled households serves as the sampling frame for the within-household sampling. Enumerators are provided with pre-populated survey tools identifying the individuals prior to administration of each survey module.”

How is the MVP measuring out-migration from the villages in the last 10 years? [+]

Our evaluation team does not currently have the resources to complete this particular project. From The Lancet protocol: “permanent out-migration (households that have left the MV1 and do not return by the 2015 data collection period) will not be logistically feasible for our team to measure.”

Is the MVP surveying in both the MV1 and MV2 areas of each cluster? [+]

From The Lancet protocol: “The first two evaluation components (adequacy and impact) are restricted to the core intervention areas (the MV1s), but the remaining three components (cost, process, and systems design) study the entire Millennium Village clusters (both MV1s and MV2s). The survey work is restricted to the MV1s due to budget constraints, so we focus on estimating the program effects in the core intervention areas (the MV1s).”

What is the MVP definition of “success”? [+]

The primary outcomes of interest in the final evaluation include both a subset of the Millennium Development Goal (MDG) indicators and proxies, and indicators that are relevant to systems delivery. The MVP was initiated as a demonstration of achieving the MDGs at a reasonable scale and cost. The adequacy assessment of the MVP can show progress toward the MDG and MVP targets with no comparison. An impact evaluation will estimate the project’s effect on all outcomes.

With many outcomes, there is concern about “fishing” and “multiple comparisons” that are essential to clarify issues of interpretation. In Section 6.5 of The Lancet protocol we address this issue and propose two summary measures as our primary objective: “The final evaluation will report and compare results from all models and all outcomes (reporting posterior intervals of uncertainty), reducing the scope for fishing, i.e. reporting a model based on the realization of the conclusion.” Additionally, to offset interpreting patterns of statistical significance that emerge from random chance (including the problem of “multiple comparisons”), we will construct two summary measures: one of all the indicators and one limited to only the Millennium Development Goal indicators and proxies. See the technical paper accompanying The Lancet protocol for further details.

How will survey and demographic census data from the MVP be released to researchers and the public? [+]

Some of the project data from the first half of the project have been made available through a password-protected online micro-data catalog, to which access is provided on request from researchers. However, most data have not been made available on this website largely due to shortage of resources for data archiving systems and a lack of streamlined data management systems at the start of the project. Before all data are added to the website, researchers can obtain survey data with a data request application process and approval from Columbia University IRB. Data requests are generally processed in 3-4 months. Questions about data request access can be directed to sbari@ei.columbia.edu.

How were the MDG targets derived for the MVP? [+]

Targets were set based on the global MDGs, and standards of UN agencies (e.g. WHO) and the World Bank. Some target definitions have been adjusted to more accurately line up with project intervention targets and goals.

Will the MVP evaluation be able to discern which interventions worked and why? [+]

Although the impact evaluation will not quantitatively estimate the effects of each individual intervention (of which there are dozens per site), the process evaluation will try to qualitatively analyze key mechanisms of action. Additionally, across the sectors of intervention (e.g. health, education, infrastructure, and agriculture), there is knowledge about causal pathways both in the literature and in development practice. We leave to future work the task of quantitatively discerning the effects of individual interventions within the context of the entire MVP package, including estimating interactions among the interventions.

Will the MVP be able to assess sustainability? [+]

As we wrote in The Lancet protocol, sustainability will be difficult to assess in this final evaluation, before follow-up surveys. One method used by ITAD in their evaluation of the Northern Ghana Millennium Village is to look at variables that have lasting impacts, e.g. child stunting. The costing and process evaluations will assess the ability of the sites to maintain the MVP’s delivery systems and interventions after 2015. However, a definitive claim about sustainability of the impact in the years following implementation will not be made with the data available in 2015-2016. We hope that in following years we may be able to shed more light on sustainability at some or all of the sites.

Please explain the role of the Independent Expert Group and other experts? [+]

The Independent Expert Group (IEG), led by Professor Robert Black of Johns Hopkins University, worked with the MVP from 2012-2014 to discuss various strategies for evaluation. The IEG put considerable emphasis on the adequacy trial (measuring progress towards the MDGs), the costing, and the process evaluation. The MVP evaluation team also consulted widely beyond the IEG (see acknowledgements in The Lancet protocol), especially for specialized and detailed advice on causal inference and help to formulate a strategy for impact evaluation, including matching the MVs with comparison sites using 2005 attributes.

Is the MVP collaborating with external researchers in the data collection phase? [+]

The MVP is engaging the Africa Population Health Research Center (APHRC) for oversight and auditing during the data collection phase.

Contact

Kyu Lee | The Earth Institute, klee@ei.columbia.edu

M&E Team

  • Jeffrey Sachs
  • Cheryl Palm
  • Rafael Flor
  • Seth Ohemeng-Dapaah
  • Patricia Namakula
  • Shira Mitchell
  • Rebecca Ross
  • Sehrish Bari
  • Kiwon Lee
  • Ryan Mariott
  • Kyle DeRosa
  • Saira Qureshi
  • May Hui
  • Bocan Wang
  • Matthew Harris
  • Kyle Schneps
  • Lucy McClellan
  • Madeline Woo
  • Lina Henao
  • Caroline Fox

Independent Expert Group Members

  • Robert Black, Director of the Institute of International Programs | John Hopkins University
  • Martin Bloem, Senior Nutrition Advisor | World Food Program
  • Peter Eriki, Health Systems Strengthening Director | African Centre for Global Health and Social Transformation (ACHEST)
  • Alex Ezeh, Executive Director | African Population and Health Research Center
  • Paul Farmer, Professor of Global Health and Social Medicine | Harvard Medical School
  • Wafaie Fawzi, Professor of Nutrition, Epidemiology, and Global Health | Harvard School of Public Health
  • Demissie Habte, President | Ethiopian Academy of Sciences
  • Charles Hongoro, Research Director | Human Sciences and Research Council
  • Peter Hotez, Dean of the National School of Tropical Medicine | Baylor College of Medicine
  • Dean Jamison, Professor Emeritus of Global Health | University of Washington
  • Prabhat Jha, Director | Center for Global Health Research
  • Megan Murray, Professor of Epidemiology | Harvard School of Public Health
  • Dan Wagner, UNESCO Chair in Learning and Literacy | University of Pennsylvania
  • Walter Willet, Professor of Epidemiology and Nutrition Chair | Harvard School of Public Health

Many thanks to the Bill and Melinda Gates Foundation, the Government of Japan, the Korea International Cooperation Agency, the Open Society Foundations, and all host countries for generously supporting the final evaluation of the Millennium Villages Project. Click here for the full list of partners supporting the Millennium Villages Project.

Tags

What You Can Do Visit the Action Center >

  • Stay Informed

    Get news and updates.

  • Spread the Word

    Share Millennium Promise.

  • Make a Donation

    Lend your financial support.

    Donate