After five years of managing monitoring and evaluation (M&E) efforts for a $10 million USAID-funded project awarded to the University of Utah, I suppose I’ve learned a few things about how to do M&E well. Aside from the practical details of how to be more effective and efficient at data collection, management, and reporting, I learned three very important lessons.
First lesson
First, the conceptualization of “M&E” matters. Is it M&E, MEL (M&E + Learning), or MERL (MEL + Research)? And should monitoring even be tacked together with evaluation, research, and learning? As noted by NSF Consulting:
“Monitoring and evaluation are often used interchangeably, yet they are different concepts. As a result, evaluation may be being compromised.”
— NSF Consulting, ‘Monitoring is not evaluation’
Monitoring involves collecting information at regular intervals to support evidence-based decision-making by the project management team. “Evaluation,” on the other hand, is an assessment of whether the development intervention succeeded in its intended effects.
If “intended effects” are conceptualized in terms of “outputs” (e.g., numbers as reported in the monitored metrics), then this is straightforward: you either achieved the targets or you didn’t. However, if “evaluation” is conceptualized as an assessment of support for the underlying development hypothesis, then it becomes more akin to a research endeavor.
In other words, although we use performance metrics as indicators of performance (as we should do), leaders in the development sector (just like good business leaders) should also use metrics for deeper analysis and understanding. As development leaders, we should have great interest in understanding the mechanisms and conditions that support a development intervention’s performance: the why of success. If doing x, y, and z produced a, b, and c in country context p, what makes us think the same results will come from the same inputs in country context q? After all, human development is not like making widgets.
Indeed, to facilitate the expansion of human capabilities, we must understand complex factors that are often case-specific — and at the same time, we must also determine if there are nevertheless meaningful patterns across these cases that might apply to new ones. We need to analyze why interventions do or do not work well. Therefore, MERL — or at least ERL — should be not so much an exercise in tracking and reporting project successes and failures per se, but rather an effort to understand and even predict those successes and failures.
If MERL is conceptualized primarily as M&E — and primarily in terms of monitoring at that — then the broader learning that is so important for “doing development better” is less likely to occur.
Second lesson
This leads me to the second big lesson I learned: theory of change matters. The development sector has a lot of lingo that was new to me back in 2015 as an academic coming from a very different frame of reference. But it quickly became clear that this concept — Theory of Change (ToC) — was a big deal in discussions about M&E, results frameworks, and impact.
I also realized quickly that if a project is interpreted as a contract in which so many “widgets” (e.g., number of people trained, number of trainings, etc.) must be produced during the life of the project, then it almost does not matter from the standpoint of an implementing team what the ToC is. Yes, I know, sounds like heresy, doesn’t it? But, consider this: even though the ToC is the rationale for the structure and design of the program, if the structure and design were established by the donor or client rather than the implementing team — and the implementing team views their work in terms of “widget production” (a.k.a. “checking boxes”) — then the ToC is effectively neutered for purposes of guiding on-the-ground implementation.
Because the project I was involved with was not a contract but rather a cooperative agreement, we as the implementing team had greater built-in autonomy in terms of decision-making and strategic thinking. We were much more in the driver’s seat than if we’d had a contract. Consequently, even though we were mostly a bunch of academics who had relatively limited prior knowledge of ToCs (never mind appreciating the distinction between theory of change and theory of action!), we had to pay attention to our ToC. It actually really mattered. It helped guide our team’s strategic adjustments and adaptations to work plans that supported the achievement of the project goals. In some cases, this meant trading off achievement of a target for achievement of a goal.
Yes, let me repeat that. Sometimes we had to trade-off achieving a target for achieving a goal.
Third lesson
And that’s basically the third big lesson: do not lose sight of the goal for the sake of its proxy indicators. The latter are just tools to achieve a broader and more significant goal. Don’t mistake the tools for the final product.
For our team, this issue manifested itself in the classic form of a quantity-quality trade-off. If the ultimate goal depends on meeting a certain threshold in terms of quality, then meeting this threshold takes precedence over meeting an arbitrary quantitative target such as numbers of participants/beneficiaries. In the case of the U.S.-Pakistan Center for Advanced Studies in Water (USPCASW) at Mehran University of Engineering and Technology, the Center’s reputation in terms of its production quality (i.e., graduates’ competencies and applied research relevance for advancing water security in Pakistan) mattered much more for institutional sustainability than production quantity. This is because the entire Center’s unique selling point is its excellence in terms of quality education and research.
What we were facing was not an uncommon problem in the development sector. In fact, the dual challenge of selecting appropriate performance indicators and using the information obtained through them wisely is one that permeates throughout the public and private sectors alike.
Final thought
I won’t say these are the only big M&E lessons that I learned through my involvement with the USPCASW project, but I do think they are the most salient. No doubt this is a reflection of my entry into the field as an academic social scientist as well as my ongoing interest in understanding how our mental categories and models shape our actions.
Disclaimer: The views expressed above are my own and do not necessarily reflect those of the USPCASW project teams or USAID.
