Monitoring and Evaluation

The distinct yet related activities of Monitoring and Evaluation offer a cross-cutting service to all our other areas of expertise and are integrated into every stage of the process from design, through implementation to lesson-learning at the end of a programme so as to help ensure we deliver lasting results and maximise Value for Money.

The importance of demonstrating Results and Value for Money has special resonance today. During hard economic times aid budgets will be subject to more scrutiny. There is a responsibility to achieve maximum Value for Money, and this is not just about sourcing the cheapest inputs; it involves delivering the best results at the lowest possible cost. This calls for a balanced approach in being able to define and assess value relating to the economy, efficiency and effectiveness of aid expenditures.

Being too results-focussed within short time frames has potential drawbacks though. This contrasts with the historical over-ambition of impact assessment which became over-sold and over-simplified. There is now a risk that we will restrict aid to short term, measurable indicators instead of linking it to sustained institutional and economic change. In both the private and public sectors there is now a growing recognition of the need to remove the notion that outcomes can be pre-determined through forecasting a path to the future. Failing to do this narrows the challenge to assessing measures of success forecast from the outset. If our work is to deliver lasting results we need to understand the broader picture by identifying processes and factors that could undermine or enhance effectiveness. Unexpected outcomes, good or bad, can matter almost as much as what projects themselves are intended to do or achieve. We need to harness them as effective contributions to achieving change.

For our projects to demonstrate Value for Money, our approach is based on how we can use Monitoring to simplify the way and improve the basis upon which we manage our programmes; and for Evaluation to provide a balanced, rigorous and realistic basis for others to make decisions. We make sure that Monitoring is seen as a function distinct from Evaluation to avoid confusion. Integrating the scope of Monitoring activities into a Management Information System helps managers deliver change, develop relationships, control expenditures and learn about clients’ responses in the context of design expectations.

Evaluations we are asked to carry out should draw on such systems and help clients account for decisions as well as inform them. Our evaluations are driven by the requirement to balance three needs:

the need to understand as much about what is being evaluated and how it came about and why, as well as how best to do and communicate the evaluation;
the need to balance the technical rigour required in our methodology with understanding the realities as explained by beneficiaries; and
the need to ensure that the evaluation itself has a consequence.

Delivering the above approach is associated with providing the following services:

Supporting our clients to build rigorous evaluation into the programmes we are asked to design and/or manage from day one;
Designing bespoke baseline surveys that are tailored to programme objectives and assumptions;
Designing and helping clients put to use the results of, for example: Theory-Based Impact Evaluations that use a mixed methods approach, Process Reviews that focus on exploring changes in behaviours and relationships and Beneficiary Assessments that gauge our performance in delivering support;
Designing, developing and running Management Information Systems that help managers define and assess Value for Money, succeed with the tactics they choose, and to occasionally test them; and
Ensuring that investments we make into the above services benefit our in-country clients and the donor.

Liberia: Reviewing the Process of a Governance and Economic Management Programme

The EC-supported Governance and Economic Management Programme in Liberia was a response to serious corruption and mismanagement of public finances in post-conflict Liberia. The Programme seeks to build a system of economic governance to promote accountability, responsibility and transparency in fiscal management so that Liberia's resources would be used in the interests of its citizens

The overall objective of the 3 month review was to review the capacity and structure of the Ministry of Finance through four broad activities, all of which focussed on exploring the behaviours and relationships among the programme’s core client groups:

Review the current institutional, legal and political structure in which the Ministry of Finance operates and the range of functions it is mandated to carry out;
Review and comment on the current systems and procedures for controlling revenue collection and expenditure monitoring and management, and their appropriateness in ensuring sound financial management;
Assess the Ministry’s current working practices and priorities as expressed in day-to-day activities and/or work plans, and its capacity (human and financial) to exercise its full range of mandated functions; and
Prepare a management plan for the short-term, medium and long-term, based on realistic performance targets and benchmarks.

The evaluation made four key recommendations that sought to further embed national reforms:

The management of expenditure at the Ministry of Finance should now plan ahead to move from securing and stabilisation of the system to reform and regular operations aimed at providing, in the medium-term, greater autonomy to the line ministries and agencies;
to advance de-concentration through planning significant capacity building assistance within the financial departments of key ministries on financial management so as to pave the way for future decentralisation;
the recently passed law on Public Financial Management and the associated Integrated Financial Management Information System should define the regulatory framework to implement the reform process; and
the national budgetary process should focus on increasing its link to poverty reduction policy objectives.

Bangladesh: Assessing the Impact of a Livelihoods Programme

DFID Bangladesh recently contracted HTSPE to carry out an Independent Impact Assessment of the first phase of the Chars Livelihoods Programme (CLP) – an Asset Transfer Programme backed up with other productive and social investments that works with poor and vulnerable households in relatively neglected areas. The objectives of the assessment are to:

identify and better understand the social and economic impacts of the programme,
document operational lessons to strengthen the delivery of the second phase and
provide a foundation for a rigorous independent impact assessment of the CLP II.

Based on a thorough review of impact assessment methodologies and recent guidance on Value for Money from the National Audit Office, we took a theory-based approach to the assessment which involved:

defining the main users of the evaluation among the Government, DFID, AusAID and CLP management and prioritising their information requirements;
mapping out the most important elements of the causal chain to establish the programme’s theory of change through a review of design documents including the LogFrame and discussions with key stakeholders and beneficiaries;
evaluating impacts using a credible counterfactual combined with critical observation from among beneficiaries to isolate and understand the effects attributable to the programme;
rigorously analysing the facts through understanding who actually benefits from the programme, how they were selected and why;
allowing for heterogeneity through a differentiated analysis of impact among different beneficiary groups and individuals;
using an appropriate mix of quantitative and qualitative methods to establish important issues, define important influences and to understand significant differences between beneficiaries; and
assessing the efficiency dimension of the programme’s Value for Money through analysing programmable costs on a per-household basis and the relative input:output ratios of contracted NGOs in delivering services to beneficiaries together with a qualitative analysis of NGO performance from beneficiary perspectives.

See more details in the links below

Nigeria: Developing a Management Information System for a Programme of State Reform

The State Partnership for Accountability, Responsiveness and Capability (SPARC) is a six year governance programme that began in mid 2008. It is supporting the strengthening of Policy and Strategy (P&S) processes, Public Financial Management (PFM) and Public Service Management (PSM) in five states - Enugu, Jigawa, Kaduna, Kano and Lagos.

The strategy for monitoring SPARC is based around the Logical framework and its relationship with the reform plans of those states it is supporting. Baselines were established using a specially developed Self Evaluation Assessment Tool, including an analysis of M&E capacity, and this contributed to strong ownership among state officials of the governance reforms being supported by the Programme.

SPARC’s Management Information System outlines the priority reporting and learning needs for monitoring progress based on the States’ Governance Reform Plans in:

Stimulating state government policy outcome milestones,
Achieving state log frame output milestones; and
Delivering state work plan outputs.

The MIS uses a hosted website where data and information related to all programme activities is stored and readily accessible via the internet for analysis and reporting.

It includes the facility to record, share and put to use a balanced suite of information relating to:

Analysing policy and strategy outcomes and how these vary across the five states;
Reporting on the performance of Technical Assistance (TA) linked to outputs from the logical framework and sequenced to the achievement of specific milestones and supporting actions from State Government; and
Scheduling TA inputs a year in advance to support the achievement of work plans, with detailed terms of reference and associated costs.

This makes it possible to monitor the quality and effectiveness of Programme outputs in relation to the budget and to make rapid and accurate forecasts of programme spend. It also helps to ensure Value for Money by maximising economy, efficiency and relevance. Individual TA inputs are planned within the limits of the Programme budget and approved and implemented when triggered by the overall work plan and corresponding actions from State Government.

See more details in the links below

To view more detail or additional projects please click on the links below:
State Partnership for Accountability, Responsiveness and Capability (SPARC)
Assessing the Impact of a Livelihoods Programme in Bangladesh
Strengthening Performance Monitoring and Impact Assessment of the Rural Support Programmes
Monitoring and Evaluation of Livelihood Programmes
Support for Earthquake Monitoring and Evaluation