[Editor’s note: This post was written by Dena Lomofsky from of Southern Hemisphere. It emerges from reflection from P&I´s online course on Monitoring, evaluating and learning on policy influence, supported by the Think Tank Fund and the Think Tank Initiative.]
Thinking small needs thinking big first. When designing how to monitor and evaluate it is best to start with the big picture and understand the intervention in its full complexity, and then narrow it down to fit the competencies and resources of the organisation. In the recent on-line course on MEL for policy influence by P&I, participants were asked to develop a MEL matrix. Given three examples of different types of matrixes, most people chose to do a theory or results based matrix. This type of matrix derives from a results-oriented approach where the main focus is placed in gathering evidence on how established objectives are being or have been achieved (or not). It consequently links policy influence goals with expected outcomes and outputs and seeks to verify if, when, and how these have been attained. The instruction suggested that people should first complete a pathway of change.
The matrix is only a representation of your programme ideas in a way that systematises your thinking. Whether you think in circles or in logic models (), a matrix is a useful tool to transform a rather complex situation into a series of logical and “manageable” sections. In short, it is a tool to fix, systematically and logically, a series of questions that we should answer, and then define the appropriate instruments to elicit answers to those questions. (P&I course on MEL for policy influence, Module 4, page 7.) The theory based matrix presented in the course had columns for an overall goal, a long term outcome, a medium term outcome, outputs and indicators.
The most common problem with the assignment was that people struggled with the logic of the intervention and the results hierarchy, and immediate outcomes would be reversed with what seemed to be long term outcomes. For example, it is usual to affect changes to capacity or attitudes of policy makers before they will bring about system level changes. Here is an example of how the order got mixed up:
Policy GoalLong term outcomeMedium term outcomeOutput
IndicatorsEnvironmental improvementIncreased awareness of policy makers about the importance of environmental legislation to achieve good economic, social developmentAdoption of regulations that will result in compliance with European standards legislationEvents held with policy makers on EU standards regarding environmental legislation
No of events
Type of events
Content of regulations resulting in EU compliance
No. of event participants who report increase awareness of legislative topics
Topics that awareness has been raised on
This indicates that most people tried to fill in the matrix table instead of first working through the logic of their programmes first – what is frequently implied in a theory of change. Skipping this kind of analysis is one of main reasons that people struggle with the logical framework matrix. The analytical phase of the logical framework approach involves stakeholder analysis, problem analysis, objectives analysis as depicted in the diagram below:
Source: Southern Hemisphere course in planning, monitoring and evaluation for development interventions – diagram by Mandy Barnett.
If you follow these steps, or use a Theory of Change process, then understanding the project logic and putting the objectives into the matrix is not a difficult task. Also, once the objectives are well defined and ordered, then developing indicators that will give a full picture of the project is much easier.
Here are some examples of objectives and related indicators:
Policy influence outcomes
IndicatorsIncreased ‘demand’ from government for researchDirectly expressed interest to receive assistance/advice on a particular policy issueInvitations to join to policy forums, provide technical assistance, comment to legislation drafts, etc.New research projects generated as a request from governmentIncreased interest on an issue or proposalNumber of meetings and educative interactions with policy-makers, profile of engaged policy-makers, level of satisfaction and demand for information, support or related services, new joint initiatives, diversification of links with different political partiesIncreased number of partners supporting an issueQuantity and profile of new trusted champions and messengersQuantity and profile of coalitions; degree of alignment of goals, focus, strategies, etc.
As shown above, once you have the results hierarchy worked out, it is easier to identify what is a long term outcome, a medium term outcome and output. Following an analytical process will also allow you to think about assumptions and context and how they influence your programme design and MEL choices. This is very important in terms of ensuring that when you give meaning to what indicators reveal you can better link them to external forces and also check whether your overall thinking is correct or some assumptions should be refined or completely re-visited.
The problem with trying to fill in the matrix without first having a good overall concept of what you want to achieve, is that the story that you are trying to tell becomes disconnected. If the objectives (outputs and outcomes) are not telling a progressive story, and the indicators are not directly related to the objectives, then when you gather the data on the indicators they will provide an incomplete picture of what you have achieved (or not).
Also, in order to tell a full story, it is best to have both qualitative and quantitative indicators. Most people in the course were comfortable with developing quantitative indicators (e.g. No. of people attending the event), but did not add qualitative indicators such as (the profile of people who attended the event). Sometimes it is more important to get the five most important decision-makers in a room than having lots of participants who cannot influence a policy decision. Getting both the numbers and the texture of things will reveal in a much clearer way whether you are walking in the right direction and provide all the pieces of the puzzle.
Comments