Further to Thevan’s post on Impact evaluations; here’s an excerpt from a very interesting article on “Evaluation Evolution” posted in the Broker magazine the other day (link to full article below);

Three approaches to evaluation

Evaluation evolution?

Read more.. 

It’s 23h00 at night and I’m sitting in a room far from home, about 500 km away from home, in Midrand, South Africa. I’m reflecting on whether the work that we do in Managing for Impact is all worth it. Today and tomorrow (09-10 June 2008) we’re facilitating a workshop on the Theory of Change and Logframes, and the purpose of the workshop is to facilitate a process where Civil Society Organisation (CSOs) working in the area of water, sanitation, and natural resource, all to improve people’s livelihoods, develop their Theory of Change.

In going through the Theory of Change today one of the particpants said, “Yes this is another tool among many, but what is lacking with us as “Craftsmen”, is that we have not yet mastered our craftsmanship. It does not matter how many tools we have or are introduced to, if we don’t know how to craft then all these tools are useless”.

In reflecting on the statement said, I then reflected on the recent Xenophobic attacks in my beloved country, South Africa, and instead of looking for someone to blame for what happened. I started thinking about whether MfI would play a meaningful role in making sure that this does not happen again, or whether this is a futile exercise, because maybe we don’t have truly committed and dedicated craftsmen and women, who are willing to give their lives to ensure that development (their craftsmanship) is a success.

What this specific individual said has a lot of truth in it, because how do you monitor and evaluate something you don’t know how to “craft”. How will you know that you have done an excellent work, and if you don’t know what impact you want to make, then what purpose does monitoring and evaluation process serve if you don’t know why you do what you are doing?

I truly miss the days when  South Africans were fighting for democracy, because then we had “craftsmen/women”, who knew and understood the change that they want to see, and the Theory of Change from apartheid to a democratic country that is a home for all who live in it, was understood and truly accepted by all involved. I am not saying I want to go back to apartheid, what I’m saying is that Africa needs committed development practitioners, who understand development and development process, and are willing to commit to this Theory of Change, and maybe it’s time that as development workers/practitioners, we have our own Theory of Change. So that we sharpen our craftsmanship, and as we continue working, we all understand, accept, and commit to this change that we want to see in Africa.

Is it all worth it , that I am sitting in a lonely hotel room, in a cold winter night, trying to do my best to make a difference, so that we don’t have another attack of one African by African? My answer to that is, no matter how small my contribution is to this process, YES this is all worth it!!!! 

“Stop asking us the why, what now, and so what questions. Because we are not used to answering questions; we’re used to just implementing what we’re told”. This was a reaction by one of the participants of the MfI workshop that was facilitated for the officials of the Department of Social Development in the Northern Cape province South Africa.

The training was specifically organised for this group and these are Development Practitioners, whose responsibility is to oversee and supervise implementation of developmental projects in local communities of the Northern Cape. At the start of the workshop they felt that the subject and the MfI concept was fun and easy to grasp, and it is something that can easily be adapted to their current way of doing things, and this was before the Theory of Change and M&E were introduced to them. The feeling was however different after the introduction of the Theory of Change and M&E, as they felt that these concpts were completely shaking their ”comfort zone” (project implementation mode, and monitoring project implementation rather than measuring the impact of the project on communities).

What was fascinating with this workshop was the fact that the participants felt that the workshop had absolutely changed their mindset around how they have always viewed development, and even though they found M&E a challange, what was interesting was the fact that they understood that they found it challanging not because it is a diffuclt process, but because it’s a process that they had never seen as part of project implementation. They also struggled to understand how Theory of Change informs the monitoring and evaluation approach and process, and because their theory of change as a department has always focused on project implementation, it was nerve-wrecking for them to understand how then does one monitor and evaluate the impact of the project on communities, when the theory of change was never about making impact on communities.

The projects that are designed by the department of Social Development, that are supposedly aimed at reducing poverty, are mostly based on the national masterplan, and this means that all provinces implement the same type of projects across the country irrespective of the local conditions and capacities. The also is not a clear Theory of Change, which specific outcomes, and objectives, and in the process what then happens is that officials then adopt a project implementation mode as this is how their performance is maesured. their performance is not measured based on whether the project has had any impact in incraesing people’livelihoods, but rather on whether the project is up and running, irrespective of its relevance or non thereof.

One positive criticism about Mfi though is that there is need to further simplify M&E, as it currently still has very strong elements of scientific research process, and yes maybe we cannot avoid this completely. What we then need to do is design different M&E modules for different audiences. This recommendation is made at the back of the workshop with government officials, where they expressed that because of MfI they understand the value and role of M&E, however they do feel that it is too “technical” for them, and this scares them off.

One of the participants when asked what he understood by M&E, said “M&E is a process where one collects information about what they do, to assess whether they are still on the right tract.  It is therefore important that before you do anything you understand what you want to achieve, because how do you know that you have achieved, or not, if you didn’t understand what you wanted to achieve in the first place? Once you know what you want to achieve, and have outlined how you are going to achieve it, it’s easy to put in place a process to measure your progress. The information you collect needs to help you make decisions on your progress, whether you are on the right tract, side tracked, or have to change course, and what is most important is that it should help you answer the why, what now, and so what questions that we cannot answer now because we don’t know what impact the projects are supposed to make“. 

As I stepped to the front of the room, I had this inner voice that kept repeating my mentor’s words “…10 people trained…so what?” And these were my first words to the team of 14 people around me comprising of donors, partners and 6 beneficiaries of the Environmental Impacts Assessment (EIA) capacity building project. This Project seeks to promote EIA professionalism by providing a “learning by doing” environment for young indigenous African EIA professionals. The Project has three key strategic activities: i) EIA refresher training; ii) Placement with an organization undertaking an EIA on a major; and iii) Networking at national, regional and International forums.  This year, 2008, the Project is running its 4^th batch of training. However, since its start, there has not been an impacts monitoring system. So I was given three hours to kick-start the discussion on establishing a monitoring system from the Project’s impacts. And we starting with what it was that we wanted to monitor.

After some discussion and agreement that we were to focus on impacts (afterall it was clear from document on the number of outputs – no of people trained, how many and which trainings), we debated on who was to change and how the change was to be observed in terms of attitude and behaviour. It was amazing how easy an-impacts monitoring discussion gets when all stakeholders are involved in identification of change expected from intervention! The beneficiaries narrated how they wanted to change, and how it was to be measured.  The team also felt that with the success of the Project, they expected the donor to change e.g. to increase or expand its funding on such an initiative. Someone mentioned that the local communities who are stakeholders in EIA processes would change. After lengthy debate, the team agreed that that cannot be linked to the Project.

In a nutshell, the discussion for identifying impact indicators drove itself.  The representation of all stakeholders involved in the Project helped a great deal. The approach of discussing expected change in terms of attitude and behaviour as a result of Project interventions made the discussion clear and easy!.

  We were on Day 5 of discussing and designing a participatory M&E system…with lots of positive feedback on this “new way of doing things”. Many individuals had come up to us, or said in plenary how excited they were to be involved in designing a system where the information would help the farmer, the researcher, the extension officer in making their own decisions. 

In the managing for impact approach; we recognize that it’s important to identify up-front what incentives need to be put in place to motivate people to get actively involved in M&E. 

So - we asked individuals to put on cards what incentives they thought would be important for them. Once they’d done so - we turned over blank flip charts with titles of different types of incentives; and asked everyone to walk around and place their cards where they thought they best fitted:

• Financial
• Material (new equipment etc)
• Clear link to decision making
• Recognition (e.g. being invited to give a presentation, or a pat on the back..)
• Clear roles & responsibilities for M&E
• Opportunities for training/skill-building
• Others

So, until we find ways to restore faith and belief in aid projects - that …with a little hard work & integrity…they can make a positive difference to people’s lives ..they can put a dent in this endless cycle of “poverty”…

Until we can find ways to restore faith & belief in M&E systems …that the information collected will really be used to help people make better decisions about their own lives - and influence the decisions of others…. I’m afraid “posho” will continue to be the stronger motivator!! 

Since my post on “Logic beyond the Logical framework“, we’ve now almost finished consolidating the work of the 4 different groups and putting it altogether into one MandE matrix.

The first step was bringing together the theories of change as viewed by the different stakeholder groups into one Programme theory. This is what it looked like (different colors represent changes related to different stakeholder groups and the arrows illustrate linkages/relationships between changes - both within stakeholder groups and between them).

From this - the Prog. was then also able to revise their logframe by clustering the outputs, outcomes (changes in behavior) and overall impacts from the diagram - which feed into the first column of the logframe. The items in red are internal/external factors that may influence the success of this theory. These feed into the 4th column of the logframe (risks/assumptions). However, they are also used to identify ways in which to strengthen the logic and reduce the probability of the risks (the green boxes are recommendations for changes to the theory). **Remember - this wasn’t a planning workshop ..if it was - we would have started from the top (impacts) and worked our way down. Instead, we were simply trying to ensure that everyone had a clear understanding of the strategy already designed and, if necessary, identify areas in which it needed to be strengthened. 

As I mentioned before - we also used this to develop the M&E matrix. Each step in the theory of change needs to be monitored, as does the relationships between them as well as the internal/external factors.  So - when thinking about information required for performance questions such as “To what extent were the intended impacts contributed to, why, why not?” - the programme stakeholders referred to the theory of change they mapped out (e.g. changes in % of income originating from sales of agricultural products in households participating directly in the Prog.). This part of the M&E matrix looks something like this: 

The next step is to think through & plan for actual data collection. At impact level, it was felt that the data/information should be collected through both participatory impact assessments & externally conducted impact assessments. Here’s an example of the part of the matrix that relates to this: 

Last, but far from least,  will be information use - defining and planning for the forums & other mechanisms that will be put in place in order to enable critical reflection on and the use of the data/information collected for decision making. 

For months, no years (!) I’ve been looking for a good software that would help in documenting electronically the theories of change that are put together in planning & MandE workshops. I can’t afford the high end software that’s out there (SmartDraw/ConceptDraw/Visio amongst others) so I’ve been using Excel - which is a little cumbersome, but ok. 

Then, yesterday, I came across Concept Mapping which I found on another bloggers site. I find the thinking behind it extremely useful and insightful in my work and the software a drea

m come true (& it’s free!)! I highly recommend to anyone wanting to map out Programme strategies etc.!  

Many many thanks to the blogger at http://pkab.wordpress.com/about/! 

Project inspection teams while conducting their review missions, almost always, ask whether projects have a monitoring and evaluation system. On the hand, I feel, they fail to give  equal attention to what happens to the information collected and stored through the monitoring and evaluation system .  

This was the background which motivated us to develop and incorporate a critical reflection mechanism as part of a participatory M&E plan. We thought that having a deliberately prepared and implemented critical reflection events will insure that the collected information is used for learning and decision making.

This reflection relates to a process we followed to understand the existing information sharing avenues the different stakeholders of a public agricultural development project have and come up with suggestion to use these as departure point  in  creating a mechanism of critical reflection for the project.   

The process followed as follows;

Phase 1. Group work

1. Workshop participants were asked to form groups with their stakeholder colleagues and list down regular meetings/events organized by their stakeholder group and the frequency of these meetings/ events within the year on A4 card  (they indicated  frequency by producing copies of the meeting name  as number of meetings held  e. g . twelve copies for monthly,  four copies for quarterly, two for biannual, one for annually meetings)  etc..
2. They were  also  asked to write issues discussed and reports made  in these events  on the same card (different color cards were given to the different stakeholder groups)
3.  Facilitators prepared a timeline with months of the year on brown paper and spread on the floor.
4. Then  participants were asked to put their card as  it fits to the timeline  

A ten-day training will be held at Haramaya University, Ethiopia from the 15th to the 25th of September 2008. The workshop is for service providers and project staff interested in developing their skills and capacity to implement the Managing for Impact approach. 

The overall purpose of the workshop is to continue to develop and strengthen the MfI network in eastern & southern Africa to support pro-poor initiatives to effectively manage towards impact.

Specific objectives include:

•  Enhancing the skills and knowledge of service providers and practitioners on what managing for impact means and how to put it into practice; and
• Identifying service providers & practitioners willing to engage with SMIP through the MfI network to support pro-poor projects in the implementation of the managing for impact approach

The training workshop draws on adult learning principles. The approach used will be highly interactive in nature, involving both theory and real world experiences; and build on the existing knowledge and experiences of the participants. Participants will apply and reflect on some of the tools they learn about during a field visit in the second week of the course. The course content draws on the conceptual framework that guides the managing for impact approach and includes theories & backgrounds: methodologies & tools related to the following:

•  Paradigms influencing knowledge & development
• Systems, Institutions & Processes (SIPs)
• Creating learning environments
• Change management/managing change
• Strategic planning
• Operationalizing strategic plans
• Participatory Monitoring & Evaluation

For further information, please visit the SMIP ERIL site