Developing M&E systems


An evolving story on MSC-PV in Zanzibar

 Monday 27th July

 The story begins to unfold….

A few women, a few men, they wait patiently for the rest to arrive. Some have travelled long distances from the nearby island of Pemba. Others are already at home. The Programme Coordinator arrives, the workshop can begin. The story starts to unfold…..

 “Introduce yourself by telling us your story about your experience with stories” turns into a real storytelling competition. One story tells us about a lion that marries and finally eats a donkey.  In another story the lion gets trapped into a well by a rabbit. Another person tells us that one night she saw someone in the room holding a snake and letting it crawl over her body. Scared she called for her father to come. He lit the torch but did not see anything, even when being called the second time. She allowed herself not to be scared anymore and to let the snake crawl over her body….. Every story is different. Every story is unique. Story telling takes time, to develop, to share, translate and to listen to.

 The next stories people tell each other are about the most significant change that they have observed over the last 2 years in the lives of farmers involved in the programme. The five stories finally selected were mainly about how knowledge about improved farming practices led to higher production, be it bananas, vegetables or chickens. The one story selected out of these five was selected as the most significant of the five not only because of increased production, but also because of spill over effect to other farmers who learned from their fellow farmers.

 The last stories today were captured on video. Each person held the video camera, briefly interviewed one colleague and captured the response. This was a story about people learning to hold a camera and to use it, some for the first time. It was a story full of laughter.

 

Tuesday’28th July 2009

Confidence is growing

The day starts with reviewing the shots taken by participants the previous day. For some, this was the first time ever they held a video camera. Others were more experienced. Still, one never stops learning.

More exercises during the day were meant to build participants’ capacity to work with the most significant change (MSC) technique and to work with a video camera. We had the ‘show and tell’ exercise, where participants were requested to make a movie of an object they choose. Reviewing this led to a lot of learning about video making, such as ‘use a tripod’, ‘avoid zooming’, the ‘rule of thirds’ and the ‘180 degrees rule’.

The next exercise further developed participants’ capacity in MSC, now in combination with the video camera and with the use of a story board. In groups of five they developed a story board, they told and captured the MSC stories on video, selected the most significant of these and reviewed some of the footage taken. For the selected SC story they developed a story board and took cutaway shots for editing purposes. Again there was a steep learning curve, both in the MSC technique, as well as in movie making. Participants learned fast!

Editing was more difficult and time was really too limited to ‘master’ it. As we needed time to prepare for the field work we decided that editing skills would be further enhance after the field work. And that editing was not going to be part of the field work process, as we originally intended: to edit, together with farmers, the SC story with cutaway shots.

During these two days we have seen people grow very fast in their capacity to work with MSC and with video. Those who were shaking all over their body the previous day when holding the camera, were now confident and knew what to do. We were looking forward to see whether all these effort would bear fruit when working with farmers the next day.

 

Wednesday 29th July 2009

Field work starts and ends with laughter….and farmers share their stories!

We are all excited. We are going to the field! Now it’s time to practice the knowledge and skills that we have gained over the last 2 days. Some last minute arrangements, some last minute adaptations of the process (yes, we are going to film all the stories and not document the full story on paper!) and we are ready to take off. The bus departs and inside the air is trembling.

When everybody is cheerfully chatting away, the ‘mzungu’ is trying to catch up on some Swahili. Let’s start with counting. One, two, three, four…… moja, mbili, tatu, nne … Oh dear, that’s not easy. When trying to say the Swahili word for ‘five’ the whole bus is killing itself laughing… A minor mistake can lead to a word that really you don’t want to be using! Ok., let’s try again, counting up to ten…. Again the bus is in tears! Another minor mistake in ‘ten’ leads to an even more serious disaster! The ‘mzungu’ stops trying Swahili, feeling very embarrassed….. The group is in a top mood!

We have arrived, the real work begins. Workshop participants are collaborating very smoothly in their teams. They each work with one group of some eight farmers. In total there are four groups, two of which have been engaged in a chicken project, the other two groups have been working on cassava. To get people in the mood, some groups get the camera out and ask farmers to hold it, and to interview their fellow farmers using it! There is excitement all over. The ice is broken.

Then farmers are requested to draw a ‘story board’: to draw pictures on cards that symbolize key issues they would like to share in their story. This can assist them in developing and telling their story. The story they are requested to tell is about what they consider to be the most significant change in their life as a result of being engaged in the project. It doesn’t take long before chickens and stick people appear on the cards. The storytelling begins, and so does the video making. As real professionals the farmer field school facilitators work with the camera, some assisted by the real professional film makers from the communication department of the ministry of agriculture, livestock and environment. Some farmers need a little encouragement, others are proud to tell their most significant change story on video. All have a story to tell. And every story is different.

When all stories have been captured, the selection process begins. One facilitator gets participants to draw symbols for themselves – no one could do a better job than the farmers themselves. Then they vote for the one they consider to be the most significant of all stories heard. A discussion around their reasons for choice takes place to learn from each other. Then story which has been selected as the most significant of all gets reviewed: what can be improved in story telling, what additional cutaway shots can be taken? They storyteller again tells the MSC story, now with more information, taking advice from their colleagues on board.

It’s time for prayer. People leave the scene. When they come back lunch is being served. As there is no time left to capture the cutaway shots we decide to go ahead with showing the selected videos to the farmers. That means having lunch and capturing the movies on laptop at the same time! Storytelling takes time, so does capturing them on a computer. Editing is a step too far away – we don’t have the skills yet and time is beating us.

The sun outside is getting very hot, but luckily the nursery rooms are cool. Here the farmers can view the first selected change movies. Meanwhile, we are working hard to capture the other movies on laptop. It just works out fine in the end. Everybody gets a chance to see the 4 stories, and there is a discussion about each movie. Then the selection process begins, through voting on cards. Each of the four selected farmers gets a big clap for their efforts: Yolanda, Hamisi, Vuai and Shami. The last one is selected as the most significant of the four. Like the stories varied, the claps also did, from rain to thunder…..We all go home tired but satisfied.

The day ends with another blunder of the ‘mzungu’ trying the numbers in Swahili. She gives up when the whole bus bursts out in laughter again. The day ends like it began…..

 

Thursday 30th July 2009

Reflecting on and learning from the field work process

The day is settling in, and so is the tiredness of the field work. There is still a lot to do so we do more warming up exercises. We start reflecting on the field work. Some of the feedback of farmers is really encouraging. Even though initially they were a bit hesitant, in the end farmers liked and were even proud to be filmed! And they wanted to see all the movies as each farmer was filmed during their story telling! We agree to go back and also to share the other change movies with them. Feedback is critical for learning and success.

We discuss the whole field work process. The importance of documenting essential information is a big lesson. Another big lesson is that people should not select the story as the most significant one because they know the storyteller very well, because he or she has an important role to play in the group, or because he or she is involved in the same enterprise as them. Facilitation of the selection process is really crucial so that people know how to select. Also getting each person to explain their reasons for choice is important. It is not a tick and go exercise. Also it is not a competition…..

We also learned important lessons about significant changes in farmers’ lives. Such as the woman whose story was selected not because she increased her chicken production a lot (she didn’t), but because she was able to practice the appropriate skills and she set an example to other farmers. Sharing these reasons for choice with other farmers is very important for the learning process. Another farmer explained a negative change as a result of being engaged in the program. He did not have land but could use some land of his brother in law. However, as the farmer and his brother got trained in improved farming practices, the brother in law wanted his piece of land back to practice his newly gained knowledge and skills. As a result of the program, the storyteller gained knowledge but he lost the land he was using for agricultural production. After reflecting on this story, the farmers’ group decided they needed to assist this farmer. Capturing negative change stories is also important for managing impact.

It’s important to be open to unexpected change, as we learned in the process. Whilst many farmers indicated increased production as a result of being engaged in the program, the result of this was different for different farmers.  Some mentioned that now they were able to eat chickens whenever they wanted to, another mentioned many changes but the indicated the ability to build a house as the most significant of all changes.

When participants were asked to review each others’ formats for story collection and for documenting the selection process, they realized the importance of good documentation. There were many gaps, such as not documenting the most significant change or why this was chosen as their most significant change. Practice, review and adapted practice is essential for proper learning.

 The day ended with some looking forward – thinking through system for MSC-PV in the program. We realized that in a program that targets some 7000 farmers it is impossible to capture all stories. We agreed on a pilot. As each of then nine districts in Zanzibar and Pemba had sent one representative farmer field school (FFS) facilitator to this workshop, we agreed to start working with these trained people in the pilot phase. The other trained participants would support them in their efforts, either in terms of facilitation or in documenting the selected stories on video. Particularly the trained participants from the communication department of the ministry of agriculture, livestock and environment would be useful in the filming and editing process. Each facilitator would be working with 6 groups of 5 farmers, selected from the 3 FFS groups they are working with. The SC stories would be captured on paper, and the selected stories would be captured on video.. one story per farmers’ group. This design process was continued the next day…..

 

 Friday 31st July 2009

Ending with inspirations for the future, beginning the story of implementation ….

It’s the last day of the MSC-PV workshop. We have gone through so much together. We have seen each other grow in confidence as we tried, reflected and put our learning into practice in the next exercise. Now it’s time to consolidate our learning and think about the future. How can we set up a system where most significant change stories can be captured? How can we use (participatory) video with this? And how can we set up the system so that it complements our current efforts in M&E? This was what the last day of the workshop was about. We developed a draft system for MSC-PV for the program. This can easily be integrated into the current M&E system. At the end the M&E officer Mr Lada indicated: ‘we feel empowered that we have developed this ourselves!’

We then had time to continue practicing our editing skills. It started with viewing a compilation of video shots, mainly from the field work. This can be viewed at (to be uploaded soon!). The groups then practiced some editing skills on the selected SC story from the farmers’ group they worked with. Even though we were not able to capture the final versions of these edited movies, you can see the unedited versions on (to be uploaded soon!). We agreed that farmers would get a chance to see all the stories as every SC story had been captured on video. In the near future the district resource centres will provide the opportunity for farmers to come and view the films of the SC stories, any time they want. And possibly editing can be done locally. As much as possible the whole process needs to be done as close as possible to farmers’ level, in collaboration with the farm field school facilitators.

The day ended with reflecting on the workshop. Workshop facilitators went outside, and participants facilitated their own evaluation, using an evaluation wheel. Generally people felt empowered to facilitate MSC and use video for this purpose. They would have liked more time: 2 or even 3 weeks! It was encouraging to see people grow in confidence. Some were interested to learn even more about participatory video. And Mr Lada said –  ‘Now we ready to implement and to train the rest of the staff in MSC-PV ourselves!’ And so their story begins here….

Advertisements

More often than not, I come across a type of monitoring I’ve started to refer to as “tick & go” (many thanks to an MfI network member for the expression). This typically begins with a frantic scramble to develop SMART indicators which sometimes (but rarely) are linked to a clearly defined and understood programme logic. Indicators are then duly listed in a neat & tidy table with columns for targets & actuals (annual & cumulative targets) – preferably expressed as numbers or percentages.

This is all very well and I can most certainly appreciate why it’s so popular. When you’re running around supervising a number of different projects/programmes..have only a couple of days to deliver a report and too many reports to read – all you really want to do is “tick & go”! Unfortunately though, numbers & ticks are can go so far in helping us understand (and make decisions) within the complex situations most programmes are having to work in. (more…)

The SMIP team is currently in the process of supporting the Zanzibar ASSP & ASDP-L projects to roll out their participatory and learning oriented M&E Plan. One of the first activities was to try and build the capacity of and motivate District Facilitators (extension officers, subject matter specialists etc) to facilitate the Farmer Field School (FFS) monitoring sessions.We did this through training – consisting of 1 1/2 days in a “class-room setting” and 4 1/2 days in the field working with the farmers. (more…)

“Stop asking us the why, what now, and so what questions. Because we are not used to answering questions; we’re used to just implementing what we’re told”. This was a reaction by one of the participants of the MfI workshop that was facilitated for the officials of the Department of Social Development in the Northern Cape province South Africa.

The training was specifically organised for this group and these are Development Practitioners, whose responsibility is to oversee and supervise implementation of developmental projects in local communities of the Northern Cape. At the start of the workshop they felt that the subject and the MfI concept was fun and easy to grasp, and it is something that can easily be adapted to their current way of doing things, and this was before the Theory of Change and M&E were introduced to them. The feeling was however different after the introduction of the Theory of Change and M&E, as they felt that these concpts were completely shaking their “comfort zone” (project implementation mode, and monitoring project implementation rather than measuring the impact of the project on communities).

What was fascinating with this workshop was the fact that the participants felt that the workshop had absolutely changed their mindset around how they have always viewed development, and even though they found M&E a challange, what was interesting was the fact that they understood that they found it challanging not because it is a diffuclt process, but because it’s a process that they had never seen as part of project implementation. They also struggled to understand how Theory of Change informs the monitoring and evaluation approach and process, and because their theory of change as a department has always focused on project implementation, it was nerve-wrecking for them to understand how then does one monitor and evaluate the impact of the project on communities, when the theory of change was never about making impact on communities.

The projects that are designed by the department of Social Development, that are supposedly aimed at reducing poverty, are mostly based on the national masterplan, and this means that all provinces implement the same type of projects across the country irrespective of the local conditions and capacities. The also is not a clear Theory of Change, which specific outcomes, and objectives, and in the process what then happens is that officials then adopt a project implementation mode as this is how their performance is maesured. their performance is not measured based on whether the project has had any impact in incraesing people’livelihoods, but rather on whether the project is up and running, irrespective of its relevance or non thereof.

One positive criticism about Mfi though is that there is need to further simplify M&E, as it currently still has very strong elements of scientific research process, and yes maybe we cannot avoid this completely. What we then need to do is design different M&E modules for different audiences. This recommendation is made at the back of the workshop with government officials, where they expressed that because of MfI they understand the value and role of M&E, however they do feel that it is too “technical” for them, and this scares them off.

One of the participants when asked what he understood by M&E, said “M&E is a process where one collects information about what they do, to assess whether they are still on the right tract.  It is therefore important that before you do anything you understand what you want to achieve, because how do you know that you have achieved, or not, if you didn’t understand what you wanted to achieve in the first place? Once you know what you want to achieve, and have outlined how you are going to achieve it, it’s easy to put in place a process to measure your progress. The information you collect needs to help you make decisions on your progress, whether you are on the right tract, side tracked, or have to change course, and what is most important is that it should help you answer the why, what now, and so what questions that we cannot answer now because we don’t know what impact the projects are supposed to make“. 

 

 

 

 

As I stepped to the front of the room, I had this inner voice that kept repeating my mentor’s words “…10 people trained…so what?” And these were my first words to the team of 14 people around me comprising of donors, partners and 6 beneficiaries of the Environmental Impacts Assessment (EIA) capacity building project. This Project seeks to promote EIA professionalism by providing a “learning by doing” environment for young indigenous African EIA professionals. The Project has three key strategic activities: i) EIA refresher training; ii) Placement with an organization undertaking an EIA on a major; and iii) Networking at national, regional and International forums.  This year, 2008, the Project is running its 4th batch of training. However, since its start, there has not been an impacts monitoring system. So I was given three hours to kick-start the discussion on establishing a monitoring system from the Project’s impacts. And we starting with what it was that we wanted to monitor.

 

After some discussion and agreement that we were to focus on impacts (afterall it was clear from document on the number of outputs – no of people trained, how many and which trainings), we debated on who was to change and how the change was to be observed in terms of attitude and behaviour. It was amazing how easy an-impacts monitoring discussion gets when all stakeholders are involved in identification of change expected from intervention! The beneficiaries narrated how they wanted to change, and how it was to be measured.  The team also felt that with the success of the Project, they expected the donor to change e.g. to increase or expand its funding on such an initiative. Someone mentioned that the local communities who are stakeholders in EIA processes would change. After lengthy debate, the team agreed that that cannot be linked to the Project.

 

In a nutshell, the discussion for identifying impact indicators drove itself.  The representation of all stakeholders involved in the Project helped a great deal. The approach of discussing expected change in terms of attitude and behaviour as a result of Project interventions made the discussion clear and easy!.

  We were on Day 5 of discussing and designing a participatory M&E system…with lots of positive feedback on this “new way of doing things”. Many individuals had come up to us, or said in plenary how excited they were to be involved in designing a system where the information would help the farmer, the researcher, the extension officer in making their own decisions. 

In the managing for impact approach; we recognize that it’s important to identify up-front what incentives need to be put in place to motivate people to get actively involved in M&E. 

So – we asked individuals to put on cards what incentives they thought would be important for them. Once they’d done so – we turned over blank flip charts with titles of different types of incentives; and asked everyone to walk around and place their cards where they thought they best fitted:

  • Financial
  • Material (new equipment etc)
  • Clear link to decision making
  • Recognition (e.g. being invited to give a presentation, or a pat on the back..)
  • Clear roles & responsibilities for M&E
  • Opportunities for training/skill-building
  • Others
I suppose you will not be too surprised to hear that the flip charts titled “financial” and “material” incentives were filled to the brim! 
Together, we reflected on this and the discussion gradually led us to development in general and the way we view it….
Now, it’s all very well for us to stand there looking down on people asking for more per diems/DSA…or a top-up on their salaries or any other form of “posho”! 
But…when I think about it a little – it makes a lot of sense. We’ve all read the endless criticism about development – and how it’s not making much of a difference. So – perhaps…over time – it’s gradually arrived at a point where the benefits of development aid projects are limited simply to “posho”.. make a quick buck while you can before the project ends! And honestly – when you think about the farmers listening to programme staff who’ve just driven up in their snazy 4-wheel drives or the “experts” flown in from abroad staying in posh(o) hotels….. you can see the message being given off right? Development project = posho! 

So, until we find ways to restore faith and belief in aid projects – that …with a little hard work & integrity…they can make a positive difference to people’s lives ..they can put a dent in this endless cycle of “poverty”…

Until we can find ways to restore faith & belief in M&E systems …that the information collected will really be used to help people make better decisions about their own lives – and influence the decisions of others…. I’m afraid “posho” will continue to be the stronger motivator!! 

Since my post on “Logic beyond the Logical framework“, we’ve now almost finished consolidating the work of the 4 different groups and putting it altogether into one MandE matrix.

The first step was bringing together the theories of change as viewed by the different stakeholder groups into one Programme theory. This is what it looked like (different colors represent changes related to different stakeholder groups and the arrows illustrate linkages/relationships between changes – both within stakeholder groups and between them).

ASSP ToC

From this – the Prog. was then also able to revise their logframe by clustering the outputs, outcomes (changes in behavior) and overall impacts from the diagram – which feed into the first column of the logframe. The items in red are internal/external factors that may influence the success of this theory. These feed into the 4th column of the logframe (risks/assumptions). However, they are also used to identify ways in which to strengthen the logic and reduce the probability of the risks (the green boxes are recommendations for changes to the theory). **Remember – this wasn’t a planning workshop ..if it was – we would have started from the top (impacts) and worked our way down. Instead, we were simply trying to ensure that everyone had a clear understanding of the strategy already designed and, if necessary, identify areas in which it needed to be strengthened. 

As I mentioned before – we also used this to develop the M&E matrix. Each step in the theory of change needs to be monitored, as does the relationships between them as well as the internal/external factors.  So – when thinking about information required for performance questions such as “To what extent were the intended impacts contributed to, why, why not?” – the programme stakeholders referred to the theory of change they mapped out (e.g. changes in % of income originating from sales of agricultural products in households participating directly in the Prog.). This part of the M&E matrix looks something like this: 

Information needsThe next step is to think through & plan for actual data collection. At impact level, it was felt that the data/information should be collected through both participatory impact assessments & externally conducted impact assessments. Here’s an example of the part of the matrix that relates to this: 

 

 

Data collection methods

 

Last, but far from least,  will be information use – defining and planning for the forums & other mechanisms that will be put in place in order to enable critical reflection on and the use of the data/information collected for decision making. 

Next Page »