Evaluation


An evolving story on MSC-PV in Zanzibar

 Monday 27th July

 The story begins to unfold….

A few women, a few men, they wait patiently for the rest to arrive. Some have travelled long distances from the nearby island of Pemba. Others are already at home. The Programme Coordinator arrives, the workshop can begin. The story starts to unfold…..

 “Introduce yourself by telling us your story about your experience with stories” turns into a real storytelling competition. One story tells us about a lion that marries and finally eats a donkey.  In another story the lion gets trapped into a well by a rabbit. Another person tells us that one night she saw someone in the room holding a snake and letting it crawl over her body. Scared she called for her father to come. He lit the torch but did not see anything, even when being called the second time. She allowed herself not to be scared anymore and to let the snake crawl over her body….. Every story is different. Every story is unique. Story telling takes time, to develop, to share, translate and to listen to.

 The next stories people tell each other are about the most significant change that they have observed over the last 2 years in the lives of farmers involved in the programme. The five stories finally selected were mainly about how knowledge about improved farming practices led to higher production, be it bananas, vegetables or chickens. The one story selected out of these five was selected as the most significant of the five not only because of increased production, but also because of spill over effect to other farmers who learned from their fellow farmers.

 The last stories today were captured on video. Each person held the video camera, briefly interviewed one colleague and captured the response. This was a story about people learning to hold a camera and to use it, some for the first time. It was a story full of laughter.

 

Tuesday’28th July 2009

Confidence is growing

The day starts with reviewing the shots taken by participants the previous day. For some, this was the first time ever they held a video camera. Others were more experienced. Still, one never stops learning.

More exercises during the day were meant to build participants’ capacity to work with the most significant change (MSC) technique and to work with a video camera. We had the ‘show and tell’ exercise, where participants were requested to make a movie of an object they choose. Reviewing this led to a lot of learning about video making, such as ‘use a tripod’, ‘avoid zooming’, the ‘rule of thirds’ and the ‘180 degrees rule’.

The next exercise further developed participants’ capacity in MSC, now in combination with the video camera and with the use of a story board. In groups of five they developed a story board, they told and captured the MSC stories on video, selected the most significant of these and reviewed some of the footage taken. For the selected SC story they developed a story board and took cutaway shots for editing purposes. Again there was a steep learning curve, both in the MSC technique, as well as in movie making. Participants learned fast!

Editing was more difficult and time was really too limited to ‘master’ it. As we needed time to prepare for the field work we decided that editing skills would be further enhance after the field work. And that editing was not going to be part of the field work process, as we originally intended: to edit, together with farmers, the SC story with cutaway shots.

During these two days we have seen people grow very fast in their capacity to work with MSC and with video. Those who were shaking all over their body the previous day when holding the camera, were now confident and knew what to do. We were looking forward to see whether all these effort would bear fruit when working with farmers the next day.

 

Wednesday 29th July 2009

Field work starts and ends with laughter….and farmers share their stories!

We are all excited. We are going to the field! Now it’s time to practice the knowledge and skills that we have gained over the last 2 days. Some last minute arrangements, some last minute adaptations of the process (yes, we are going to film all the stories and not document the full story on paper!) and we are ready to take off. The bus departs and inside the air is trembling.

When everybody is cheerfully chatting away, the ‘mzungu’ is trying to catch up on some Swahili. Let’s start with counting. One, two, three, four…… moja, mbili, tatu, nne … Oh dear, that’s not easy. When trying to say the Swahili word for ‘five’ the whole bus is killing itself laughing… A minor mistake can lead to a word that really you don’t want to be using! Ok., let’s try again, counting up to ten…. Again the bus is in tears! Another minor mistake in ‘ten’ leads to an even more serious disaster! The ‘mzungu’ stops trying Swahili, feeling very embarrassed….. The group is in a top mood!

We have arrived, the real work begins. Workshop participants are collaborating very smoothly in their teams. They each work with one group of some eight farmers. In total there are four groups, two of which have been engaged in a chicken project, the other two groups have been working on cassava. To get people in the mood, some groups get the camera out and ask farmers to hold it, and to interview their fellow farmers using it! There is excitement all over. The ice is broken.

Then farmers are requested to draw a ‘story board’: to draw pictures on cards that symbolize key issues they would like to share in their story. This can assist them in developing and telling their story. The story they are requested to tell is about what they consider to be the most significant change in their life as a result of being engaged in the project. It doesn’t take long before chickens and stick people appear on the cards. The storytelling begins, and so does the video making. As real professionals the farmer field school facilitators work with the camera, some assisted by the real professional film makers from the communication department of the ministry of agriculture, livestock and environment. Some farmers need a little encouragement, others are proud to tell their most significant change story on video. All have a story to tell. And every story is different.

When all stories have been captured, the selection process begins. One facilitator gets participants to draw symbols for themselves – no one could do a better job than the farmers themselves. Then they vote for the one they consider to be the most significant of all stories heard. A discussion around their reasons for choice takes place to learn from each other. Then story which has been selected as the most significant of all gets reviewed: what can be improved in story telling, what additional cutaway shots can be taken? They storyteller again tells the MSC story, now with more information, taking advice from their colleagues on board.

It’s time for prayer. People leave the scene. When they come back lunch is being served. As there is no time left to capture the cutaway shots we decide to go ahead with showing the selected videos to the farmers. That means having lunch and capturing the movies on laptop at the same time! Storytelling takes time, so does capturing them on a computer. Editing is a step too far away – we don’t have the skills yet and time is beating us.

The sun outside is getting very hot, but luckily the nursery rooms are cool. Here the farmers can view the first selected change movies. Meanwhile, we are working hard to capture the other movies on laptop. It just works out fine in the end. Everybody gets a chance to see the 4 stories, and there is a discussion about each movie. Then the selection process begins, through voting on cards. Each of the four selected farmers gets a big clap for their efforts: Yolanda, Hamisi, Vuai and Shami. The last one is selected as the most significant of the four. Like the stories varied, the claps also did, from rain to thunder…..We all go home tired but satisfied.

The day ends with another blunder of the ‘mzungu’ trying the numbers in Swahili. She gives up when the whole bus bursts out in laughter again. The day ends like it began…..

 

Thursday 30th July 2009

Reflecting on and learning from the field work process

The day is settling in, and so is the tiredness of the field work. There is still a lot to do so we do more warming up exercises. We start reflecting on the field work. Some of the feedback of farmers is really encouraging. Even though initially they were a bit hesitant, in the end farmers liked and were even proud to be filmed! And they wanted to see all the movies as each farmer was filmed during their story telling! We agree to go back and also to share the other change movies with them. Feedback is critical for learning and success.

We discuss the whole field work process. The importance of documenting essential information is a big lesson. Another big lesson is that people should not select the story as the most significant one because they know the storyteller very well, because he or she has an important role to play in the group, or because he or she is involved in the same enterprise as them. Facilitation of the selection process is really crucial so that people know how to select. Also getting each person to explain their reasons for choice is important. It is not a tick and go exercise. Also it is not a competition…..

We also learned important lessons about significant changes in farmers’ lives. Such as the woman whose story was selected not because she increased her chicken production a lot (she didn’t), but because she was able to practice the appropriate skills and she set an example to other farmers. Sharing these reasons for choice with other farmers is very important for the learning process. Another farmer explained a negative change as a result of being engaged in the program. He did not have land but could use some land of his brother in law. However, as the farmer and his brother got trained in improved farming practices, the brother in law wanted his piece of land back to practice his newly gained knowledge and skills. As a result of the program, the storyteller gained knowledge but he lost the land he was using for agricultural production. After reflecting on this story, the farmers’ group decided they needed to assist this farmer. Capturing negative change stories is also important for managing impact.

It’s important to be open to unexpected change, as we learned in the process. Whilst many farmers indicated increased production as a result of being engaged in the program, the result of this was different for different farmers.  Some mentioned that now they were able to eat chickens whenever they wanted to, another mentioned many changes but the indicated the ability to build a house as the most significant of all changes.

When participants were asked to review each others’ formats for story collection and for documenting the selection process, they realized the importance of good documentation. There were many gaps, such as not documenting the most significant change or why this was chosen as their most significant change. Practice, review and adapted practice is essential for proper learning.

 The day ended with some looking forward – thinking through system for MSC-PV in the program. We realized that in a program that targets some 7000 farmers it is impossible to capture all stories. We agreed on a pilot. As each of then nine districts in Zanzibar and Pemba had sent one representative farmer field school (FFS) facilitator to this workshop, we agreed to start working with these trained people in the pilot phase. The other trained participants would support them in their efforts, either in terms of facilitation or in documenting the selected stories on video. Particularly the trained participants from the communication department of the ministry of agriculture, livestock and environment would be useful in the filming and editing process. Each facilitator would be working with 6 groups of 5 farmers, selected from the 3 FFS groups they are working with. The SC stories would be captured on paper, and the selected stories would be captured on video.. one story per farmers’ group. This design process was continued the next day…..

 

 Friday 31st July 2009

Ending with inspirations for the future, beginning the story of implementation ….

It’s the last day of the MSC-PV workshop. We have gone through so much together. We have seen each other grow in confidence as we tried, reflected and put our learning into practice in the next exercise. Now it’s time to consolidate our learning and think about the future. How can we set up a system where most significant change stories can be captured? How can we use (participatory) video with this? And how can we set up the system so that it complements our current efforts in M&E? This was what the last day of the workshop was about. We developed a draft system for MSC-PV for the program. This can easily be integrated into the current M&E system. At the end the M&E officer Mr Lada indicated: ‘we feel empowered that we have developed this ourselves!’

We then had time to continue practicing our editing skills. It started with viewing a compilation of video shots, mainly from the field work. This can be viewed at (to be uploaded soon!). The groups then practiced some editing skills on the selected SC story from the farmers’ group they worked with. Even though we were not able to capture the final versions of these edited movies, you can see the unedited versions on (to be uploaded soon!). We agreed that farmers would get a chance to see all the stories as every SC story had been captured on video. In the near future the district resource centres will provide the opportunity for farmers to come and view the films of the SC stories, any time they want. And possibly editing can be done locally. As much as possible the whole process needs to be done as close as possible to farmers’ level, in collaboration with the farm field school facilitators.

The day ended with reflecting on the workshop. Workshop facilitators went outside, and participants facilitated their own evaluation, using an evaluation wheel. Generally people felt empowered to facilitate MSC and use video for this purpose. They would have liked more time: 2 or even 3 weeks! It was encouraging to see people grow in confidence. Some were interested to learn even more about participatory video. And Mr Lada said –  ‘Now we ready to implement and to train the rest of the staff in MSC-PV ourselves!’ And so their story begins here….

The African Evaluation Association (AfrEA), the Network of Networks on Impact Evaluation (NONIE) and the International Initiative for Impact Evaluation (3ie) have joined forces to bring to Africa some of the best expertise from all continents on one of the most discussed topics among evaluation and development communities worldwide.

The conference defines impact evaluations as those studies which concern themselves with determining and understanding the short, medium and long term outcomes or impacts of projects, programs and policies. The term is not limited to any specific methodology in any particular discipline. (more…)

Dear All,

Last week, I was deeply privileged to be invited to this blog by Mine, and sure enough my expectations were way below what I read on the blog! I shared with Mine my first impressions soon after skimming through the rich experineces shared on this forum, and particularly had my attention caught by the metaphor of “craftsmanship” / “craftswomenship”.

I was lately reflecting over the metaphor of craftsmanship / craftswomanship, and for me, this brought memories of past times when we (in one of the projects using PRA techniques) were driven by techniques rather than the development issues / questions. Naively, we churned up maps, chapati diagrams, timelines and all tribes of matrices, certainly exciting and drawing mamoth crowds of communities, but certainly – at the best simply skirting around the real development issues! Thankfully, the project registered some unintended positive gains, but could have perfomed better on the scale of her objectives.

The talk about craftsmanship also reminds me of glaring mismatches / disparities in some of the publications we have; often times, data on a particular parameter, sometimes from the same source and for the same period has outright differences…4 years ago, I juxtaposed data on adult literacy in Uganda, compiled into a leaflet (developed with reference made to “How to lie with statistics”  – click on illustration for full view)

 

\

\

I was baffled by these “contradictions”, and loudly posed questions about the perplection: could it be that we view the same parameter from very different planes, and hence our definition / description of the same parameter is extremely different? Is it that we are using very different sampling frames, different sample questions, generalising very different sample answers, or could the “problem” be emanating from using different statistical tools (eg median, mode, mean for average)? The latter could be partly true, graduating some of our calculations to “manipulations” to suit a particular audience for a specific request (politically, state very high literacy rates registered, but to potential funders, state much lower figures etc).

My contention will always be that as the craftsmen and craftswomen get more exposed to, and equipped with the necessary tools, that we still have the development question as our guide (to determine what combination of tools to use, and not vice versa) and that we do not use the tools for (wo)manipulating the audience… just thoughts stemming from the crafting metaphor (apologies for this digression from your focussed discussions)

On another note, I would like to believe you are already in the know about the forthcoming European Evaluation Society Conference in Lisbon – late September to early October 2008. Is there anyone on this blog planning to participate? I think it would be of interest as impact evaluation will be given thrust – with active participation of NONIE, DFID and probably 3IE… Just in case you need more information about the same (if it is not too late), I will be happy to send / share more.

Once agian, I an honoured to be a part of the “family” of this blog, and as earlier mentioned to Mine, would be certainly one on the learning curve, always gleaning a learning from the rich experiences that you all have.

Great many thanks and kind regards

 Simon Kisira

Further to Thevan’s post on Impact evaluations; here’s an excerpt from a very interesting article on “Evaluation Evolution” posted in the Broker magazine the other day (link to full article below);

Three approaches to evaluation

Evaluation evolution?

Politicians are calling for evaluations that measure the effects of development cooperation. However, good development cooperation focuses on long-term processes that cannot be measured in terms of cause and effect. Alternative approaches to evaluation are needed.

By Otto Hospes 

Development cooperation is one of the most evaluated areas in public policy. Over the past 30 years, several studies of evaluator types and their approaches have been undertaken.1 But there are three approaches that fundamentally characterize and distinguish types of evaluators: evidence-oriented evaluation, which seeks hard evidence; realistic evaluation, which tests how and why outcomes of policy occur; and complexity evaluation, which focuses on the complexity of social issues and governance (see table).

The evidence-oriented approach is the most dominant in development cooperation evaluation, but it is not necessarily the most illuminating. The realistic approach, meanwhile, is mainly applied in the fields of justice, health and social services in European countries. According to some theorists, however, a shift is occurring away from evidence-oriented to realistic and complexity evaluation. The importance of the social and political contexts in which a policy is employed is increasingly recognized. This context is dynamic, complex and multilayered, with many different agencies and networks involved.

Complexity evaluation is related to the more recent use of complexity theory in social science. This emerging approach may provide useful insights to help overcome serious flaws in current evaluation practice, particularly in developing countries.2″

Read more.. 

Rural women in Manica, Mozambique expressing themselves through dramaI recently attended a seminar organised by SAMEA (the South African Monitoring and Evaluation Association) on the strengths and weaknesses of conducting Impact Evaluations (IE) considering that there is a tendency to conduct IE towards the end of a programme or project, or even not at all! The guest speaker at the seminar was Dr. Howard White, an internationally reknowned evaluation guru who has led the World Bank’s IE programme and trained close to 2000 DfID staff on M&E, and most recently was instrumental in establishing the International Initiative for Impact Evaluation (3ie). Phew, quite an impressive resume!!
In any event the seminar was extremely informative in that it raised several key issues that either support or contradict the MfI approach. I hope that this little report- back and some of the questions that I’ve raised from those issues that I found particularly useful, stimulates some debate amongst us MfI converts!
Dr. White gave a substantial input into the origins of IE, namely the need for the big and powerful international aid agencies (USAID for the USA and DfiD for the UK) to justify their billions of dollars worth of investments in terms of the difference this has made on global development. There’s such a proliferation of information and knowledge generated from IE’s since the 1990’s, yet very little (in my opinion) has changed with regards to global poverty. Is this a question of how much, or how often, or even when we analyse poverty alleviation goals through IE’s, or is it a question of the poor quality of the analysis of poverty used to inform policy change??  Well into the 21st cenury very same challenge remains intact- how do we really know that we are making a difference!
The speaker suggests that the logframe (although limited to logical and linear reasoning in the causal chain), it is without a doubt the most feasible mechansim by which to measure and show the direct links between inputs and outcomes. Granted that its main critique is that it is highly prositivist in orientation, the stark reality of our time is that policy making itself is undeniably highly prositivist in approach. The logframe helps us to understand how it is that X leads to Y in the most cost- effective and efficient way, which the non- positivist approach doesn’t seem to do. As someone steeped in the non- positivist approach to conducting IEs, I found this argument very difficult to digest, and yet at the same time I found myself wondering whether despite all our rhetoric on the dangers of positivism and the wonders of participatory M&E, we are still accustomed to working this way- it is after all quick-and-dirty! So let’s be honest, do we really engage with time- consuming and expensive participatory M&E that has at its core the empowerment of the poor and marginalised??
There were other interesting arguments put forward, but in the meantime I found the issues raised in this blog the most stimulating. It would be great to hear other practitioners’ responses to whether as development practitioners we are genuinely shifting the goal posts of poverty alleviation through M&E, and IE in particular.

I must confess. After Kenya’s 2007 elections – the work I am doing with development aid didn’t make much sense anymore. It seemed to me that no amount of “development aid” can make a positive change in peoples lives as long as; i) corruption went unchecked; ii) the bulk of national budgets went to line the mansions of the leaders; and iii) people lost their property and lives every election cycle and had to start all over again. Development aid started to seem very similar to pouring water into a pot full of holes. The minute you take a breath – it gushes out to the ground. 

If this is the case, – how can strategic planning, monitoring, learning, and all our well-intentioned initiatives in SMIP & MfI make any difference? To me, it was (and still is) extremely frustrating and I even considered taking up shop-keeping instead! 

I wrote about these sentiments to Irene Guijt (one of the co-author’s of the IFAD Guide to Project M&E) who’s work and opinion I highly respect. As part of her response, she sent me a link to the International Budget Project, that works to build civil society’s capacity to “analyze and influence government budget processes, institutions and outcomes.” The project’s overall aim is to “make budget systems more responsive to the needs of society and, accordingly, to make these systems more transparent and accountable to the public.” In Africa, the project is working with the Uganda Debt Network. 

Reading about their work was very exciting – particularly the involvement of civil society groups. For Government development initiatives such as the Kenya’s Constituency Development Fund (CDF) or even loan financed initiatives – where tax payers in the recipient countries “pay the price” (so to speak) – this seems critical! And in engaging civil society/non-government actors in processes where they are in a position to hold their leaders accountable – surely this in turn influences their own ways of viewing governance & leadership? Perhaps contributing to a shift from – “we have no choice but to accept what our leaders tell us” to “we do have a choice and a responsibility”. 

Has anyone else come across similar initiatives? If so, please do share them with us all…after all – those working in development in Africa must have bumped into the irresponsible governance issue at once in their lives (if not regularly)!

Mine

 

I recently read a paper on “The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement“. This, other papers, and my own experiences all point to the fact that evaluation practice is rapidly evolving. We’re seeing a gradual shift in the dominant paradigms – which includes the need to view things from the point of view of those that are directly benefitting (or not!) from a development initiative. Also – the need to take into account PROCESS (how we do things) and not simply WHAT we do and WHAT is achieved. 

We also see this in a number of aid instruments (such as the Paris Declaration or Sector Wide Approaches) which seem to support the shift towards greater ownership of the recipients of aid (and therefore greater responsibility). But – the question is ..do they enable both recipients and funding agencies to gain knowledge and the learning that they need in order to take up these new responsibilities in a world of shifting paradigms? Robert Chambers’ paper on “Poverty Unperceived: Traps Biases and Agenda” really got me thinking about this. 

Have a read and please share your thoughts! 

Until soon,

Mine