Africa Evaluation Blog: My nine months of immersion in Participatory Impact Assessment & Learning (PIALA) in Africa

Glowen Kyei-Mensah

Posted 1 year ago

Sitting across the table from two representatives of soon to be clients, I didn’t think that this evaluation would be much different from many others we had worked on over the years. I knew that it would involve many days of participatory fieldwork but participation was the name of the work we constantly did – this was going to be business as usual.

After all, our organization is not called Participatory Development Associates just for name. We live, eat, drink and sleep participation, so what else is new on that front? Little did I know that 9 months of my life would be submerged in Participatory Impact Assessment & Learning Approach (PIALA), which soon attempted to change the P in our name (PDA) to ‘Pressure’. The PDA PIALA story began earlier in 2013 when we expressed interest in a Government of Ghana nationwide evaluation work. The application submission process was quite dramatic. We were given very short notice to put in a bid so Beatrice, a Senior Programmes Officer of PDA, found herself behind a motto rider in an attempt to beat heavy traffic to deliver our proposal in time. She made it 10 minutes before the deadline. We all exhaled and figured this was going to be the only stressful incident as far as this piece of work was concerned. No wonder, I sat at my meeting with the two reps in a relaxed manner, waiting to complete the contractual stage in order to get my hands dirty with work.

You must be wondering what is ‘PIALA’? This is the same question that as running through the heads of me and my evaluation team as we met the IFAD consultant Adinda Van Hemelrijck (who was to work with us throughout this process) for the first time. We soon found out that PIALA isn’t just a participatory methodology, it is a novel approach for impact evaluation called Participatory Impact Assessment & Learning Approach which was developed through an innovation initiative led by the IFAD consultant and funded by the International Fund for Agricultural Development (IFAD) and the Bill & Melinda Gates Foundation (BMGF).

PDA was commissioned by IFAD and the Government of Ghana to conduct the impact evaluation of the IFAD funded Roots and Tuber Improvement and Marketing Programme (RTIMP) in Ghana. I handled the evaluation management and coordination, while my team consisting of 3 team leaders (seasoned researchers) and 9 researchers (speaking multiple local languages) did the fieldwork. They were divided into 3 teams: one for the Northern part of Ghana, one working in the South, and one covering the Middle belt of the country. I worked closely with the IFAD Consultant to design and monitor the quality of the evaluation processes. As we at PDA take great pride in delivering quality, the P-factor standing for Participatory as well as Pressure, became our biggest challenge in this large-scale impact evaluation.

Once the team was in place, our first task was to ensure that the entire team had a basic grasp of what PIALA is and how it differs from other impact evaluation approaches. Adinda and I put together a plan for an initial 2-day introduction to PIALA training (we later carried out a more in-depth 5-day train ing and testing of tools). As the team leader, I had to read the literature and ask the consultant as many questions as necessary for me to get a good understanding of the approach, as I was asked to co-facilitate the training. My head was exploding, as I had many questions, most of which were never really answered until we were in the field carrying out the actual work. However, I had a nice trump card up my sleeve as team members were well versed in participatory methodologies. PIALA was a novel approach. As a matter of fact, PIALA had only ever been used in Vietnam where it was piloted in 2013, so I doubt there were more than a handful of evaluators around the world that were familiar with the approach. That made me and my team excited, we were going to be part of a new approach that would introduce us to exciting new ways of using participatory methodologies.

We soon learned that “PIALA is designed to produce rigorous quantitative and qualitative evidence and generate solid debate around such evidence in order to influence policy, planning, targeting and management for generating greater and more sustainable impact. Its purpose is threefold:

(a) to report on a project’s or program’s contributions to impact on rural poverty;

(b) to learn why impact occurred or not and where mechanisms need to be changed or newly created; and

(c) to debate how impact could be enhanced and future program investments could have a greater influence.

Different from process and performance evaluation approaches, PIALA focuses on ‘impact’ and ‘contributions to impact’, [which are] broader than the intended outcomes and performance against pre-set targets. Impact is viewed from a systemic perspective, as a system of interactions between various causes and changes, as opposed to a more linear approach that looks at the direct relationship between intervention and effect. The systemic approach seeks to move beyond a merely what works metrics and also answers the more difficult why and how questions. [It also] investigates the likely sustainability of the changes observed. [PIALA] does so by looking at both the intended and unintended, positive and negative, primary and secondary effects of a project or program relative to other influences that directly or indirectly contributed to the impact on rural poverty.” (Excerpt from Final Report on the participatory impact evaluation of the RTIMP authored by Adinda Van Hemelrijck and Glowen Kyei-Mensah).

It was our task to go across the country and carry this out. The impact evaluation was conducted nation-wide and full-scope. It was a mixed method study that was extremely participatory. We planned, budgeted and took a leap of faith in some instances, hoping to complete fieldwork in about 35 days on average. At PDA, we had never sent researchers into the field for more than 25 continuous days, so I was quite concerned about 35 days. Little did I know that one of our teams would end up spending a total of 56 days carrying out fieldwork.

The PIALA evaluation was highly stressful and full of ups and downs, but the team was determined to follow the process step by step. Once we completed the evaluation and the report was submitted, my team and I agreed that, yes indeed, PIALA is a superb approach we definitely would want to use in our future work, if opportunities emerge. It resonates with our reality, our principles and our practice, this with what we see as important at PDA.

Let me address the 5 main elements of PIALA and explain:

1. Theory of Change (ToC) – This was not completely new to the team at PDA. We have used a ToC approach before, so it wasn’t difficult to comprehend and use during the PIALA work. I find the systemic perspective of PIALA’s ToC approach, and the causal links and feedback loops it highlights, quite useful though, as it is pictorial and helps make it easy to focus and frame and thus operationalize the evaluation.

2. Multi stage sampling of/in market-bounded systems – We worked with a seasoned statistician who was as excited as we were to try something different. Most challenging was the identification of our principle sample unit. For sure it was NOT the household, since we were supposed to inquire supply chains of four different commodities (fresh yam, plywood cassava flour, gari and high quality cassava flour), each of which had a different population to sample from across the entire country. There was quite some variation in the organisation and geographic coverage of the supply chains, leaving our statistician with a not-easy task to define and sample them, and to sub-sample the households within their geographic areas. For him too, this turned into quite an interesting piece of work!

3. Participatory Mixed Methods – As PDA is known for participatory research and evaluation approaches, this was where the team excelled. It was interesting during the design, planning and training stage to replicate some of our best known PLA tools and adapt and systematise them for this PIALA impact evaluation without losing out on quality of process. Having researchers in the teams who know the local political economy and understand and speak the local languages of course helped them to build trust and handle power dynamics, even though they were working within tight time frames.

4. Participatory Sensemaking Workshops – We organised 23 district-level workshops with a total of 650 participants (of which 70 % beneficiaries), and one national workshop with a little over 100 participants (with 30 % beneficiaries). To us, these were much like validation workshops (Adinda will scream to hear me say this). It served as a forum for cross-validating and debating the evidence with all the different local, regional and national stakeholders. I found the sensemaking workshops quite useful as they were very engaging and inclusive – many more people that contributed to data collection were able to see some analysis of data from their districts and learned about the broader systemic interplay. Most useful for them was to engage in the process of validating the ToC (or refuting parts of it) as they reconstructed its pictorial diagram with the evidence and discussed and compared the differences across the different zones in the country. This is probably something that goes beyond a classic validation exercise. But it certainly goes with the PDA spirit of inclusion.

5. Configurational Analysis – This was REALLY new to us, and quite challenging. We code transcribe and analyse large amounts of qualitative data, and we have our ways of doing this quite rigorously. But this was different, as it entailed the comparison of a large amount of distinct data sets of “systems” in order to identify the causal patterns. So we found this really useful. We have since proposed its use in a few of our bids.

Reminiscing on the whole PIALA evaluation, I realize that it was one of the most fulfilling experiences of my work life. Why won’t I feel on top of the world when I received an email (in reference to this work) from one of the internationally acclaimed gurus of participatory methods which contained these words: “This is the most innovative combination of methods I remember ever having come across. It could be that I am simply out of date and out of touch. Probably, but I hope not entirely. All of you who have been involved – Adinda, PDA, Glowen, Irene, PDA colleagues and others deserve congratulations and thanks. The congratulations extend to IFAD and BMGF for having done what we have been advocating so long and fruitlessly with donors, to fund methodological innovation. I can’t suppress my excitement. What a pay-off!”