Using participatory impact diagrams to evaluate a community development project in Kenya

Participatory approaches for impact assessment are increasingly becoming popular with development organisations for engaging multiple stakeholders. We present our use of participatory impact diagrams as an evaluation tool within a mixed methods impact assessment of several drought-reduction interventions in Kenya. Results show that because men and women have different roles, their experiences of interventions vary. We discuss how this methodology encouraged communities to describe various intervention outcomes including unintended impacts, often overlooked by conventional impact assessment approaches. Methodological challenges included the integration of quantitative data; opportunities for its application within the wider discipline of monitoring and evaluation are considered.

Utilisation des diagrammes d'impact participatives pour évaluer un projet de développement communautaire au Kenya

Les approches participatives de l’évaluation de l'impact sont de plus en plus populaires parmi les organisations de développement pour faire participer des parties prenantes multiples. Nous présentons notre utilisation des diagrammes participatifs des impacts comme un outil d’évaluation dans le cadre d'une évaluation multi-méthodes de l'impact de plusieurs interventions de réduction des inondations au Kenya. Les résultats montrent que, les hommes et les femmes ayant des rôles différents, leurs expériences des interventions varient. Nous discutons de la manière dont cette méthodologie a encouragé les communautés à décrire divers résultats des interventions, y compris les impacts non désirés, souvent oubliés par les approches conventionnelles d’évaluation de l'impact. Parmi les défis méthodologiques figurait l'intégration des données quantitatives ; les occasions de les mettre en application dans le cadre de la discipline du suivi et de l’évaluation sont examinées.

Utilizando diagramas participativos de impacto para evaluar un proyecto de desarrollo comunitario en Kenia

Con el objetivo de involucrar a diversos actores, las organizaciones de desarrollo utilizan cada vez más los enfoques participativos durante la realización de evaluaciones de impacto. En este artículo, las autoras presentan una herramienta de evaluación que utilizaron en el marco de una valoración de impacto: los diagramas de impacto participativo. Éstos incorporan métodos mixtos y fueron aplicados a varias acciones realizadas para reducir los efectos de la sequía en Kenia. Los resultados demuestran que, dado los distintos roles ejercidos por los hombres y las mujeres, sus opiniones en relación a las acciones llevadas a cabo difieren. Las autoras analizan cómo esta metodología promovió que las comunidades describieran los distintos resultados de las intervenciones, incluyendo aquellos impactos no previstos, los cuales a menudo son ignorados por los tradicionales métodos de evaluación de impacto. Entre los retos metodológicos enfrentados destacó la integración de datos cuantitativos. El artículo analiza las oportunidades que existen de aplicar esta metodología en un contexto de monitoreo y evaluación más amplio.

Usando diagramas de impacto participativas para avaliar um projeto de desenvolvimento comunitário em Quênia

As abordagens participativas de avaliação de impacto estão se tornando cada vez mais populares entre as organizações de desenvolvimento em seu objetivo de engajar múltiplos colaboradores. Apresentamos aqui nosso uso de diagramas participativos de impacto como uma ferramenta de avaliação dentro de uma avaliação de impacto com métodos variados sobre várias intervenções para redução da seca no Quênia. Os resultados mostram que pelo fato de homens e mulheres terem diferentes papéis, suas experiências de intervenção variam. Discutimos como essa metodologia incentivou comunidades a descreverem vários resultados de intervenção, inclusive impactos não-intencionais, frequentemente negligenciados por abordagens de avaliação de impacto convencionais. Os desafios metodológicos incluíram a integração de dados quantitativos; são consideradas também as oportunidades para sua aplicação dentro da disciplina mais geral de monitoramento e avaliação.     

Between logframes and theory of change: reviewing debates and a practical experience

Theory of Change (ToC) is an emerging methodology in the practice of development programmes, often contrasted with the dominant logical framework. This article reviews current debates around ToC before identifying five aspects that are appreciated in practice. It appears that these aspects mostly cover areas where the logical framework is not – or is no longer – meeting the needs of practitioners. Subsequently, the article analyses experiences in ToC training for NGO staff and concludes that ToC can address shortcomings of the logical framework – if only by going back to some of the roots of the logical framework.     

Joint (ad)ventures and (in)credible journeys evaluating innovation: asset-based community development in Ethiopia

Acknowledging debates about what constitutes effective and useful evaluation practice, this paper explores the particular challenges of evaluating an innovative approach to community development for multiple stakeholders with different interests and different levels of confidence in particular evaluation methodologies. The innovation – applying an Asset Based approach to Community Development (ABCD) in an Ethiopian context – presents further challenges to evaluation because of its open-ended nature and problems of attribution. On the other hand, the culture of risk-taking encouraged by the supporters of innovation provides the space for genuine lessons to be learnt about failure as well as success.     

One village one product: evaluations and lessons learnt from OVOP aid projects

The One Village One Product project that originated in Japan's Oita Prefecture is just one component of many official Japanese development assistance projects aiming to promote rural development in more than 30 countries. Not all efforts, however, result in the same level of sustainability. Oita Prefecture built its movement on three principles: local yet global, self-reliance and creativity, and human resources development. The projects that satisfy these principles are generally found to be sustainable. These principles can also be applicable to overseas projects.     

Pledges not kept: JICA's project challenges in Ghana

Japan's project aid initiatives for capacity development still occupy a large part of the aid discourse of the country's development cooperation programme. Over the years, Japan's development rhetoric has been significantly adapted to the wider policy shifts of Western donor countries and has introduced in its programme and project documents such terms as ‘ownership’, ‘needs-based approach’, and ‘participatory’ initiatives in order to achieve more ‘sustainable’ results. Furthermore, the importance of ‘institutional memory’ has been repeatedly highlighted by Japan as an important element for greater project effectiveness. This article investigates how this development rhetoric translates in Japan International Cooperation Agency's (JICA) project practice in a three-year community-based initiative in rural Ghana. The results illustrate that despite the short-term benefits the project has brought to the targeted communities, the development practice of JICA falls short of its discursive representation and more effort is needed by JICA toward a more committed and inclusive project practice.     

A new way of measuring behavioural compliance for prevention programme interventions using KAP Score

KAP Score is an evaluation tool developed to enable aid and donor organisations to monitor and evaluate the outcome of interventions, particularly those where the link between programme activities and outcomes constitutes an “unknown”. This article articulates KAP Score and demonstrates how it has been applied to two separate interventions related to risk avoidance (human trafficking) and demand reduction (wildlife consumption) to generate quantitative evidence for assessing how they contributed to increasing compliance. The examples presented demonstrate how KAP Score attributed the effectiveness of these interventions to a proxy indicator that can measure incremental behavioural compliance.     

Participatory gender-responsive monitoring of women land rights: empowering and smart

This viewpoint argues that a genuine participatory approach to defining a theory of change and outcome indicators of enhanced women land and property rights is essential to include the standpoint of women and men who are directly involved, and to acknowledge their power to frame the change and solutions they envision. Moreover, outcomes and outcome indicators defined by women and men who are targeted by women rights interventions typically better distinguish outcomes to be reached among women, among men, and in terms of gender relations, are more concrete and easier to measure.     

Effective evaluation of projects on violence against women and girls

What makes evaluations useful for interventions on violence against women and girls (VAWG)? This article reports on a qualitative comparative analysis of 39 evaluations, and shows that it takes a combination of elements to produce good evaluation effects. It identifies eight configurations of conditions which have generated effective evaluations. Key elements were the evaluation context, the evaluators’ sensitivity to gender and to the participants’ rights and security, and consultation with persons considered to be the ultimate “beneficiaries” of the intervention. Both qualitative and quantitative approaches could lead to effective evaluation. The article concludes with recommendations for evaluation users.     

Cordaid's experience with impact evaluation

In 2007, Cordaid started a pilot on participatory impact assessment, intended to enhance accountability and to improve learning. The methodology was based on quasi-experimental design, complemented with qualitative research. This case study illustrates the challenges INGOs and their partners face in their attempt to find a rigorous yet, relevant, useful, and socially acceptable methodology for evaluation and impact assessment purposes. While most local NGOs participating in this pilot consider (parts of) this methodology useful for their learning, this approach proves unsuitable, costly, and inappropriate for an INGO such as Cordaid since it does not respond to its own accountability and learning needs.

Cortaid et leur expérience avec l'évaluation d'impact

En 2007, Cordaid a lancé un programme pilote sur l’évaluation participative de l'impact, dont l'objectif était d'améliorer la redevabilité et l'apprentissage. La méthodologie se basait sur la conception quasi-expérimentale, assortie de recherches qualitatives. Cette étude de cas illustre les défis que les ONGI et leurs partenaires doivent relever au moment de trouver une méthodologie rigoureuse, mais aussi pertinente, utile et socialement acceptable aux fins des évaluations en général et de celles de l'impact en particulier. Si la plupart des ONG qui prennent part à ce programme pilote considèrent que (certaines parties de) cette méthodologie est (sont) utile(s) pour leur apprentissage, cette approche se révèle inadaptée, coûteuse et peu appropriée pour une ONGI comme Cordaid, car elle ne répond pas à ses propres besoins en matière de redevabilité et d'apprentissage.

Cortaid y su experiencia con la evaluación del impacto

En 2007, Cordaid inició una experiencia piloto sobre evaluaciones de impacto participativas, cuyo objetivo consistió en mejorar la rendición de cuentas y el aprendizaje. La metodología utilizada se fundamentó en un diseño semiexperimental, que fue complementado con investigaciones cualitativas. El presente estudio de caso ilustra los retos enfrentados por las ONGI y por sus aliados en su intento por encontrar una metodología rigurosa, pertinente, útil y socialmente aceptable para fines de evaluación y de valoración de impacto. Si bien la mayoría de las ONG locales participantes en esta experiencia piloto consideró que esta metodología, o partes de la misma, era útil para su aprendizaje, el método demostró ser inadecuado, costoso e inapropiado para una ONGI como Cordaid, debido a que no atendió sus propias necesidades en las áreas de rendición de cuentas y de aprendizaje.

Cortaid e sua experiência com avaliação de impacto

Em 2007, a Cordaid iniciou um projeto-piloto sobre avaliação participativa de impacto destinado a melhorar a prestação de contas e aperfeiçoar o aprendizado. A metodologia foi baseada em um modelo quase-experimental, complementado com pesquisa qualitativa. Este estudo de caso ilustra os desafios que as ONGIs e seus parceiros enfrentam em sua tentativa de encontrar uma metodologia rigorosa, porém relevante, útil e socialmente aceitável para fins de avaliação e análise de impacto. Embora a maioria das ONGs locais que participam deste projeto-piloto considere parte dessa metodologia útil para seu aprendizado, essa abordagem mostra-se insustentável, cara e inapropriada para uma ONGI como a Cordaid pois não atende às suas próprias necessidades de prestação de contas e aprendizado.     

Can ICT-enabled real-time data contribute to adaptive management in development programming?

This article aims to explore whether ICT-enabled real-time data (RTD) systems can help to improve the operationalisation of adaptive management of international development programmes. Using a qualitative multi-method approach consisting of 48 semi-structured key expert interviews and four exploratory case studies, we found that RTD can successfully inform rapid tactical adaptive management in development programmes but is, on its own, of only limited use for most strategic adaptive management. The research identified multiple contextual barriers to the use of RTD for adaptive management. These barriers need to be addressed to realise the full potential of real-time adaptive management of programmes.     

Using the Five Capabilities (5C) model: making a virtue of necessity

The development sector is rife with complaints about strict accountability requirements imposed by donors. However, this paper argues that the imposition of a new accountability framework can sometimes be converted into a useful tool. This note describes how the IMPACT alliance used the Five Capabilities (5C) model prescribed by the Dutch Ministry of Foreign Affairs. It describes the major findings from a baseline assessment, and reflections on that process. The authors argue that the operationalisation of the 5C model provides useful opportunities for discussing capacity development priorities with partners.

Utilisation du modèle des cinq capacités (5C): faire de nécessité vertu

Les plaintes sont nombreuses dans le secteur du développement concernant les exigences strictes de reddition de compte imposées par les bailleurs de fonds. Cependant, cet article soutient que l'imposition d'un nouveau cadre de reddition de comptes peut parfois être convertie en outil utile. Cette note décrit la manière dont l'alliance IMPACT a eu recours au modèle des cinq capacités (5C) prescrit par le ministère néerlandais des Affaires étrangères. Elle décrit les principales conclusions tirées d'une évaluation de référence, ainsi que des réflexions sur ce processus. Les auteurs soutiennent que l'opérationnalisation du modèle des 5C donne des occasions utiles de discuter des priorités de développement des capacités avec les partenaires.

Operando el modelo de las Cinco Capacidades (5C): hacer de la necesidad una virtud

En el ámbito de desarrollo, prevalecen las quejas en relación a las estrictas medidas de rendición de cuentas impuestas por los donantes. Sin embargo, el presente artículo sostiene que el requisito de usar un nuevo marco para la rendición de cuentas, a veces puede convertirse en un ejercicio útil. En el artículo se describe cómo la alianza impact usó el modelo Cinco Capacidades (5C) recomendado por el Ministerio de Asuntos Exteriores de Holanda. En este sentido, se analizan las conclusiones de una evaluación de referencia y se hacen reflexiones respecto a este proceso. Los autores sostienen que la implementación del modelo 5C genera oportunidades útiles para revisar las prioridades en torno al desarrollo de capacidades con las contrapartes.

Utilizando o modelo das Cinco Capacidades (5C): fazendo da necessidade uma virtude

O setor de desenvolvimento é repleto de reclamações sobre exigências rigorosas de prestação de contas impostas por doadores. Porém, este artigo argumenta que a imposição de uma nova estrutura de prestação de contas pode às vezes ser convertida em uma ferramenta útil. Esta nota descreve como a aliança IMPACT utilizou o modelo das Cinco Capacidades (5C) prescrito pelo Ministro holandês de Assuntos Estrangeiros. Ele descreve os principais resultados de uma avaliação de base e reflexões sobre este processo. Os autores argumentam que a operacionalização do modelo 5C oferece oportunidades úteis para se discutir as prioridades de desenvolvimento de capacidade com parceiros.     

Capacity building at mid-programme: an international education development programme in Malawi

This article reports on a case study of a mid-programme capacity building evaluation within a large education aid programme collaboration between non-governmental educational organisations in Malawi and US university literacy faculty. The article outlines the programme context and its formal and informal capacity building inputs. Analyses of data collected on capacity building at the midpoint of the programme are offered. The authors argue that capacity is built along the life of large programmes, and evaluating capacity building development (and understanding its challenges) before the end of the programme can help cross-national teams of administrators and implementers in modifying programme operations.     

Mobilising faith-based and lay leaders to address antenatal care outcomes in northern Ghana

ABSTRACT

Despite the benefits of antenatal care, evidence from sub-Saharan Africa suggests that women often initiate these services after the first trimester of pregnancy and do not complete the recommended number of visits. This study examines the impact of mobilising faith-based and lay leaders to address the socio-cultural barriers to antenatal care uptake in northern Ghana in the context of a broader child survival project. A quasi-experimental design was used, and data were analysed using a difference-in-differences approach. The results presented in this article indicate the potential for faith-based and lay leaders to promote uptake of maternal and child health behaviours.     

Wiki approaches to wicked problems: considering African traditions in innovative collaborative approaches

Wicked problems are complex problems that are seemingly impossible to solve. However, an analysis of selected traditional African philosophies provides insight into how certain traditions may be applied in a practical sense to address social and environmental problems. Further, many newer collaborative and ‘wiki’-based solutions provide a natural way for Africans and other global actors to participate in lessening the impact of global wicked problems. Ushahidi and the Geo-Wiki Project serve as examples of organisations that have provided a platform for this type of open development.     

Videos that speak for themselves: when non-extensionists show agricultural videos to large audiences

In 2008, an NGO showed videos about rice to farmers in 19 villages in Benin. A study in 2013 showed that farmers remembered the videos, even after five years had passed. In most of the villages at least some farmers experimented with rice farming or with new technology after the video screenings, which attracted large audiences of community members, including youth and women. Some of the villagers also visited extension agencies to get rice seed, and occasionally to seek more information. Farmers can benefit from agricultural learning videos shown by organisations with little previous agricultural experience. Videos do not necessarily need to be facilitated by an expert who knows the subject. Sometimes the video can speak for itself.     

One size does not fit all: choosing methods to inform area development

There have always been debates about the methods that should be used to inform and assess development programmes. Experimental methods have become highly advocated as agencies seek rigorous ways to show programme value. However, the benefits and appropriateness of these methods are frequently overstated. We use the Aga Khan Development Network's Quality of Life studies to show that periodic mixed methods approaches are useful to analyse programme contributions and inform area development. We argue that experimental methods should not be idealised, and that research questions and organisational learning should guide pragmatic methodological choices to inform development intervention in real-life contexts.

Une taille unique ne convient pas toujours : comment choisir des méthodes pour éclairer le développement des régions

Il y a toujours eu des débats sur les méthodes qui devraient être employées pour éclairer et évaluer les programmes de développement. Les méthodes expérimentales sont devenues très préconisées dans le cadre de la quête par les organisations de manières rigoureuses de démontrer la valeur de leurs programmes. Cependant, les avantages et le caractère approprié de ces méthodes sont souvent exagérés. Nous utilisons les études sur la qualité de la vie de l'Aga Khan Development Network pour montrer que les approches périodiques basées sur des méthodes mixtes sont utiles pour analyser les contributions des programmes et éclairer le développement des régions. Nous soutenons que les méthodes expérimentales ne devraient pas être idéalisées, et que les questions des recherches et l'apprentissage organisationnel devraient orienter des choix méthodologiques pragmatiques afin d'éclairer les interventions en matière de développement dans les contextes réels.

La unitalla no le queda a todos: seleccionando métodos que informen el desarrollo de áreas

Siempre han existido debates en torno a los métodos que deben usarse para fundamentar los programas de desarrollo y valorarlos. A medida que las agencias han buscado métodos rigurosos para demostrar el valor de sus programas, ciertos métodos experimentales han sido altamente promovidos. Sin embargo, los beneficios y lo apropiado de tales métodos a menudo se ha exagerado. En el presente artículo, los autores utilizan los estudios sobre la Calidad de Vida realizados por la Red de Desarrollo Aga Khan, con el fin de mostrar que los enfoques de métodos mixtos aplicados periódicamente resultan útiles para analizar los aportes de los programas y para potenciar el área de desarrollo. Asimismo, sostienen que no deben idealizarse los métodos experimentales y que las inquietudes de investigación y el aprendizaje organizacional deberán guiar las opciones metodológicas pragmáticas orientadas a fundamentar las acciones de desarrollo en la vida cotidiana.

Um único tamanho não serve para todos: Escolhendo métodos para informar a área de desenvolvimento

Tem havido frequentemente debates sobre os métodos que deveriam ser utilizados para informar e avaliar programas de desenvolvimento. Métodos experimentais têm sido altamente defendidos quando as agências buscam formas rigorosas de mostrar o valor do programa. Porém, os benefícios e adequação destes métodos são frequentemente superestimados. Utilizamos os estudos sobre Qualidade de Vida da Aga Khan Development Network para mostrar que abordagens de métodos mistos periódicos são úteis para analisar as contribuições dos programas e informar o desenvolvimento de área. Argumentamos que métodos experimentais não devem ser idealizados e que questões de pesquisa e aprendizado organizacional devem orientar escolhas metodológicas pragmáticas para fornecer informações à intervenção de desenvolvimento em contextos da vida real.     

Identification and analysis of smallholder producers’ constraints: applications to Tanzania and Uganda

This article puts forward a method for the analysis of constraints faced by developing countries’ smallholder producers. It is consistent with theories of constraints, efficient in terms of cost and researchers’ time, and accessible to a non-technical audience. A hybrid of workshop discussion and individual data collection, it also draws on data and analyses available in most developing countries. The article presents an application to smallholder livestock systems in Tanzania and Uganda, reporting results and analysis relating constraints to households’ characteristics and conditions, and their stated goals. While limitations are identified, it is proposed for application in other development fields.     

How projects rise and fall: the lifecycle of a dietary modernisation scheme

How do projects grow? How do they fail? What accounts for their changing fortunes? This paper uses the archives of a 1970s modernisation scheme to explore the life cycle of a long-running project, concerning the production of leaf protein in Nigeria. It argues that archives can be very useful for understanding success and failure, and encourages practitioners to take an interest in the story of past projects, even those that failed. Drawing on Actor-Network Theory, it argues that alliances are key to understanding project lifecycles, suggesting that practitioners focus on strengthening local relationships, rather than seeking answers in universal management templates.     

The persisting challenge of age-for-grade non-compliance in post-free primary education in Kenya

This article examines the prevalence of non-compliance on the age-for-grade policy in the post-free primary education system in Kenya. We utilised data from the 2009 cohort of enrolled primary schoolchildren. The analysis revealed non-compliance to be a persisting concern in the school system, characterised by both under-age and over-age enrolment at the age-for-grade level of analysis in spite of the introduction of free primary education in 2003. Irrespective of the nature of non-compliance, overcrowding in the lower grades is more prevalent, with potential for adverse compromises in quality of learning and the attendant added dimension of the mismatched grade-for-age curricula.     

Strategic planning through a participatory learning and action framework: a Kenyan study

This paper reflects on the use of a participatory learning and action (PLA) framework in a strategic planning process (SPP) at a primary school in rural Kenya. Successes and challenges are discussed, focusing on the barriers to fully implementing the PLA framework. Additionally, lessons learnt and ways forward for participatory SPPs are outlined. The PLA framework allowed the project stakeholders to engage in a planning process that gave participants a voice in the process. Although PLA methodology faces shortcomings, the application of this model is a step towards making community development more sustainable.