By Dr Jess Dart

The counterfactual

Today there was a very interesting panel discussion with Robert chambers, Patricia Rogers, Dean, and Martin Ravaillion. So two economists, a realist, and an advocate of participatory approaches. The differences were quite stark, but there were agreements too. All agreed that there should be a theory of change, all agreed that you cannot always use randomization, and all agreed that you needed to consider the context carefully. The main disagreements were whether it was necessary to have a counterfactual comparison to make claims around causality. Robert reminded people that we needed to ensure the processes we use to do evaluate do benefit the primary constituents(eg poor people) and that the only way of determining that was to ask people. I found debate really interesting, and it made me realize that there are many ways of defining impact.

How to evaluate impacts of networks

I went to a session on how to evaluate the impacts of networks, and clearly there are many people scratching their head over this one. There was a general consensus that logframes are inadequate to describe the outcomes of a network, as many of the outcomes are emergent and way beyond what can be imagined at the start. Also it is hard to attribute the impacts of a network back to the network investment. Interestingly I noted that these issues are pretty stock standard for just about all the evaluations we conduct! But as the conference went along, I began to wonder whether the work we do in Australia is more the exception rather than the rule to how evaluations are conducted in the world at large. In fact a few people commented as much!

Robert chambers session on participatory impact evaluation

Robert made the point that participatory techniques can generate numbers. He talked about using matrix ranking and the 10 seed technique to do all sorts of before and after comparisons with people about what had changed in their community. These are the basic PRA techniques being applied in an impact evaluation setting. Funny, I did this very thing back in 1997 in India - and the photos on this website show. When people asked about how you aggregate local indicators across different communities, Robert suggested that you can have some basic pre-set indicators, then let the community decide on the other criteria, depending on what is important to them.

In one example he gave, the local community came up with 120 indicators of poverty. The community themselves were happy to score themselves against all of these! This reminds me of some of the work that Rick Davies did in Vietnam, where the community created a set of indicators for poverty. Some very interesting things were included in the list. But all this goes back to the basic question of, “in an impact evaluation, who decides what success looks like? “On whose values are the determination of merit or worth made?”