Jane Davidson was the key note speaker on final morning of the conference. Her talk was on ‘Extreme genuine evaluation makeovers”. She and Patricia Rogers have a blog on this topic which might be worth checking out. Her argument centered on a critique around how evaluations are done, proposing that we need to redefine what we are calling a credible rigorous evaluation. I think some of this is a back lash against the big move in the United States towards experimental design and randomize-control-experiments as the only rigorous impact evaluation approach. She contends that evaluation that is value-free is more of a worry, and lays out a simple set of guidelines that should frame any evaluation. She was very adamant that evaluation reports should be structured against the key evaluation questions, and that these questions should be “judgment” questions that involve a level of assigning value to results. Apparently she has writing a great book about evaluation basics. She was very clear and accessible, a trait I value!

Her talk made me reflect on our Collaborative Outcomes Reporting approach, and whether it was valid under this lens. I think the fact that it is not a full evaluation report but outcomes report makes it a slightly different beast. We do of course have a central evaluation question to frame a COR report, I guess we should make sure that this question is answered and that the answer is clearly shown upfront. The answer to this question is determined in the outcomes panel of a COR process, and judgments are made, but maybe we should bring this to the fore more.

I also went to a useful talk by Chris Milne who was discussing the use and abuse of the terms “evaluation plan, evaluation framework and evaluation strategy” and attempting to define what they mean and what the difference is between these terms. He did advocate not trying to pin the words down too much, but to always clarify with a client what they are meaning. However, he did define them all and I found his definitions useful. I think I have been using the terms in a somewhat scatter gun manner! I have discovered we do them all at Clear Horizon. I am going to suggest that within our company we at least come to an agreement on what the terms mean. So, for example, our MERI steps are a “MERI framework”! Not a plan and not a strategy!

I went to another session on soft systems methodology and evaluability assessment. Do you know I studied soft systems (Peter Checkland and others) as part of my Masters degree nearly 20 years ago! While I don’t hold this methodology as a core part of my work, it certainly influenced my thinking very much. I reflected on the fact that our “people-centred program logic’ very much draws from soft-systems methodology. In fact Bob Williams has put an entry on his book about people-centred logic.

Reflecting on the conference on my way home in the plane, I’m happy to say I got a lot out of the conference this year. I will make a mental note that it is always worth going along, not the least for all the amazing connections I make, but also for the reflective space it affords! I encourage you to come and check it out. Next year it is Sydney and I will be definitely be there. I am going to try to present about the big civil society evaluation work we have done with the Office of Development Effectiveness and our new training product around Evaluating Engagement.

Oh yes! We did decide to set up a “special interest group” around evaluating aid in Australasia. This is going to be an informal chance to catch up and chat about issues on the last Tuesday of each month in Lygon Street, Melbourne. We will set up a national email group that will meet at each AES conference. It is an Australasian Evaluation Society conference group at the moment, so we encourage members to come along!