It is exciting to be back again at the annual AES Conference. It is fantastic to listen to new ideas from some exciting thinkers in the sector as well as present and get feed back on some of the issues I have been grappling with in the last twelve months.

The closing  key note speaker on Day 1 was Donna Mertens. She was  brilliant. She talked about applying a social justice lens to evaluation. I went to her workshop in Cairo a couple of years ago and she continues to inspire me. She is a Professor at the Gallaudet University, the world’s only university in which all programs and services are specifically designed to accommodate deaf and hard of hearing students. She talks about the evaluator as provocateur, asking the hard questions, especially around marginalised people. Listening to her really made me think again about our evaluative practice. I was challenged to remember that at times, we evaluators do have to take a stand and ask questions about things that are not being raised by our inquiry and dig a little deeper.

It was a busy conference for me. I ran a pre-conference workshop on MERI using the Castlemaine 500 evaluation as a case study. The workshop seemed to go down a treat, and I did enjoy it despite fretting over the fact that my bag failed to arrive from Sydney!

I also presented a session on “Reflecting on Evaluation in Australian Aid’ alongside Cate Rogers from AusAID’s Office of Development Effectiveness and Lucia Boxelaar (World Vision Australia). We were all quite excited at the end of the talk, which stimulated lots of conversation.

While the foundations of good evaluative practice are being put in place, and leaps and bounds have been made, the quality of evaluations in this sector appears to remain low. My paper was drawn from my experience over the last 12 months of reviewing 30 external evaluations, four (4) evaluation frameworks, as well as leading several evaluations.  The 30 evaluations covered a range of topics from HIV, civil society strengthening, education, market access, law and justice and more. They were conducted in 15 countries within South East Asia, Central Asia, the Pacific, Melanesia and Africa and evaluated programs run by a range of Government and NGO’s organisations.  This study, although not exhaustive, indicates that in Australasia the evaluation of aid effectiveness is still in its infancy and appears to have developed in a somewhat different mode to that of public sector evaluation.

In my experience of the evaluation process in the aid sector, the terms of reference are often very detailed and are written by the client. The evaluator has little input into the evaluation scope or design. Evaluation practice in the aid sector also seems to be dominated by ‘content-led’ evaluation teams.  And with the exception of rarely conducted, cross-country or strategic evaluations, evaluations are typically conducted over a short period (typically 10 days in country and 10 days to write the report).

Due to the inherent difficulty of collecting data in developing countries, and the lack of existing data, these studies tend to rely strongly on semi-structured interview with a carefully selected number of informants. Evaluation reports commonly present findings in an aggregated form interwoven with the expert opinion of the evaluation team.  Some agencies impose a strict page limit, which restricts the presentation of more in-depth findings. The methodology is generally described in a very brief manner, and I found several instances where there was no methodology section included in the report at all. Given the short time frames and lack of existing data, it is unsurprising that little evidence is presented in, what  may be more accurately described as ‘expert reviews’ than evaluations. On the positive side, this review of evaluation found that many of these studies had managed to address the question of “whether the aid activity was relevant to the context” surprisingly well, but gave limited attention to addressing questions of efficiency and effectiveness.

In terms of monitoring practice in aid, while most aid agencies appear to mandate the creation of monitoring and evaluation plans at the onset of program design, these plans often focus strongly on output indicators with little focus on intermediate or longer term outcomes. There also appears to be a tendency of monitoring for compliance and accountability to donors rather than for learning or downward accountability.

Professor Eliot Stern came to our talk (editor of the Evaluation journal) and was interested to hear more about our experiences -  I feel a paper coming on!