Her talk made me reflect on our Collaborative Outcomes Reporting approach, and whether it was valid under this lens. I think the fact that it is not a full evaluation report but outcomes report makes it a slightly different beast. We do of course have a central evaluation question to frame a COR report, I guess we should make sure that this question is answered and that the answer is clearly shown upfront. The answer to this question is determined in the outcomes panel of a COR process, and judgments are made, but maybe we should bring this to the fore more.

I also went to a useful talk by Chris Milne who was discussing the use and abuse of the terms “evaluation plan, evaluation framework and evaluation strategy” and attempting to define what they mean and what the difference is between these terms. He did advocate not trying to pin the words down too much, but to always clarify with a client what they are meaning. However, he did define them all and I found his definitions useful. I think I have been using the terms in a somewhat scatter gun manner! I have discovered we do them all at Clear Horizon. I am going to suggest that within our company we at least come to an agreement on what the terms mean. So, for example, our MERI steps are a “MERI framework”! Not a plan and not a strategy!

I went to another session on soft systems methodology and evaluability assessment. Do you know I studied soft systems (Peter Checkland and others) as part of my Masters degree nearly 20 years ago! While I don’t hold this methodology as a core part of my work, it certainly influenced my thinking very much. I reflected on the fact that our “people-centred program logic’ very much draws from soft-systems methodology. In fact Bob Williams has put an entry on his book about people-centred logic.

Reflecting on the conference on my way home in the plane, I’m happy to say I got a lot out of the conference this year. I will make a mental note that it is always worth going along, not the least for all the amazing connections I make, but also for the reflective space it affords! I encourage you to come and check it out. Next year it is Sydney and I will be definitely be there. I am going to try to present about the big civil society evaluation work we have done with the Office of Development Effectiveness and our new training product around Evaluating Engagement.

Oh yes! We did decide to set up a “special interest group” around evaluating aid in Australasia. This is going to be an informal chance to catch up and chat about issues on the last Tuesday of each month in Lygon Street, Melbourne. We will set up a national email group that will meet at each AES conference. It is an Australasian Evaluation Society conference group at the moment, so we encourage members to come along!

I presented a panel today with Susan Williams (from DSE) on the evaluation we did of the Victorian Biodiversity Strategy. The talk went down very well. It described the bread and butter of what we do at Clear Horizon, including science panels, summit workshops and bringing stakeholders along for the journey. But clearly it was new to a lot of the audience.

However, it was really nice to be able to say that the evaluation did have significant influence! 　All the recommendations from the evaluation are listed in the draft revised strategy along with a page about how the revised strategy had accommodated these recommendations. It also appears that the process we used to engage stakeholders in the evaluation inspired the strategy team to continue with these relationships and develop a partnership approach. One person asked me in the question session: but when you do a summit workshop, you often end up analyzing things twice, is it worth it? I answered: absolutely it is worth it! How many evaluations are there that sit on the shelf and never get used? How many beautifully worded recommendations never get acted on? By getting the stakeholders and key actors involved in analyzing the data and developing the recommendations they are far more likely to get taken up! Lots of people came to talk to us afterwards.

Professor Eliot Stern closed the day with a keynote address about the possible demise of　the evaluation industry. It was a pretty gloomy talk! He began by talking about the demise of the industrial era, and how we are currently at a turning point. He called for a re-think of the whole evaluation profession, advocating that evaluators need to expand their services otherwise our space will be absorbed by others. He pointed out that we now have evaluators, auditors, quality assessors, designers, etc all working in separate silos.

He believes we have to become cross cutting, and use evaluative practice in different contexts. This was interesting for me because at Clear Horizon we already seem to do lots of things beside evaluation. We use our ‘evaluative practice’ in the fields of engagement, strategic planning, program design and organisational effectiveness. I think his talk was a reflection more of the situation in the UK, which I hear is going through terrible economic times and budget cuts. Nonetheless, the call for evaluation to embrace evaluation of evaluands that are ‘fuzzy’ such as policy evaluation, multi-partner strategies, emergent adaptive programs, felt relevant.

The closing  key note speaker on Day 1 was Donna Mertens. She was  brilliant. She talked about applying a social justice lens to evaluation. I went to her workshop in Cairo a couple of years ago and she continues to inspire me. She is a Professor at the Gallaudet University, the world’s only university in which all programs and services are specifically designed to accommodate deaf and hard of hearing students. She talks about the evaluator as provocateur, asking the hard questions, especially around marginalised people. Listening to her really made me think again about our evaluative practice. I was challenged to remember that at times, we evaluators do have to take a stand and ask questions about things that are not being raised by our inquiry and dig a little deeper.

It was a busy conference for me. I ran a pre-conference workshop on MERI using the Castlemaine 500 evaluation as a case study. The workshop seemed to go down a treat, and I did enjoy it despite fretting over the fact that my bag failed to arrive from Sydney!

I also presented a session on “Reflecting on Evaluation in Australian Aid’ alongside Cate Rogers from AusAID’s Office of Development Effectiveness and Lucia Boxelaar (World Vision Australia). We were all quite excited at the end of the talk, which stimulated lots of conversation.

While the foundations of good evaluative practice are being put in place, and leaps and bounds have been made, the quality of evaluations in this sector appears to remain low. My paper was drawn from my experience over the last 12 months of reviewing 30 external evaluations, four (4) evaluation frameworks, as well as leading several evaluations.  The 30 evaluations covered a range of topics from HIV, civil society strengthening, education, market access, law and justice and more. They were conducted in 15 countries within South East Asia, Central Asia, the Pacific, Melanesia and Africa and evaluated programs run by a range of Government and NGO’s organisations.  This study, although not exhaustive, indicates that in Australasia the evaluation of aid effectiveness is still in its infancy and appears to have developed in a somewhat different mode to that of public sector evaluation.

In my experience of the evaluation process in the aid sector, the terms of reference are often very detailed and are written by the client. The evaluator has little input into the evaluation scope or design. Evaluation practice in the aid sector also seems to be dominated by ‘content-led’ evaluation teams.  And with the exception of rarely conducted, cross-country or strategic evaluations, evaluations are typically conducted over a short period (typically 10 days in country and 10 days to write the report).

Due to the inherent difficulty of collecting data in developing countries, and the lack of existing data, these studies tend to rely strongly on semi-structured interview with a carefully selected number of informants. Evaluation reports commonly present findings in an aggregated form interwoven with the expert opinion of the evaluation team.  Some agencies impose a strict page limit, which restricts the presentation of more in-depth findings. The methodology is generally described in a very brief manner, and I found several instances where there was no methodology section included in the report at all. Given the short time frames and lack of existing data, it is unsurprising that little evidence is presented in, what  may be more accurately described as ‘expert reviews’ than evaluations. On the positive side, this review of evaluation found that many of these studies had managed to address the question of “whether the aid activity was relevant to the context” surprisingly well, but gave limited attention to addressing questions of efficiency and effectiveness.

In terms of monitoring practice in aid, while most aid agencies appear to mandate the creation of monitoring and evaluation plans at the onset of program design, these plans often focus strongly on output indicators with little focus on intermediate or longer term outcomes. There also appears to be a tendency of monitoring for compliance and accountability to donors rather than for learning or downward accountability.

Professor Eliot Stern came to our talk (editor of the Evaluation journal) and was interested to hear more about our experiences -  I feel a paper coming on!

I was invited to speak at a Conference last week in the Netherlands that was exploring rigorous evaluation practice that embraces complexity. The inspiration for this event was last years’ World Bank conference on impact evaluation for aid effectiveness in Cairo. In Cairo the conference was structured around two presentation streams: ‘rigorous evaluation’ and ‘other evaluation methods’. The rigorous stream comprised presentations on randomised-control trials (RCTS) and meta-synthesis. It was argued that experimental and quasi-experimental designs had a comparative advantage because they provide an unbiased numeric estimate of impact. My presentation in Cairo fell in the ‘other stream’ along with anything else that did not constitute experimental design and randomisation. In a sense the Netherlands conference was an attempt to claim back some of the notions of ‘rigour’ outside of experimental design.

Professor Patricia Rogers was one of the keynote speakers at the Netherlands conference and she talked about the notion of the domains of known, knowable, complicated and complex taken from the Cynefin Model (see www.ibm.com/services/cynefin). The Cynefin Model was developed by Dave Snowden and is a sense-making framework that offers a useful way of understanding and considering appropriate evaluation for different program contexts. The model is a flexible tool and a single program usually demonstrates aspects of different Cynefin domains at the same time. Professor Rogers put forward the idea that different methods and approaches are needed to evaluate program elements that are characterised by these different domains.

The idea is that approaches such as randomised control trails (RCTs) are good for determining known, or knowable program contexts, where cause and effect can be predicted with enough knowledge and investigation. However, there are other contexts which are characterised as ‘complex’ in which state ‘cause and effect’ relationships cannot be predicted as many things are affecting many things. In this domain, however, it is possible to retrospectively understand what happened, but these understandings cannot necessarily be used to predict the same outcomes in a different context. In this situation we many need more inductive methods of inquiry, and perhaps the focus is more appropriately on learning and ‘navigating the course forward’ for that program. RCTs in a complex program context would not be appropriate

Maarten Brouwer, special Ambassador of the International Cooperation, DGIS in The Hague spoke on the second day about ‘web 2.0′ and how evaluation and programming might play out in the future. He was very much espousing a greater focus on ‘outcomes’ and ‘real time evaluation’ as well as on learning. He suggested that perhaps it was becoming less necessary to focus on attribution as donors harmonise and development is conducted through multiple collaborations. 

The conference presented nine case studies of rigorous evaluation practice, and five workshop methods. Interesting methods presented included an approach to social return on investment; an approach for assessing capacity development and my presentation on ‘Collaborative Outcomes Reporting’ (nee PPSR) also went down very well and raised lots of interest.

So what did we conclude about ‘rigorous evaluative practice that embraces complexity’. Well I can tell you my musings. Firstly, I think there are possibly more implications for appropriate program design and contracting when working with complex program elements. The designs will need to be more dynamic, with a greater focus on process that prescribed SMART targets. With regard to evaluation, the things that come to mind would be an emerging need for:

• short cycles of measurement and reflection, so as to repeatedly check to see if things are working and if not modify.
• inclusion of methods ( ‘butterfly nets’) to catch unexpected outcomes: such as MSC technique
• sense-making approaches to decide what to do about the unexpected outcomes
• broad outcomes that are not too prescriptive
• iterative program logic/ program theory
• values inquiry, as if there are lots of unexpected outcomes, where will also be a range of way of valuing these outcomes that has not already been established
• rigour: generally mixed methods are advocated, but whatever methods are used it is still important that they hold up to scrutiny
• participatory approaches can be appropriate when working in the more complex arenas as they often allow spaces for sense-making, deliberating on outcomes and ensuring changes in program direction are appropriate to different stakeholder groups.

For Clear Horizon the conference was an endorsement that we are on the right track with the methods and products that we are developing and using. There was also plenty of exciting food for thought. One of the ideas of the Cynefin Model is to try to identify the knowable and known aspects of a program, which requires a greater emphasis on quantitative methods. I was also excited by some of their sense making techniques; we would need to find some hexagonal post-it notes!

So I have been experimenting with using theory of change to map out both development outcomes and aid outcomes. I think I have finally had a breakthrough! In the last couple of days I was using theory of change to clarify the design of a sector specific government strengthening program in Indonesia. Firstly we used theory of change to map what we felt the system should look like in future, and we also broke it down into intermediate outcomes that would need to be in place to get to that future state. Secondly we did some work around what the program could offer, and the principles of engagement. Then we identified points in the system where this aid might best be applied. Then we looked across this picture and synthesized it to create a theory of change for the aid program - and it worked! So, we ended up with two inter related theory of change models, one for development and one for Aid. Of course the development theory of change is best developed with partners.

‘Intervention logic’ is the term commonly used in New Zealand for program logic. I have been musing a lot recently over the definitions and whether there is any difference in the names. I realize that people do use the terms interchangeably but some people draw clear distinctions for  each. But for me, I notice that program logic/ intervention logic are used a lot by evaluators. Evaluators are often using this tool to clarify an existing program, hence they work bottom up - from the activities up to the outcomes. They call the stuff in the middle intermediate outcomes as they are on their way upwards. Evaluators do from time to time end up using program logic for planning, then they might map downwards. But I think most would be more used to map upwards.

However, theory of change is nearly always mapped downwards. Starting at the outcomes we identify the necessary preconditions to achieve the outcomes .(We call the stuff in the middle preconditions rather than intermediate outcomes) . At the end of the day the models might look very similar, but the thinking process is quite different! Backwards mapping from the outcome down does make one’s head hurt more! Its much more of a planning tool than a clarification tool. So there you go, I add my distinction to the rest of them!

They are a set of outcomes (ideally) that the organisation promises to achieve. It is accountable for managing for these outcomes. In a more global theory of change, there may be a whole set of intermediate outcomes, and only a few will be chosen as objectives for that given program/ strategy.

But in practice objectives tend to be located at varying places in a theory of change model. Sometimes they are activities, sometimes they are foundational activities, sometimes milestones, sometimes intermediate outcomes and sometimes high level outcomes! Objectives are not defined by their position in a hierarchy of change, they are defined more by the fact that they are a promise to deliver. So it all depends where the program is sitting to where the objectives will sit. They are a relative term, not an absolute term.

I’d love to hear what you think of my take on this

Some dictionary definitions that relate to organsational use of the term (selected from wickipedia):

• Objective (military), the achievement of a final set of actions within a given military operation …
• Objective - aim: the goal intended to be attained (and which is believed to be attainable); “the sole object of her trip was to see her children” 
• goal or objective is a projected state of affairs that a person or a system plans or intends to achieve-a personal or organizational desired …
• The Objective is a 2008 science fiction horror film directed by award-winning filmmaker Daniel Myrick, who also directed The Blair Witch Project. It stars Jonas Ball, Matt Anderson, and Michael C. Williams. …
• A military objective is a clearly defined desired result in a given campaign, major operation, battle, or engagement set by the senior command for their formations and units to achieve. …
  are the broadest statements of what the organization chooses to accomplish. (300.02.1)
• In a proposal, the objectives are statements of anticipated outcomes. Objectives are stated in measurable terms and tell the grantor who will do what, when, how much, and how it will be measured. …
• objective - A statement of intention or description of a goal. As with any objective, it should be specific, measurable, achievable, realistic and time-bound …
• objective - A concrete statement describing what the project is trying to achieve. The objective should be written at a low level, so that it can be evaluated at the conclusion of a project to see whether it was achieved or not.

There is no doubt that these terms are being used interchangeably, but some people do see them as different things.

It seems I have my own personal distinction which doesn’t match the literature (blast!). I have been making a distinction that a program logic model concerns itself with how one specific program will contribute to outcomes - (eg its program centric). It doesn’t necessarily map out how other programs/ stakeholders/issues are contributing to the same outcome. I have been reserving the term ‘Theory of change’, for the situations when you map out a much bigger scenario - more of a whole systems view of what change needs to occur in order to achieve that outcome (outcome centric).

Under this framework a program logic focuses on just one set of pathways in the broader theory of change - depending what that program can uniquely offer that is not already being done. I think Paul Dougan refers to it as ‘global logic’. And its great when you have a complicated program to map this bigger theory of change out first, then decided which bits your program can usefully focus on to leverage maximum contribution to outcomes. Hey I still like my distinction!! Also I guess a theory of change under my distinction is very useful in the planning stages of a program.

Others, (notably the Aspen Institute) seem to be focusing on the theory of change being more about specifying the magnitude of a particular precondition required to achieve an outcome.

www.evaluationtoolsforracialequity.org/…/TOCs_and_Logic_Models_forAEA.ppt

I am not so sure about their distinction, I am sure you could do that with a logic model if you wanted! Also it all sounds very mathematical - but I guess it does come out of this whole evidence based movement. Eg prove how much of this precondition is required to achieve the next condition. I bet this could be done nicely with Basian network software!

I would love to hear about how you define the difference!

Being a first timer at the AES conference my goal was simple - to immerse myself in the bath of evaluation! I participated for one full day and thoroughly enjoyed the experience. I took the opportunity to just chat with anyone who was nearby and met some very interesting and diverse people. People worked in very different fields, but all were there to learn more about evaluation. This expanded network may prove useful to me in the future.

Through the sessions I enjoyed learning about the use of photographs in conducting evaluations in foreign countries and giving a voice to children; a historical perspective of the Federal government’s attitude to evaluation (not as boring as it sounds I confess); conducting evaluations with Indigenous participants; social network analysis (still not too sure about this approach); and finally planning to set up an evaluation framework for the ‘enormous beast’ that is CSIRO. After spending what remained of the day bonding with colleagues and the key note speaker about the stereotypical native fauna of Australia and the USA, I certainly slept well that night! A full day lived indeed….