I was invited to speak at a Conference last week in the Netherlands that was exploring rigorous evaluation practice that embraces complexity. The inspiration for this event was last years’ World Bank conference on impact evaluation for aid effectiveness in Cairo. In Cairo the conference was structured around two presentation streams: ‘rigorous evaluation’ and ‘other evaluation methods’. The rigorous stream comprised presentations on randomised-control trials (RCTS) and meta-synthesis. It was argued that experimental and quasi-experimental designs had a comparative advantage because they provide an unbiased numeric estimate of impact. My presentation in Cairo fell in the ‘other stream’ along with anything else that did not constitute experimental design and randomisation. In a sense the Netherlands conference was an attempt to claim back some of the notions of ‘rigour’ outside of experimental design.

Professor Patricia Rogers was one of the keynote speakers at the Netherlands conference and she talked about the notion of the domains of known, knowable, complicated and complex taken from the Cynefin Model (see www.ibm.com/services/cynefin). The Cynefin Model was developed by Dave Snowden and is a sense-making framework that offers a useful way of understanding and considering appropriate evaluation for different program contexts. The model is a flexible tool and a single program usually demonstrates aspects of different Cynefin domains at the same time. Professor Rogers put forward the idea that different methods and approaches are needed to evaluate program elements that are characterised by these different domains.

The idea is that approaches such as randomised control trails (RCTs) are good for determining known, or knowable program contexts, where cause and effect can be predicted with enough knowledge and investigation. However, there are other contexts which are characterised as ‘complex’ in which state ‘cause and effect’ relationships cannot be predicted as many things are affecting many things. In this domain, however, it is possible to retrospectively understand what happened, but these understandings cannot necessarily be used to predict the same outcomes in a different context. In this situation we many need more inductive methods of inquiry, and perhaps the focus is more appropriately on learning and ‘navigating the course forward’ for that program. RCTs in a complex program context would not be appropriate

Maarten Brouwer, special Ambassador of the International Cooperation, DGIS in The Hague spoke on the second day about ‘web 2.0′ and how evaluation and programming might play out in the future. He was very much espousing a greater focus on ‘outcomes’ and ‘real time evaluation’ as well as on learning. He suggested that perhaps it was becoming less necessary to focus on attribution as donors harmonise and development is conducted through multiple collaborations. 

The conference presented nine case studies of rigorous evaluation practice, and five workshop methods. Interesting methods presented included an approach to social return on investment; an approach for assessing capacity development and my presentation on ‘Collaborative Outcomes Reporting’ (nee PPSR) also went down very well and raised lots of interest.

So what did we conclude about ‘rigorous evaluative practice that embraces complexity’. Well I can tell you my musings. Firstly, I think there are possibly more implications for appropriate program design and contracting when working with complex program elements. The designs will need to be more dynamic, with a greater focus on process that prescribed SMART targets. With regard to evaluation, the things that come to mind would be an emerging need for:

• short cycles of measurement and reflection, so as to repeatedly check to see if things are working and if not modify.
• inclusion of methods ( ‘butterfly nets’) to catch unexpected outcomes: such as MSC technique
• sense-making approaches to decide what to do about the unexpected outcomes
• broad outcomes that are not too prescriptive
• iterative program logic/ program theory
• values inquiry, as if there are lots of unexpected outcomes, where will also be a range of way of valuing these outcomes that has not already been established
• rigour: generally mixed methods are advocated, but whatever methods are used it is still important that they hold up to scrutiny
• participatory approaches can be appropriate when working in the more complex arenas as they often allow spaces for sense-making, deliberating on outcomes and ensuring changes in program direction are appropriate to different stakeholder groups.

For Clear Horizon the conference was an endorsement that we are on the right track with the methods and products that we are developing and using. There was also plenty of exciting food for thought. One of the ideas of the Cynefin Model is to try to identify the knowable and known aspects of a program, which requires a greater emphasis on quantitative methods. I was also excited by some of their sense making techniques; we would need to find some hexagonal post-it notes!