A reflection on the recent Office of Development Effectiveness (ODE) ‘Evaluation of DFAT Investment Level Monitoring Systems’ (Dec 2018)

Disclaimer: the views expressed below are solely those of Damien Sweeney and do not represent Clear Horizon’s.

The ODE recently released a report on DFAT Investment Level Monitoring Systems, with the purpose of improving how Australian Aid investments are monitored. The report focused on the design and use of monitoring systems by DFAT investment managers and managing contractors.

My first observation was the title of the report, specifically the term ‘monitoring systems’. Why? Because so often Monitoring is joined to Evaluation (M&E), which in my experience can (and often does) lead to confusion between what is monitoring, and what is evaluation…. sometimes with the focus shifting to evaluation, at the expense of monitoring. This confusion between the M and the E is most often seen in field/implementation staff, who are often responsible for the actual data collection on a day-to-day basis.

I’ve been reflecting on this issue a fair bit over the past decade, having provided M&E backstopping to programs facing a distinct lack of monitoring and adaptive management, as well from developing monitoring, evaluation and learning (MEL) frameworks and plans (the jargon and acronyms in this field!).

Differentiating between little ‘e’, and big ‘E’

Monitoring is commonly defined as the systematic collection of data to inform progress, whereas evaluation is a more periodic ‘evaluative’ judgement, making use of monitoring, and other information.

However, as the evaluation points out, good monitoring is critical for continual improvement, by managing contractors (and other implementers) and DFAT investment managers. Continual improvement through monitoring requires an evaluative aspect too, as managing contractors (field/implementation teams, M&E advisors, leadership) and DFAT investment managers reflect on progress, and make decisions to keep going, or adjust course. I refer to this regular reflection process as little ‘e’, as differentiated from more episodic assessment of progress against key evaluation questions, or independent evaluations, which is the big ‘E’ (in M&E).

Keeping monitoring systems simple

Einstein was credited with the quote “Everything should be made as simple as possible, but not simpler”. This should be a principle of all monitoring systems, as it will promote the ownership across all responsible parties, from M&E advisors who develop systems, to those that will collect data and use it for continual improvement.

I have often seen cases where field/implementation teams don’t understand, and therefore don’t feel ownership, of complex M&E systems. A literature review supporting the report (Attachment A) notes the that better-practice monitoring systems are kept as simple as possible to avoid the lack of implementation that generally accompanies complex monitoring systems (too many indicators, too much information, and resultant paralysis).

The need for a performance (and learning) culture

Interestingly but not surprisingly, a survey of managing contractors noted that ‘good news’ often took precedence. This goes back to the importance of a performance culture across DFAT and managing contractors (and subcontractors) that embraces the opportunity to learn and improve (safe-fail vs fail-safe). There needs to me more incentive for managing contractors and investment managers to reflect, learn and adapt, and not just focus on the positives.

The importance of fostering a strong performance (and learning) culture is expressed in the recommendations. Learning should not be from periodic evaluations, but a regular and continuous process, with the regularity of reflection driven by the operational context (more complex contexts requiring more regular reflections of what monitoring information is indicating). I know of investments where implementation staff will informally meet on a weekly or fortnightly basis to track progress and make decisions on how to improve delivery.

Building capacity

The literature review notes the importance of capacity of staff for effective monitoring. I like to use the term capability (knowledge and skills) along with capacity (time and resources), as both are required, and are yet distinct from each other. The literature review focused on the importance of managing contractors recruiting and holding on staff who could design and manage monitoring systems. However, my experience indicates that it is not the M&E advisors that are a constraint or enabler of good monitoring systems, but the ownership of the system by those that implement the programs. Therein, for me, lies a key to good monitoring systems – getting field/implementation staff on board in the design and review of monitoring systems, so that they understand what is to be collected and why, including how it helps their work, through improving performance.

What we’re doing at Clear Horizon to focus on monitoring emergent outcomes and facilitate adaptive management

Clear Horizon has been developing fit-for-purpose plans and tools for our partners and clients, linking theory and practice and continually reflecting and learning on how to improve this.

I’m currently work shopping with my Clear Horizon Aid Effectiveness colleagues how we can make M&E tables more clearly accentuate the M, and how this informs the E. More to come on that front! Byron Pakula will be presenting at next week’s Australasian Aid Conference a we developed titled ‘No plan survives contact with the enemy – monitoring, learning and evaluation in complex and adaptive programming’ that takes in key issues raised in ODE’s evaluation. So check that one out if you’re in Canberra.

What are your thoughts on ODE’s evaluation of monitoring systems?

In parallel, to our reflections on our work through the Breath In sessions, we have been working out how we can do Breath Ins in a way that is worthwhile for all involved, that respects our associated ‘responsibilities’ and manages for some of the inherent conflicts present in the group.

Participating in the Breath In sessions are a CEO, three consultants, a government employee working in a central backbone and a backbone leader based out of a NGO. We come together well because we are all practitioners and all have a connection to Clear Horizon.

This is our third Breath In, and it feels as if after a rocky start, we have come to a much better place where some of the obvious conflicts have settled down. There is a much greater level of trust and understanding of each other and our different contexts. This is allowing us to have more open discussions … maybe even generative discussions.

Through place based work I have been introduced to different theories, one of them is Otto Sharmer’s change management theory, the Theory U which has a strong focus on listening as a means for transformation. He describes four levels of listening:

1.       Downloading – “yeah, I know that already..” re-confirm what I already know. (I-in-ego/Politeness) – Listening from the assumption that you already know what is being said, therefore you listen only to confirm habitual judgements.

2.       Factual – pick up new information…factual, debates, speak our mind (I-in-it/Debate) – Factual listening is when you pay attention to what is different, novel, or disquieting from what you already know.

3.       Empathic – see something through another person’s eyes, I know exactly how you feel. Forget my own agenda (I-in-thou/Inquiry) Empathic listening is when the speaker pays attention to the feelings of the speaker.  It opens the listener and allows an experience of “standing in the other’s shoes” to take place.  Attention shifts from the listener to the speaker, allowing for deep connection on multiple levels.

4.       Generative – “I can’t explain what I just experienced” (I-in-now/Flow) – This deeper level of listening is difficult to express in linear language.  It is a state of being in which everything slows down and inner wisdom is accessed. In group dynamics, it is called synergy.  In interpersonal communication, it is described as oneness and flow.

I found it useful to reflect on our Breath In journey through these four levels of listening. I can’t speak for everyone else, so from my perspective I have observed myself do the first level of listening really well! I think that I had moments of factual listening (comparing the work across our experiences), I think I had instances of empathetic listening with one person at a time and I’m not sure I was able to reach much beyond that.

I’m curious as to how everyone else felt. I’m also aware that the transformation needs to happen in each of us first before it can happen in the group. So I suppose I have some homework to do!

The above is a reflection on the deeper experience of the Breath In. At a different level, that of developing understanding and theory I think we achieved more than we have in the past. See previous blog!

The New Year has once again reared its head, leaving the dusty resolutions of 2018 on the cupboard shelf next to the re-gifted ‘bad santa’ present from last December’s Christmas party (unless you got home made sweets or condiments that is!!). Whether our Clear Horizonites had relaxing tropical holidays or productive working staycations here in Melbourne, all team members are ready and eager for and exciting 2019.

Last year saw Clear Horizon’s first steps (of many) into digital evaluation techniques, huge steps towards creating frameworks for evaluating place based initiatives and the fine tuning of Clear Horizon’s approach to evaluating co-design processes. Needless to say it was a big year! In 2019 we are looking ahead to hone in our participatory skills, move further into the digital space and build on the co-design work from 2018.

2019, we’re ready for you!

Some of our staff have shared their goals for this year.

Jen Riley, Digital Transformation Lead

“Digital Transformation super highway for Evaluation”

In 2019, I am looking forward to leading Clear Horizon in digitally transforming from the inside out. I want to learn more about artificial intelligence, machine learning and blockchain and what these new developments mean for the social sector. I am especially interested in how we harness the digital transformation super highway for evaluation and make data collection, reporting and evaluation more automated, agile and innovative to meet the demands of evaluating complex social issues. I am excited about getting the Clear Horizon Academy, an online digital learning space for co-evaluators, up and going and seeing Track2Change, our data visualisation and reporting platform become part of everything we do at Clear Horizon.

Kaisha Crupi, Research Analyst

“Breathing life into quantitative data”

In 2019, I would like to further work on my quantitative skills in an evaluation. As I enjoy bringing qualitative voices to life in an evaluation, I would like to work on my skills for quantitative data to ensure that this can also be done. It’s not just making pretty graphs and charts – it’s about making meaning of these numbers and polishing it to make them robust and as effective as can be.

Georgia Vague, Research Analyst

“Using the context that matters”

Being a new member of Clear Horizon in late 2018, my resolution for 2019 is two-fold. Firstly, I would like to strengthen my data-analysis skills, particularly strengthening how to analyse large amounts of data using the most appropriate, context specific techniques. Secondly I want to be able to gain confidence in my facilitation skills, particularly in participatory workshops. This means being aware of any unconscious bias that I might hold and really placing the client and participant voice in the centre of the evaluations.

Eunice Sotelo, Research Analyst

“Capacity development for all”

If 2018 was a big year of learning and discovery, 2019 is no different. In fact, I want to extend myself further – honing skills in facilitation and stakeholder engagement – while continuing to expand my evaluation toolkit. I’m also keen to dig deeper into capacity building, internally at Clear Horizon and with our clients. I think we can do better at making our practice more inclusive and accessible, and what better way than to ‘teach’ by example.

Ellise Barkey, Senior Principle

“Applying, trialling and improving our approaches to co-design”

In 2019 I am looking forward to continuing my learning with the inspired communities and partners around Australia working to create positive change for families, children and young people. My resolution is to deepen my understanding and practice of designing relevant and flexible approaches and tools that cater for the diverse learning and evaluation needs of these fabulous collectives driving place-based approaches and systems level change. Clear Horizon’s work last year developing the Place-based Evaluation Framework for the Commonwealth and Queensland Governments made good ground towards a relevant framework, and was a fascinating exercise as it was co-designed with many stakeholders. This year, I look forward to applying, trialling and improving on these approaches with partners and clients, and embracing a learning stance through the challenges and successes.

Jess Dart, CEO

“Building co-evaluation – getting everyone involved!”

In 2019 I want to think deeply about how we strengthen practice and tools around collaborative and participatory evaluation – the time has come to re-invigorate this practice! The world of co-design has really begun to make inroads, so the time is ripe to build the practice of co-evaluation. I am going to dedicate my year to it!  I would love to see more diverse stakeholders really engaging in planning and analysis and co-designing recommendations.

Victoria Pilbeam, Consultant

“Learn about and from Indigenous evaluation approaches”

In 2019, I want to learn about and from Indigenous approaches to evaluation. Our team is increasingly getting invited to work with Traditional Owners in natural resource management spaces. We need to understand Indigenous evaluation methodologies to engage respectfully and effectively with rights holders. More broadly in the Sustainable Futures team we are always evaluating at the interface between people and environment.  Evaluation methodologies based on a holistic understanding of people and nature could play an important role in informing our practice.