In our last article, we talked with co-creators of Australia’s first Place-Based Evaluation Framework, Jess Dart and Ellise Barkley, along with systems-change expert Anna Powell about the unique challenges these approaches present. In this week’s instalment, we’ll be discussing practical techniques and tips for navigating evaluation in this emerging field, including how to make sure everyone’s on the same songsheet (even if we’re all playing different notes!).
You mentioned systems change is non-linear – how do you measure change when you don’t know what it’s going to look like or what to expect?
Dr Jess Dart
To address emergence you are going to need different types of measures that can uncover unplanned or unexpected changes in the system and help you make sense of them. This often means qualitative and dynamic methods, such as Most Significant Change, Outcomes Harvesting and Significant Instances of Policy and Systems Influence (SIPSI). These methods search for change, harvest and make sense. They do not predict ahead of time what the change will be, instead they throw out a broad net to catch changes in the system and make sense of them.
You mentioned that evaluators, we have on an ongoing learning role – what are some of the things you’re still learning or wanting to learn more about?
Dr Jess Dart
I’m still wrestling with how to develop indicators of systems health. There is potentially another piece of the measurement puzzle where we have lead indicators to help us see whether we are on track to a more optimised system. There is an appetite to co-design these indicators of the system health with communities. In the environmental sector I was often taken by the notion of “systems health indicators”, for example platypus prefer clean water, so where you see them thriving it is a sign that the ecosystem is healthy. So I’m interested in what would be equivalent of signs of a healthy system in the social sector. It might be how the system is experienced by the community, whether they feel it is safer and healthier. That’s is one area I hope to work on more in the next year or so.
I think we have much more to learn about how we can decolonise evaluation so that methods and approaches are culturally safe and relevant, particularly with our First Nations partners. We are learning through our partnerships, such as with Blak Impact, that decolonising evaluation means honouring Traditional ways of knowing, being and doing and recognising that First Nations cultures and people were evaluators before First Contact. It is about a two-way learning and facilitation of non-First Nations learning, and involves holding traditional knowledge and Western knowledge to further goals for First Nations people.
With many of the initiatives we work with, we’re still in the process of learning from our First Nations and community partners how to work together in culturally safe ways so the voices of groups that are often not heard in evaluation are included and amplified. First Nations communities, Culturally and Linguistically Diverse Groups, families and children – how do we ensure these voices are heard and represented throughout the MEL cycle? It takes intention, time and resources for good processes and relationship building.
I think place-based approached can be an incredible opportunity to advance this, where we can contribute to knowledge and power sharing among First Nations and non-Indigenous communities through our MEL work, with the ultimate goal of supporting power-sharing, self-determination, and reconciliation.
For those looking to delve deeper into this topic, there are still spots available in our Evaluating Systems Change and Place-Based Approaches course, starting August 31.