A conversation with Dr Jess Dart, Anna Powell and Dr Ellise Barkley (Part 1)
While systems-change and place-based approaches to social issues have been growing in number and popularity over the past decade, our sector remains very much in the ‘exploration’ stage when it comes to tried and tested methodologies for answering the key questions of: Do they work? And, are we creating (our desired) impact?
In fact, it was less than two years ago that the Place-Based Evaluation Framework was developed in Australia, a collaborative undertaking that offers a set of minimum standards for evaluators and change-makers. As proof of concept, the framework was tested with the Logan Together, a collective impact initiative working towards population-level change for kids and families in the region of Logan.
It was an illuminating process. And there’s still much to be learnt. While the exercise identified many methodologies and resources from the evaluator’s toolkit that could, with some tweaks, prove useful for evaluating change, there were often more questions raised than answered. These systems-change and place-based approaches surfaced issues around power, trust, and leadership that many evaluators hadn’t previously had to grapple with to the same extent. Instead, evaluators and change makers found themselves needing to draw from a broad range of disciplines, fusing them together and constantly improvising as they went.
In last month’s webinar, we talked with two of the key authors of the national framework, our own Jess Dart and Ellise Barkley, along with systems change thought-leader Anna Powell about the challenges and opportunities presented by these approaches, and why evaluators in the space are truly the jazz players of the evaluation world.
Before we dive into the challenges, what are some of the defining aspects of systems-change and place-based approaches?
Dr Jess Dart
At a very practical level, systems-change and place-based approaches require different ways of working, different funding cycles, and different expectations. We need new systems, tools and contracts to support the complex, interconnected nature of the work and the myriad of people and roles involved. This is never more evident than when we consider the timeframes involved with systems-change and place-based approaches.
A report from the Spark Policy Institute looked at 24 collective impact initiatives and found that the average time for achieving population change was 9 years or more. Most funding cycles simply aren’t set up to manage results in this way. So managing expectations around results over such a long timeframe is a big challenge. It can lead to what Mark Cabaj refers to as expectations failure. No matter how much you keep telling funders what’s feasible, they still expect to see population level results earlier.
The final challenge I want to share is personal – we as evaluators need to accept that sometimes evaluation is part of what may be holding problems in place. We need to think about how “we show up”. We need to re-think what our roles are, and how we support others within the initiative. We need to show a willingness to adapt and improvise as the system shifts and we need to respond to unexpected knock-on effects. It’s a big, noisy, interconnected band we’re all part of, and we’ve all got our own notes to play. Sometimes we’re in harmony, sometimes it’s discordant, and we have to be ok with that.
In Part 2 of this series, we’ll delve into the learnings and practical approaches used by our panellists when working in this emergent space.
And for those looking to learn more, there are still spots available in our Evaluating Systems Change and Place-Based Approaches course, starting August 31.