The client 

The Department of Communities, Disability Services,and Seniors and Department of Social Services, in partnership with Logan Together.

 

Clear Horizon was commissioned to lead the co-design and development of the Place-based Evaluation Framework and test the framework with Logan Together as proof of concept. Logan Together is a large place-based initiative (established in 2015)applying a Collective Impact (CI) approach to improve the wellbeing of children (0-8 years) in Logan, Queensland.

 

As part of this work, in 2018, Clear Horizon conducted a Progress Study for Logan Together focused on the progress towards shared outcomes of the collective and the enabling role of the backbone team.

The challenge 

Place-based delivery approaches (PBAs) are not new and have increasingly received attention and investment by Commonwealth and State Governments over the past decade. However, at the time of this project’s inception, no Australian framework for evaluating PBAs existed, including for collective impact initiatives, and there was limited understanding of how to measure progress of PBAs due to their complexity, the number of stakeholders involved and ‘moving parts’,and their long-term nature.

Ahead of the aspirational longer-term social and environment changes that PBAs work towards, we find that community, collaboration, and funding partners are asking:

·        “How do we know if we are on track?”

·        “How do we know our collective impact model is working?”

·        “Is our collaboration in good shape?”

·        “How good is our process and practice principles?”

·        “Are the early changes we are seeing going to lead to the longer-term population/environmental impact? ”

These questions are important to answer for all partners involved. Communities need evidence for building the movement, funders need assurance, and the collective needs to measure progress, learn from the work and adapt.

And there is not a ‘one size fits all’ evaluation strategy or methodology that can be applied. In the past, many shared measurement systems have also lacked the necessary focus on the intermediate community and systems level outcomes as a means for gauging progress.

Evaluating and tracking progress of PBAs is challenging and emergent work. Here is how we approached it for the Logan Together Progress Study (now available in the public domain).

The solution 

Clear Horizon worked closely with Logan Together to co-develop a short and long term Monitoring, Evaluation and Learning (MEL) strategy (aligned with the Place-based Evaluation Framework).This was supported by a 2 year operational plan. We worked with The Australian Centre for Social Innovation (TACSI) to co-design this process.

Arising from this, the progress study was designed as phase appropriate for the Logan Together initiative, and set clear expectations for what types of outcomes would be reasonable to expect at their level of maturity (along the collective impact journey). Population level change were not expected for this phase, and the focus was on investigating early instances of impact for families and children and systemic outcomes. The methodology of the study also fit within Logan Together’s cycle of learning and looked to incorporate ‘community voice’into the evaluative process.

For the progress study we used a mixed methodology based on a modified version of outcomes harvesting which involved a broad-scale sweep of outcomes potentially relevant to or linked to Logan Together activity since inception. We did this via semi-structured interviews, the Most Significant Change technique and desktop scans of existing data collected. You can readabout the methodology in the full report.

Progress was measured against Logan Together’s plan for change (their Roadmap) and the underpinning theory of change. The Roadmap is based on international research, community priorities and the shared outcomes that people are expecting to see as Logan Together moves forward.  Underpinning the Roadmap is a more general theory of change which maps out the expected enablers for change, and the systemic outcomes considered necessary to support sustained population level changes. Clear Horizon refined the theory of change as part of the process of designing the MEL strategy ahead of the progress study.

The benefits 

The progress study was conducted as a participatory process and involved building the capacity and capability of the Backbone Team to collect data, and track and evaluate progress. The team were trained and supported to conduct ‘Most Significant Change’ interviews and develop measurement tools (such as performance rubrics).

The Progress Report provides a comprehensive and rigorous assessment of LoganTogether’s progress toward shared outcomes to date. It also documented the Logan Together journey in ways that had not been previous done. Accompanying the report, Clear Horizon delivered a 'Statement of Achievement' summarising the key results and areas of attainment. A tranche of 20 Most Significant Change stories were also transcribed and provided to Logan Together.

For more information about the Logan Together initiative visit their website.

Photos by Logan Together.