Posts

Tips for developing a theory of change that you and your team will love

Properly planned for and used, your impact measurement system and the thinking that comes with it can be one of your most powerful tools for driving and amplifying change.

The 9 Most Common Measurement, Evaluation and Learning Questions – Answered

Properly planned for and used, your impact measurement system and the thinking that comes with it can be one of your most powerful tools for driving and amplifying change.

Our Community

Sometimes it can be challenging being the only evaluation or impact measurement specialist in your organisation. You might enjoy a unique bird’s eye view across all of your organisation’s activities and impact. But you may also sit (virtually, these days) slightly apart from the teams knee-deep in design and implementation. And so when you run into a really sticky MEL issue – where do you go to for advice?

ChangeFest 2021

An invitation to deepen place-based approaches

I acknowledge the Traditional Owners of the land on which we met for ChangeFest, the Larrakia People, and pay respect to Elders past, present, and emerging. Thank you to the Larrakia Elders, hosts, and community, for the generous welcome and for sharing your rich insights and experience.

During 8th – 11th June, hundreds of place-based change-makers met on Larrakia country for ChangeFest 2021 in Darwin. The national gathering has been happening since 2018 and its focus on First Nations inclusion and leadership was strong. Participating were community-led collaborations, service providers, project teams, networks, activists, government partners and policy makers, sponsors, community members, designers, and evaluators interested in improving social outcomes for Australian communities.

Key take-homes to deepen practice

Inspiring examples of local and systems led change were shared, as well as lessons and stories of trauma and inequity. By design, and through this sharing, I left with a take-home pack of timely questions that will help me deepen my place-based practice.

  • First, ChangeFest asked me to reflect on how I can more fully and proactively bring The Uluru Statement from the Heart to life in my work and partnerships.
  • Second, I was asked to explore how I show up in collaborations (the ‘dance’ we do with others in the system), and look behind the curtain at how and why I might step back from collaborating when things get challenging (check out Deep Collaboration).
  • Third La Boite Theatre asked: What commitments can I make to include young people and what commitments will I take back to them?

The first one is particularly important – it will take me beyond endorsing and sharing the statement to deeply engaging with it in conversation with others and using it as a compass for my practice.

Enablers and barriers of place-based approaches

As an evaluator specialising in systems change and place-based approaches, I am always interested in hearing insights about what’s helping support systems change and what’s getting in the way when it comes to community-led change and assessing progress and social impact.

A few of the resounding enablers for place-based change included:

  • Importance of building respectful and trusted relationships and networks, and then embedding these so collaboration goes beyond only relationships.
  • Keeping community voice at the forefront and listening deeply to community.
  • Connecting people and efforts across the ‘system’.
  • Drawing on/ using Indigenous ways of working to keep culture at the centre of practice for First Nations communities. One wonderful example was from Galiwin’ku Connected Beginnings who are using local Yolŋu metaphors, language, and their kinship model as the central framework for their collective practice and measurement, evaluation and learning.
  • Using data and stories to demonstrate change, build momentum, and celebrate the wins (see Clear Horizon’s resources page for tools to help with this).

On the flipside, some of the ongoing challenges expressed by change-makers included lack of trust between partners and community, the impact of trauma on individuals, communities, and intergenerationally, lack of communication, duplication and fragmented efforts across systems, and the ongoing work of power sharing to enable communities to have greater control and self-determination.

These themes certainly fit with what we see through our work with change-makers in place-based contexts. And never too far away were the much-asked questions ‘How do we know if we are on track given all this complexity?’ or ‘Are we making a difference for kids and families?’

For my own practice, knowing if we are making a difference requires the short and messy loops of learning about what’s working in place, the shifts happening in the way people collaborate and put equity for First Nations peoples at the centre, and tracking how our mindsets and practice affects collectively driven change. Simultaneously, robust ways of checking impact over the long term are needed to show the contribution of our place-based work in improving the experience and lives of local communities. This requires looking at the numbers and stories that matter for community and partners and acknowledging there will be diverse perspectives on what counts as evidence and that specific outcome areas we might be seeking to improve such as ‘health’, ‘education’ or ‘identity and culture’ are all interconnected in the bigger story of social change.

Thanks to all the change-makers and ChangeFest organisers for getting us together to deepen our place-based work. It was a rich week of connecting and learning.

Hero image: The Change-makers of ChangeFest

Image above: Opening ceremony and Welcome to Country with Aunty June and Larrakia community

Images courtesy of ChangeFest organisers 2021

#ChangeFest

#UluruStatement

What it takes to build a Liveable Company

If this year has proven anything, it’s that we are capable of change and of rising to new challenges. In that spirit, we’re challenging ourselves to better walk the talk and build a liveable company. But what does that mean?

Let’s bridge the cultural divide: why evaluation and anthropology need each other

Research Analyst Alessandra Prunotto reflects on the Australian Anthropological Society conference, 2-5 December 2019.

This year’s Australian Anthropological Society conference in Canberra was themed “Values in anthropology, values of anthropology”. With this focus on values, you might assume that there would be a panel on evaluation, or at least a paper or two.

But there wasn’t.

How is this possible? I thought. After all, anthropology and evaluation are disciplines that overlap significantly.

Socio-cultural anthropology is the study of human societies in their environments. And evaluation is the practice of ascribing value to an initiative that’s creating change within a socio-environmental system. So at an anthropology conference on values, some debates on the thorny issues in evaluation practice seemed like par for the course.

But this was not the case. Instead, the undercurrent of concern that permeated the conference was the value that anthropology as a discipline has to those outside academia.

I suspect the conference theme of “value” manifested in this way because the majority of attendees were academics or PhD students. I believe that some of these intellectuals are struggling to understand the meaning of their work in a world where the pursuit of knowledge for its own sake is seen as retreating into an ivory tower.

As I listened to the various discussions circulating around “applied” and “public” anthropology, I noted down the themes emerging around the principles and skills that anthropology can offer the world beyond academia.

Looking around at the work we’re doing at Clear Horizon, I believe anthropology has much to offer evaluation.

In what follows, I expand on the ways that anthropological perspectives can help us navigate emerging trends in evaluation practice, as we see change-making initiatives move towards systems-change approaches and as cultural safety becomes essential for evaluation in First Nations contexts.

Anthropology’s superpowers

What are anthropology’s strengths, in comparison with other social sciences? At the conference, I noticed several themes emerging that painted a picture of applied anthropology done well.

Understanding of the other. As the comparative study of human societies, anthropologists intend to understand other people’s perspectives, no matter how radically different. This requires you to listen deeply and attentively, with humility. You aim to learn and empathise before reflecting critically.

Challenging assumptions. With an awareness that your way of being and thinking is one of many, anthropology compels you to question your own assumptions, as well as that of others. Anthropology is a critical discipline that aims to surface the most fundamental aspects of different worldviews and value systems.

Bringing together different worlds. As the goal of anthropology is to bridge cultural divides, it gives you the foundation to bring together different stakeholders, creating a space for different voices and perspectives to come together. It helps you to find zones where mutual understanding can flourish.

Reflexivity and awareness of power dynamics. Sharpening your observation skills and handing you a toolkit of theorists from Fanon to Foucault, anthropology enables you to identify power dynamics and maintain an awareness of your own position within them.

Contextualisation and comfort with complexity. Anthropology aims to understand human behaviour and beliefs within a broader social, historical, environmental and political context. It asks you to embrace messiness and complexity, and avoid reductive explanations.

Anthropology in systems-change evaluation

Across the change-making sector, we’re seeing a move away from programs and towards systems-change initiatives, such as place-based and collective impact approaches.

A number of elements that characterise these approaches heighten the complexity required of evaluation practice. These elements include the increased number and diversity of stakeholders involved, long timeframes, the range of changes initiated and the difficulties in identifying how initiatives contribute to certain outcomes.

As we can see from the list above, anthropological training leaves you well-placed to tackle these kinds of evaluations.

It allows you to bring together diverse stakeholders and facilitate the negotiations between different value systems and worldviews. It helps you to understand and manage the power relations that might exist between these different groups. And sitting with the complexity of the work is a sign that you’re dealing with the reality of change, not boxing it down into something that is both manageable and fictional.

Anthropology in First Nations evaluation

Attitudes in relation to First Nations evaluation are changing. At Clear Horizon, we’re hearing more compelling demands for decolonising evaluation and power-sharing. A merely participatory and inclusive approach to evaluating with First Nations peoples is no longer acceptable – instead, evaluation needs to be conducted by First Nations peoples “as” First Nations peoples, a principle that is underpinned by the rights of Indigenous peoples in the UNDRIP.

Anthropological perspectives have much to offer to help non-Indigenous evaluators collaborating with First Nations peoples navigate these issues sensitively and appropriately.

Most importantly, anthropology can provide a means to understand that hegemonic practices in evaluation and change-making are not the only ways of working, nor are they necessarily the right ones. Furthermore, anthropological perspectives can foreground how histories of colonisation can continue to manifest in power relations and structural disadvantage today. These aspects of anthropology can lend the humility needed to truly shift evaluative practice back to First Nations peoples and communities who have the expertise to evaluate in their own context.

Bridging the divide

It’s clear that anthropology can bring important skills and perspectives to the emerging trends in evaluation practice.

But it seems that evaluation is not on the radar as a potential career for anthropology students. As I’ve highlighted, it was not even mentioned at the most recent annual conference, despite the focus on “value” and a more public anthropology. And from my own experience, having graduated from an anthropology degree just last year, I’d never heard evaluation mentioned as a potential career path.

What can we do to bring evaluation and anthropology together?

One option might be a partnership, or at least more dialogue, between the Australian Evaluation Society and anthropology departments. Local chapters of the AES might have the potential to run careers information talks, mentorships, or even graduate placement programs.

Perhaps advertisements for evaluation positions could specifically mention that anthropology graduates are welcome in the list of disciplinary backgrounds accepted.

But the skills and ideas of anthropology should not just be limited to those who studied it at university. There could also be an opportunity for anthropologists to run training for evaluators who come from disciplines more removed from anthropology, to upskill them for these emerging areas of work.

Evaluation and anthropology have the potential to achieve more impact together. Evaluation can benefit from the nuanced, critical perspectives anthropology can bring. And anthropologists are seeking ways to make themselves useful outside universities.

Let’s reach out.

Reflections on transferring to the international evaluation space

Kaisha Crupi is a consultant at Clear Horizon and has recently made the move from the domestic Sustainable Futures team into Clear Horizon International. Below is a reflection piece from Kaisha, discussing her learnings and observations in her new role.

Before joining Clear Horizon 18 months ago, I had only a small taste of the international working world. However, since I made the full transition into ‘Clear Horizon International’, affectionately known as CHI (pronounced chai), I feel as if I have jumped into the deep end with each team member supporting me to learn how to swim, and to swim fast. I also feel that the rest of the organisation is on the sideline cheering me on. Below are my reflections and learnings from the last few months of being part of the CHI team.

Working in the international space is tricky

When I was in the domestic team I was exposed to many different industries, types of work and ways of working and met so many people who are passionate about their work through workshops, interviews and product development. Now starting to work in the international space, I am learning about things outside my home country, and in countries that I am working in (and not necessarily living in). Trying to understand different cultural customs, to work around language barriers, across different time zones and to understand different political systems and social contexts is proving to be quite tricky. I am learning a lot more, am asking a lot of questions and reading wider than my easily accessible news. I am also being kind to myself – I know that I am not expected to know everything and am not putting pressure on myself to do so, especially in a short amount of time!

The work is the same, but different

When I first joined the CHI team, I thought this would be great to learn a whole new skill set and challenge myself even further by learning something different and working in a different way. To my surprise, my first task when I joined the team was to conduct a document review against the monitoring and key evaluation questions for an upcoming workshop, which is something that I had finished doing for a domestic project a week earlier to feed into a report! The questions were similar and the way to go about it was the same. The only thing (which was quite a big thing, mind) was that the language and jargon was different, and instead of talking about an area or region, the project was focusing on a whole country! My biggest challenge in joining the team so far is getting used to all the acronyms in reports and discussions with my peers and our clients. I am slowly getting there; though someone should quiz me in the early stages of next year.

Understanding complex challenges

By going to a destination for an international holiday versus going for work, you learn about a country in a very different way. There is the saying that you should not discuss politics when you are in polite company – this is very different in the international working space, particularly when working in design, monitoring and evaluation. You learn about a country’s context on a more granular level, ask the difficult political questions and try to understand the country as much as you can, as fast as you can, especially whilst in-country. I have been exposed to the complex ways of working, what the client deals with and small spot fires they must put out on a day-to-day basis (which are quite different than in the domestic space). These issues also do not have quick-fix solutions. There is at time a feeling of helplessness – now that you know about this information, what are you going to do with it? I believe that doing design, monitoring and evaluation work helps with this, as knowledge is power and can be a communication tool to change someone’s life for the better.

I feel very fortunate to have landed where I am today. Not many people can say that they have ended up with their dream job, especially in such a great values-driven organisation in a very niche area. I have a great team of people supporting me and putting up with my insurmountable amount of questions and reflections, whilst also looking back fondly at my time in the domestic team, where I was able to build my strong foundational knowledge and be supported in every direction. I am looking forward to continuing to swim out toward the horizon and reflecting on the successes and challenges that are happening in such a complex world.

2019 Changefest: Evaluation Tools session

Evaluation Tools session, with Monique Perusco (Jesuit Social Services and Working Together in Willmot), Skye Trudgett and Ellise Barkley (Clear Horizon)

Our session started with a contextual summary of the work and characteristics of ‘Together in Willmot’, a collaborative social change effort in Mt Druitt involving The Hive, Jesuit Social Services, service providers, schools and many other partners. Clear Horizon is working with Together in Willmot as an evaluation partner. Our shared approach to learning and evaluation responds to the challenges of evaluating systems change and place-based approaches, and is tailored to the phase, pace and strengths of the collaboration. We introduced the process for evaluation we are undertaking, which has involved training in Most Significant Change Technique and local data collection which will feed into building a theory of change and then an evaluation plan. We are planning next year to do co-evaluation focused on the efforts and outcomes to date.

During the session we looked at examples of some Most Significant Change stories so far collected as part of this work.

Most Significant Change (MSC) technique was developed by Jess Dart and Rick Davies. Together Jess (Clear Horizon’s founder and CEO) and Rick authored the User Guide in 2005, and MSC is now applied to innumerable contexts worldwide. MSC story based method that can be used for participatory monitoring and evaluation. The process follows a simple interview structure that can generate a one page change story. It is participatory because many stakeholders are involved both in deciding the sorts of change to be recorded, and in analysing the data. MSC uses stories as a form of data collection. Stories are collected from those most directly involved, such as project participants and field staff. Stories are usually collected by asking a simple question such as ‘during the past year, what in your opinion, has the been the most significant change for participants as a result of this program? Stories are then collected, and stakeholders sit together to analyse the stories, at this time, participants are asked to select the story that represents the most significant change for them. The process of selecting the most significant story allows for dialogue from project stakeholders about what is most important. This dialogue is then used as evaluation data to create knowledge about the project and what it is achieving. 

We also covered concepts and tools for evaluating systems change and place-based approaches from the Place-based Evaluation Framework  and Place-based Evaluation Toolkit, which was commissioned by Commonwealth and Queensland governments last year and is a leading guide for evaluation in this context. We introduced the generic theory of change for place-based approaches and ‘the concept cube’ that shows the multiple dimensions of evaluation in this context. Clear Horizon worked with TACSI and CSIA to lead the co-design of the framework and have been working with government, community, philanthropy and non-government partners to test, apply and progress these learning, measurement and evaluation approaches.

AES Conference 2019 – for a first-time attendee & emerging Evaluator

By now I’ve had a few weeks to reflect on my first Australian Evaluation Society conference, where I was exposed to an amazing variety of evolving and inspiring ideas from within the Australasian evaluation community.

On the first day I found myself inspired by Dr Tracey Westerman, who raised the dubiousness of using or adapting Western-focused data collection tools in a First Nations context. For me, this highlighted the necessity of remaining reflective and adaptive in our approach, as well as tailoring every evaluation’s approach and methods to the context, preferably in partnership with community. This in turn made me reflect on the work of one of my Clear Horizon colleagues, Skye Trudgett, a proud Aboriginal woman who is passionately pursuing Indigenous Data Sovereignty and is leading a program to build First Nations young women’s evaluation capacity in remote Australia.

Following Dr Tracey Westerman’s plenary, I attended a Systems Evaluation Theory Application presentation by Lewis Atkinson, Brian Keogh and Ralph Renger, which helped frame my thinking regarding complexity. I found myself identifying with the concept of cascading success or failure as a neat way to consider the upstream challenges which produce a downstream issue. I could see the concept’s similarity to framing challenges through approaches such as the Problem Tree and found it a resonating concept in which to couch my thinking on casual pathways.

My third and final reflection was on the space for emerging evaluators. The conference provided a valuable sounding board for ideas and challenges facing those new to the profession and was highlighted on the final day by Eunice Sotelo and Francesca Demetriou, who are conducting research on these experiences. I found myself identifying closely with the key findings, and introspectively, the session highlighted a need to establish a community of support and practice for emerging evaluators. Personally, I will be seeking to develop an informal network, but I believe that an AES-associated group would be invaluable in attracting, retaining and developing those new to the space. Young people are not only the future (inevitably), but have the potential to bring new ideas, perspectives and approaches to evaluation. I feel that this potential influx of creativity and vigour should be encouraged and enhanced through more formal arrangements.

As Kailash Satyarthi is credited as saying, ‘No segment in the society can match the power, idealism, enthusiasm and courage of the young people’, or in this context, young (not necessarily age-specific) evaluators.

AES International Evaluation conference Day 2!

It’s been an amazing AES conference so far – lots of interesting topics and great conversations. Like Jess, the highlight for me so far has been the key note speaker from day one, Dr Tracey Westerman – an Aboriginal woman from the Pilbara in WA. She has been a trail blazer in Aboriginal mental health. The key take away message for me was that measurement matters – but even more importantly, the right measures matter. She described that in many cases of Aboriginal youth suicide, there was been no prior mental health assessment. But when assessment tools are used, they are western based and not culturally appropriate. This can lead to misdiagnosis. For over 20 years, Tracey has argued that it is not appropriate to ‘modify’ existing measures because of their inherent racism. The only way is to develop new tools from the ground up. Tracey has developed seven tools specifically for Aboriginal youth mental health with not a lot of funding – no easy feat. It was a truly inspiring presentation from an amazingly passionate and optimistic woman who really cares about her people.

A highlights from day 2 was a panel of designers and evaluators from Australia and New Zealand: Jess Dart, Kate McKegg, Adian Field, Jenny Riley and Jacqueline (Jax) Wechsler, who explored how we might move out of the traditional box of program evaluation, to make a bigger difference. They discussed the role of evaluators in supporting people to move beyond measuring to think though whether we are doing the right things and whether we are really making a difference across complex systems. Questions were covered such as where can evaluators add value in a co-design process, does evaluation get in the way and slow things down, do evaluators need new skills to help analyse and make sense of big data? Jenny reminded us that evaluators are learners and we are curious, and that we need to get on board with the digital revolution.

One really interesting concurrent session I attended was on the use of Rubrics by Julian King, Kate McKegg, Judy Oakden and Adrian Field. They presented the basics of rubric and then described how rubrics can be a tool for democratising evaluative reasoning, stakeholder engagement and communicating of results. They presented two very different examples– one in a developmental evaluation and the other was using rubrics to evaluate the value for money of an agricultural funding program. I found the second example particularly interesting having experienced the challenges of answering the value for money question. Using a rubric in this way is great for balancing the multiple dimensions of value from different perspectives.

Another memorable moment was at an ignite session (which is a really short presentation). Damien Sweeny and Dave Green from Clear Horizon did a great job at presenting a rather convincing argument for placing more emphasis on monitoring over evaluation – Big M vs small e as they call it. And they cheekily suggested changing the name of the AES to AMeS. An interesting thought.

The day finished with a second keynote speaker, Gary VanLandingham, from the Askey School of Public Administration and Policy. He reminded us of the vast amount of evaluative information available through ‘What Works’ warehouses. They are a good place to start when starting an evaluation, but there are warnings. The main caution for me is that they privilege certain types of data over others, and they don’t include what doesn’t work or things not measured using experimental approach (such as randomised control trials, and quasi-experimental methods).

The day was topped off by a short ferry ride to Luna Park where we had a lovely conference dinner overlooking the opera house. Sydney is a very beautiful city and a great setting for a wonderful conference.

Now for day three….

Have you visited our booth at the conference?