If this year has proven anything, it’s that we are capable of change and of rising to new challenges. In that spirit, we’re challenging ourselves to better walk the talk and build a liveable company. But what does that mean?
Research Analyst Alessandra Prunotto reflects on the Australian Anthropological Society conference, 2-5 December 2019.
This year’s Australian Anthropological Society conference in Canberra was themed “Values in anthropology, values of anthropology”. With this focus on values, you might assume that there would be a panel on evaluation, or at least a paper or two.
But there wasn’t.
How is this possible? I thought. After all, anthropology and evaluation are disciplines that overlap significantly.
Socio-cultural anthropology is the study of human societies in their environments. And evaluation is the practice of ascribing value to an initiative that’s creating change within a socio-environmental system. So at an anthropology conference on values, some debates on the thorny issues in evaluation practice seemed like par for the course.
But this was not the case. Instead, the undercurrent of concern that permeated the conference was the value that anthropology as a discipline has to those outside academia.
I suspect the conference theme of “value” manifested in this way because the majority of attendees were academics or PhD students. I believe that some of these intellectuals are struggling to understand the meaning of their work in a world where the pursuit of knowledge for its own sake is seen as retreating into an ivory tower.
As I listened to the various discussions circulating around “applied” and “public” anthropology, I noted down the themes emerging around the principles and skills that anthropology can offer the world beyond academia.
Looking around at the work we’re doing at Clear Horizon, I believe anthropology has much to offer evaluation.
In what follows, I expand on the ways that anthropological perspectives can help us navigate emerging trends in evaluation practice, as we see change-making initiatives move towards systems-change approaches and as cultural safety becomes essential for evaluation in First Nations contexts.
What are anthropology’s strengths, in comparison with other social sciences? At the conference, I noticed several themes emerging that painted a picture of applied anthropology done well.
Understanding of the other. As the comparative study of human societies, anthropologists intend to understand other people’s perspectives, no matter how radically different. This requires you to listen deeply and attentively, with humility. You aim to learn and empathise before reflecting critically.
Challenging assumptions. With an awareness that your way of being and thinking is one of many, anthropology compels you to question your own assumptions, as well as that of others. Anthropology is a critical discipline that aims to surface the most fundamental aspects of different worldviews and value systems.
Bringing together different worlds. As the goal of anthropology is to bridge cultural divides, it gives you the foundation to bring together different stakeholders, creating a space for different voices and perspectives to come together. It helps you to find zones where mutual understanding can flourish.
Reflexivity and awareness of power dynamics. Sharpening your observation skills and handing you a toolkit of theorists from Fanon to Foucault, anthropology enables you to identify power dynamics and maintain an awareness of your own position within them.
Contextualisation and comfort with complexity. Anthropology aims to understand human behaviour and beliefs within a broader social, historical, environmental and political context. It asks you to embrace messiness and complexity, and avoid reductive explanations.
Anthropology in systems-change evaluation
Across the change-making sector, we’re seeing a move away from programs and towards systems-change initiatives, such as place-based and collective impact approaches.
A number of elements that characterise these approaches heighten the complexity required of evaluation practice. These elements include the increased number and diversity of stakeholders involved, long timeframes, the range of changes initiated and the difficulties in identifying how initiatives contribute to certain outcomes.
As we can see from the list above, anthropological training leaves you well-placed to tackle these kinds of evaluations.
It allows you to bring together diverse stakeholders and facilitate the negotiations between different value systems and worldviews. It helps you to understand and manage the power relations that might exist between these different groups. And sitting with the complexity of the work is a sign that you’re dealing with the reality of change, not boxing it down into something that is both manageable and fictional.
Anthropology in First Nations evaluation
Attitudes in relation to First Nations evaluation are changing. At Clear Horizon, we’re hearing more compelling demands for decolonising evaluation and power-sharing. A merely participatory and inclusive approach to evaluating with First Nations peoples is no longer acceptable – instead, evaluation needs to be conducted by First Nations peoples “as” First Nations peoples, a principle that is underpinned by the rights of Indigenous peoples in the UNDRIP.
Anthropological perspectives have much to offer to help non-Indigenous evaluators collaborating with First Nations peoples navigate these issues sensitively and appropriately.
Most importantly, anthropology can provide a means to understand that hegemonic practices in evaluation and change-making are not the only ways of working, nor are they necessarily the right ones. Furthermore, anthropological perspectives can foreground how histories of colonisation can continue to manifest in power relations and structural disadvantage today. These aspects of anthropology can lend the humility needed to truly shift evaluative practice back to First Nations peoples and communities who have the expertise to evaluate in their own context.
Bridging the divide
It’s clear that anthropology can bring important skills and perspectives to the emerging trends in evaluation practice.
But it seems that evaluation is not on the radar as a potential career for anthropology students. As I’ve highlighted, it was not even mentioned at the most recent annual conference, despite the focus on “value” and a more public anthropology. And from my own experience, having graduated from an anthropology degree just last year, I’d never heard evaluation mentioned as a potential career path.
What can we do to bring evaluation and anthropology together?
One option might be a partnership, or at least more dialogue, between the Australian Evaluation Society and anthropology departments. Local chapters of the AES might have the potential to run careers information talks, mentorships, or even graduate placement programs.
Perhaps advertisements for evaluation positions could specifically mention that anthropology graduates are welcome in the list of disciplinary backgrounds accepted.
But the skills and ideas of anthropology should not just be limited to those who studied it at university. There could also be an opportunity for anthropologists to run training for evaluators who come from disciplines more removed from anthropology, to upskill them for these emerging areas of work.
Evaluation and anthropology have the potential to achieve more impact together. Evaluation can benefit from the nuanced, critical perspectives anthropology can bring. And anthropologists are seeking ways to make themselves useful outside universities.
Let’s reach out.
Kaisha Crupi is a consultant at Clear Horizon and has recently made the move from the domestic Sustainable Futures team into Clear Horizon International. Below is a reflection piece from Kaisha, discussing her learnings and observations in her new role.
Before joining Clear Horizon 18 months ago, I had only a small taste of the international working world. However, since I made the full transition into ‘Clear Horizon International’, affectionately known as CHI (pronounced chai), I feel as if I have jumped into the deep end with each team member supporting me to learn how to swim, and to swim fast. I also feel that the rest of the organisation is on the sideline cheering me on. Below are my reflections and learnings from the last few months of being part of the CHI team.
Working in the international space is tricky
When I was in the domestic team I was exposed to many different industries, types of work and ways of working and met so many people who are passionate about their work through workshops, interviews and product development. Now starting to work in the international space, I am learning about things outside my home country, and in countries that I am working in (and not necessarily living in). Trying to understand different cultural customs, to work around language barriers, across different time zones and to understand different political systems and social contexts is proving to be quite tricky. I am learning a lot more, am asking a lot of questions and reading wider than my easily accessible news. I am also being kind to myself – I know that I am not expected to know everything and am not putting pressure on myself to do so, especially in a short amount of time!
The work is the same, but different
When I first joined the CHI team, I thought this would be great to learn a whole new skill set and challenge myself even further by learning something different and working in a different way. To my surprise, my first task when I joined the team was to conduct a document review against the monitoring and key evaluation questions for an upcoming workshop, which is something that I had finished doing for a domestic project a week earlier to feed into a report! The questions were similar and the way to go about it was the same. The only thing (which was quite a big thing, mind) was that the language and jargon was different, and instead of talking about an area or region, the project was focusing on a whole country! My biggest challenge in joining the team so far is getting used to all the acronyms in reports and discussions with my peers and our clients. I am slowly getting there; though someone should quiz me in the early stages of next year.
Understanding complex challenges
By going to a destination for an international holiday versus going for work, you learn about a country in a very different way. There is the saying that you should not discuss politics when you are in polite company – this is very different in the international working space, particularly when working in design, monitoring and evaluation. You learn about a country’s context on a more granular level, ask the difficult political questions and try to understand the country as much as you can, as fast as you can, especially whilst in-country. I have been exposed to the complex ways of working, what the client deals with and small spot fires they must put out on a day-to-day basis (which are quite different than in the domestic space). These issues also do not have quick-fix solutions. There is at time a feeling of helplessness – now that you know about this information, what are you going to do with it? I believe that doing design, monitoring and evaluation work helps with this, as knowledge is power and can be a communication tool to change someone’s life for the better.
I feel very fortunate to have landed where I am today. Not many people can say that they have ended up with their dream job, especially in such a great values-driven organisation in a very niche area. I have a great team of people supporting me and putting up with my insurmountable amount of questions and reflections, whilst also looking back fondly at my time in the domestic team, where I was able to build my strong foundational knowledge and be supported in every direction. I am looking forward to continuing to swim out toward the horizon and reflecting on the successes and challenges that are happening in such a complex world.
Evaluation Tools session, with Monique Perusco (Jesuit Social Services and Working Together in Willmot), Skye Trudgett and Ellise Barkley (Clear Horizon)
Our session started with a contextual summary of the work and characteristics of ‘Together in Willmot’, a collaborative social change effort in Mt Druitt involving The Hive, Jesuit Social Services, service providers, schools and many other partners. Clear Horizon is working with Together in Willmot as an evaluation partner. Our shared approach to learning and evaluation responds to the challenges of evaluating systems change and place-based approaches, and is tailored to the phase, pace and strengths of the collaboration. We introduced the process for evaluation we are undertaking, which has involved training in Most Significant Change Technique and local data collection which will feed into building a theory of change and then an evaluation plan. We are planning next year to do co-evaluation focused on the efforts and outcomes to date.
During the session we looked at examples of some Most Significant Change stories so far collected as part of this work.
Most Significant Change (MSC) technique was developed by Jess Dart and Rick Davies. Together Jess (Clear Horizon’s founder and CEO) and Rick authored the User Guide in 2005, and MSC is now applied to innumerable contexts worldwide. MSC story based method that can be used for participatory monitoring and evaluation. The process follows a simple interview structure that can generate a one page change story. It is participatory because many stakeholders are involved both in deciding the sorts of change to be recorded, and in analysing the data. MSC uses stories as a form of data collection. Stories are collected from those most directly involved, such as project participants and field staff. Stories are usually collected by asking a simple question such as ‘during the past year, what in your opinion, has the been the most significant change for participants as a result of this program? Stories are then collected, and stakeholders sit together to analyse the stories, at this time, participants are asked to select the story that represents the most significant change for them. The process of selecting the most significant story allows for dialogue from project stakeholders about what is most important. This dialogue is then used as evaluation data to create knowledge about the project and what it is achieving.
We also covered concepts and tools for evaluating systems change and place-based approaches from the Place-based Evaluation Framework and Place-based Evaluation Toolkit, which was commissioned by Commonwealth and Queensland governments last year and is a leading guide for evaluation in this context. We introduced the generic theory of change for place-based approaches and ‘the concept cube’ that shows the multiple dimensions of evaluation in this context. Clear Horizon worked with TACSI and CSIA to lead the co-design of the framework and have been working with government, community, philanthropy and non-government partners to test, apply and progress these learning, measurement and evaluation approaches.
By now I’ve had a few weeks to reflect on my first Australian Evaluation Society conference, where I was exposed to an amazing variety of evolving and inspiring ideas from within the Australasian evaluation community.
On the first day I found myself inspired by Dr Tracey Westerman, who raised the dubiousness of using or adapting Western-focused data collection tools in a First Nations context. For me, this highlighted the necessity of remaining reflective and adaptive in our approach, as well as tailoring every evaluation’s approach and methods to the context, preferably in partnership with community. This in turn made me reflect on the work of one of my Clear Horizon colleagues, Skye Trudgett, a proud Aboriginal woman who is passionately pursuing Indigenous Data Sovereignty and is leading a program to build First Nations young women’s evaluation capacity in remote Australia.
Following Dr Tracey Westerman’s plenary, I attended a Systems Evaluation Theory Application presentation by Lewis Atkinson, Brian Keogh and Ralph Renger, which helped frame my thinking regarding complexity. I found myself identifying with the concept of cascading success or failure as a neat way to consider the upstream challenges which produce a downstream issue. I could see the concept’s similarity to framing challenges through approaches such as the Problem Tree and found it a resonating concept in which to couch my thinking on casual pathways.
My third and final reflection was on the space for emerging evaluators. The conference provided a valuable sounding board for ideas and challenges facing those new to the profession and was highlighted on the final day by Eunice Sotelo and Francesca Demetriou, who are conducting research on these experiences. I found myself identifying closely with the key findings, and introspectively, the session highlighted a need to establish a community of support and practice for emerging evaluators. Personally, I will be seeking to develop an informal network, but I believe that an AES-associated group would be invaluable in attracting, retaining and developing those new to the space. Young people are not only the future (inevitably), but have the potential to bring new ideas, perspectives and approaches to evaluation. I feel that this potential influx of creativity and vigour should be encouraged and enhanced through more formal arrangements.
As Kailash Satyarthi is credited as saying, ‘No segment in the society can match the power, idealism, enthusiasm and courage of the young people’, or in this context, young (not necessarily age-specific) evaluators.
It’s been an amazing AES conference so far – lots of interesting topics and great conversations. Like Jess, the highlight for me so far has been the key note speaker from day one, Dr Tracey Westerman – an Aboriginal woman from the Pilbara in WA. She has been a trail blazer in Aboriginal mental health. The key take away message for me was that measurement matters – but even more importantly, the right measures matter. She described that in many cases of Aboriginal youth suicide, there was been no prior mental health assessment. But when assessment tools are used, they are western based and not culturally appropriate. This can lead to misdiagnosis. For over 20 years, Tracey has argued that it is not appropriate to ‘modify’ existing measures because of their inherent racism. The only way is to develop new tools from the ground up. Tracey has developed seven tools specifically for Aboriginal youth mental health with not a lot of funding – no easy feat. It was a truly inspiring presentation from an amazingly passionate and optimistic woman who really cares about her people.
A highlights from day 2 was a panel of designers and evaluators from Australia and New Zealand: Jess Dart, Kate McKegg, Adian Field, Jenny Riley and Jacqueline (Jax) Wechsler, who explored how we might move out of the traditional box of program evaluation, to make a bigger difference. They discussed the role of evaluators in supporting people to move beyond measuring to think though whether we are doing the right things and whether we are really making a difference across complex systems. Questions were covered such as where can evaluators add value in a co-design process, does evaluation get in the way and slow things down, do evaluators need new skills to help analyse and make sense of big data? Jenny reminded us that evaluators are learners and we are curious, and that we need to get on board with the digital revolution.
One really interesting concurrent session I attended was on the use of Rubrics by Julian King, Kate McKegg, Judy Oakden and Adrian Field. They presented the basics of rubric and then described how rubrics can be a tool for democratising evaluative reasoning, stakeholder engagement and communicating of results. They presented two very different examples– one in a developmental evaluation and the other was using rubrics to evaluate the value for money of an agricultural funding program. I found the second example particularly interesting having experienced the challenges of answering the value for money question. Using a rubric in this way is great for balancing the multiple dimensions of value from different perspectives.
Another memorable moment was at an ignite session (which is a really short presentation). Damien Sweeny and Dave Green from Clear Horizon did a great job at presenting a rather convincing argument for placing more emphasis on monitoring over evaluation – Big M vs small e as they call it. And they cheekily suggested changing the name of the AES to AMeS. An interesting thought.
The day finished with a second keynote speaker, Gary VanLandingham, from the Askey School of Public Administration and Policy. He reminded us of the vast amount of evaluative information available through ‘What Works’ warehouses. They are a good place to start when starting an evaluation, but there are warnings. The main caution for me is that they privilege certain types of data over others, and they don’t include what doesn’t work or things not measured using experimental approach (such as randomised control trials, and quasi-experimental methods).
The day was topped off by a short ferry ride to Luna Park where we had a lovely conference dinner overlooking the opera house. Sydney is a very beautiful city and a great setting for a wonderful conference.
Now for day three….
Have you visited our booth at the conference?
Dr Tracey Westerman, proud Aboriginal woman, had me totally gripped throughout her keynote presentation on Day 1 at the AES International Evaluation Conference. She began with the statistic that Australia has the highest rate of child suicide in the word. But she cautioned us to be optimistic and focus also on the positive outcomes that are occurring, such as the six young Aboriginal people graduating from medicine in WA this year alone. She stressed that education is the most powerful solution in the world and described how in one generation her family ‘closed the gap’ – she gained a doctorate despite living very remotely, with a family background of very limited formal education.
She walked us through the importance of developing assessment tools that are culturally sensitive and to avoid confusing causes with risk factors. It seems many tools in existence today are culture-blind and can lead to stereotyping and actual discrimination. She has developed a whole range of specific assessment tools that are culturally sensitive, including an assessment tool for work-based culture. She made the case that there hasn’t been the right sort of research with Aboriginal people, that the causes are different, and need to be assessed in culturally sensitive ways.
She’s working on ‘building an army’ of Indigenous psychologists across the country to address child suicide and influence child protection. She ended with the note that there is nothing that can’t be achieved by Aboriginal people if they believe in themselves.
After this I had the privilege of moderating a panel about the client-consultant relationship, a topic dear to my heart and my business! The panel was from the Victorian Department of Education and training (DEET), as well as consultants from Melbourne Uni and Deloitte Access Economics. DEET have set up a ‘state of the art’ supplier panel with over 30 suppliers on it, and are working to deepen the partnership between evaluation suppliers and commissioners, as well as embedding a culture of evaluation. They were generous in sharing their challenges, including lots of tricky moments around data sharing and IP.
Just before lunch I had the pleasure of indulging in a session of evaluation theory led by Brad Astbury and Andrew Hawkins from (ARTD). The explored the seven great thinkers of evaluation, who laid the foundations of our seven-decade-long journey of building our theoretical foundations. So lovely to wallow in theory, I remember savouring that learning when studying for my PhD. Their conversation was framed around Foundations of Program Evaluation by Shadish, Cook and Leviton (1991) – it was my evaluation textbook back then and good to see it’s still valued!
It was a busy day for me, and I also convened a fun panel on digital disruption. We had a great spread of panellists, with a designer from Paper Giant, Ruben Stanton; a data scientist, Kristi Mansfield; a social procurement expert, Chris Newman; as well as our very own Chief Innovation Officer, Jenny Riley. We explored the trends, the opportunities and the scary stuff that might come with the fourth industrial revolution – yes, the robots are here! I saw a few people jump in their seats when Jen described how Survey Monkey stores data overseas and is not bound by the same data security regulations. We also looked into the amazing opportunities for evaluators to be the sensemakers of Big Data. When the firehose of data hits you in the face, maybe the evaluators will be there to bring you calmly back to the most important questions. We also explored the opportunities for evaluators to get involved in evaluating new technology and innovation, and to help consider how ready the system is to receive these innovations. I didn’t want it to end!
The day was topped off with a keynote from David Fetterman on empowerment evaluation. Empowerment evaluation is now 26 years old! David explained how empowerment evaluation is a self-evaluation approach designed to help people help themselves. It felt familiar!
Progress 2019 was held over 2 days in June to address the pressing social and environmental issues the world is currently facing. The conference is a biennial event at which progressive thinkers and change makers in the social and environmental space come together to discuss current issues. Over 1500 attendees from Australia and around the world attended the event at Melbourne Town Hall. Headline speakers included Anat Shenker-Osorio, Ellie Mae O’Hagan, Bruce Pascoe, Kumi Naidoo, Owen Jones and Behrouz Boochani. The two-day program was emceed by Yassmin Abdel-Magied.
Kaisha Crupi and Shani Rajendra represented Clear Horizon at Progress 2019 in a bid to understand how Clear Horizon can better engage in this space. Their key learnings are as follows:
- We work in complex and messy systems that are tricky to navigate
We found that intersectionality was a common thread throughout this year’s conference. The key speakers noted that Australia cannot achieve true social change unless social change includes all Australians, especially the interests of First Nations people. Panels such as the “First Nations Justice” and “First Nations Land Justice highlighted the significant gaps in current social change movements. Similarly, “A Very Human Climate Crisis” panel reinforced that Australia cannot achieve true social change unless it is coupled with environmental change. Progress 2019 drove home the message that our everyday challenges in the social and environmental change space are complex and multi-faceted in nature. It was an important reminder that we all need to work collaboratively across different initiatives to tackle such complex issues.
The discussion on intersectionality tied well with the other discussions in the conference around embedding a greater understanding of complex systems in which we operate into progressive practice. In his keynote address, Kumi Naidoo, the Secretary-General of Amnesty International, referenced the role of systems change. The three requirements where the world needs to push the boundaries include system re-design, system innovation and system transformation. With these requirements, we will be in a better position to achieve social and environmental change. Kumi’s sentiments were also echoed by Lyn Morgain, Chief Executive at CoHealth who stated that:
“We need to acknowledge that transformation is iterative, dynamic and murky.”
For us working at Clear Horizon, the discussion of intersectionality and systems thinking was really exciting for us as this is how we like to work. Clear Horizon prefers to work with organisations who are already working across multi-faceted issues, particularly where social and environmental justice intersect. This also includes systems-thinking projects and organisations.
- Working in complexity is best done through strong partnerships
Each panel discussion at Progress reinforced the importance of partnerships in trying to achieve social and environmental change. At the “Philanthropists & Changemakers: Effective Partnerships to Win Systemic Change” conversation, the panellists discussed that although developing relationships and partnerships are challenging, true partnership is rewarding once it is fully developed. It was noted that the key element is to come to an agreed understanding, in order to build trust. This includes the fact that building trust takes time and that understanding what everyone wants to achieve from the partnership (both as individuals and as together) takes careful research and negotiation. We learnt that all organisations can benefit from developing strong partnerships with like-minded organisations and working together to achieve a common goal.
From this discussion, it reinforced how lucky we are to work in a space which already has already existing partnerships, which have formed over the years and are now incredibly strong. Clear Horizon has long-standing relationships with several organisations. We have found that by working in increasingly complex environments, the need to have a wide range of strong partnerships helps have more of a social and environmental impact, rather than if we were going at it alone.
- We need to make space for marginalised voices
We were reminded throughout the conference about the importance of creating a space for people from underrepresented groups in the social and environmental change space. Speakers at Progress highlighted the often used and sometimes tokenistic “let them speak” approach to inclusion which is used a lot in current practice. Instead, change makers should be looking towards organisations who are providing a supportive platform and tools in which people from underrepresented groups can choose to use when speaking for themselves. Phillip Winzer from Seed said that the question that organisations should be asking themselves is:
“What can we do to support Aboriginal and Torres Strait Islander people to implement the solutions they want?”
This question made us reflect on how organisations currently work, both at Clear Horizon and other organisations we work with. We also thought about current Clear Horizon strategies in how to recruit people from underrepresented communities as well as how we work with these communities. This includes people who advising us on how to better create spaces for people to feel welcome and accepted.
Progress taught us that no matter how progressive or values-driven organisations are, to achieve social and environmental change, they need to be proactive in involving underrepresented groups. As Kumi Naidoo said in his keynote address:
“If progressive organisations do not internalise fully the challenges we face, then we are part of the problem.”
- We need to think about how we communicate
Progress also showed us that social and environmental sectors need to think about how they connect with wider audiences. This includes talking using more accessible and plain language so everyone can understand. This includes not speaking in jargon, providing image descriptions for people with a visual impairment, or communicating with drawings and cartoons rather than text. Anat Shenker-Osorio, Founder and Principal of ASO Communications did a presentation on communication in campaigning which highlighted the need for strengths-based communication that avoids “othering”. This was a quite important learning for organisations who are trying to use alternative voices in the spaces that they work in.
From our Clear Horizon perspective, we understood that we can further develop our practices. This includes how to communicate in our workshops with people with a disability (through digital storytelling or verbal visual descriptions), not using our evaluation and design (and sometimes sector-specific) jargon as well as having clear messaging and metrics of success.
Progress 2019 was a thought-provoking two-days. A major learning for us was how Clear Horizon could include more diverse voices in our spaces. This could be done through more meaningful partnerships and working on having jargon free, accessible communication. We felt that as a values-based, mission-driven organisation, the benefit for Clear Horizon in continuing to engage with progressive movements such as Progress is a must-need. We also realised that our current work in systems thinking, monitoring and evaluation, and design provide us with a useful toolkit to contribute to the social and environmental justice space. We hope to work with more organisations striving for this in the future.
Both facilities themselves and debates on their effectiveness have proliferated in recent years. Working papers, blog posts, conference presentations (see panel 4e of the 2019 AAC), DFAT internal reviews, and even Senate Estimates hearings have unearthed strong views on both sides of the ledger. However, supporting evidence has at times been scarce.
Flexibility – a two-edged sword?
One root cause of this discord might be a lack of clarity about what most facilities are really trying to achieve – and whether they are indeed achieving it. This issue arises because the desired outcomes of facilities are typically defined in broad terms. A search of DFAT’s website unearths expected outcomes like “high quality infrastructure delivery, management and maintenance” or – take a breath – “to develop or strengthen HRD, HRM, planning, management, administration competencies and organisational capacities of targeted individuals, organisations and groups of organisations and support systems for service delivery.” While these objectives help to clarify the thematic boundaries of the facility (albeit fuzzily), they do not define a change the facility hopes to influence by its last day. This can leave those responsible for facility oversight grasping at straws when it comes to judging, from one year to the next, whether progress is on track.
Clearly, broad objectives risk diffuse results that ultimately don’t add up to much. With ‘traditional’ programs, the solution might be to sharpen these objectives. But with facilities – as highlighted by recent contributions to the Devpolicy Blog – this breadth is desired, because it provides flexibility to respond to opportunities that emerge during implementation. The decision to adopt a facility mechanism therefore reflects a position that keeping options open will deliver greater dividends than targeting a specific endpoint.
Monitor the dividends of flexibility
A way forward might be to better define these expected dividends of flexibility in a facility’s outcome statements, and then monitor them during implementation. This would require hard thinking about what success would look like – for each facility in its own context – in relation to underlying ambitions like administrative efficiency, learning, relationships, responsiveness to counterparts, or multi-sectoral coherence. If done well, this may provide those in charge with something firmer to grasp as they go about judging and debating the year-on-year adequacy of a facility’s progress, and will assist them to manage accordingly.
This is easier said than done of course, but one tool in the program evaluation toolkit that might help is called a rubric. This is essentially a qualitative scale that includes:
- Criteria: the aspects of quality or performance that are of interest, e.g. timeliness.
- Standards: the levels of performance or quality for each criterion, e.g. poor/adequate/good.
- Descriptors: descriptions or examples of what each standard looks like for each criterion in the rubric.
In program evaluation, rubrics have proved helpful for clarifying intent and assessing progress for complex or multi-dimensional aspects of performance. They provide a structure within which an investment’s strategic intent can be better defined, and the adequacy of its progress more credibly and transparently judged.
What would this look like in practice?
As an example, let’s take a facility that funds Australian government agencies to provide technical assistance to their counterpart agencies in other countries. A common underlying intent of these facilities is to strengthen partnerships between Australian and counterpart governments. Here, a rubric would help to explain this intent by defining partnership criteria and standards. Good practice would involve developing this rubric based both on existing frameworks and the perspectives of local stakeholders. An excerpt of what this might look like is provided in Table 1 below.
Table 1: Excerpt of what a rubric might look like for a government-to-government partnerships facility
|Standard||Criterion 1: Clarity of partnership’s purpose||Criterion 2: Sustainability of incentives for collaboration|
|Strong||Almost all partnership personnel have a solid grasp of both the long-term objectives of the partnership and agreed immediate priorities for joint action.||Most partnership personnel can cite significant personal benefits (intrinsic or extrinsic) of collaboration – including in areas where collaboration is not funded by the facility.|
|Moderate||Most partnership personnel are clear about either the long-term objectives of the partnership or the immediate priorities for joint action. Few personnel have a solid grasp of both.||Most partnership personnel can cite significant personal benefits of collaboration – but only in areas where collaboration is funded by the facility.|
|Emerging||Most partnership personnel are unclear about both the long-term objectives of the partnership and the immediate priorities for joint action.||Most partnership personnel cannot cite significant personal benefits of collaboration.|
Once the rubric is settled, the same stakeholders would use the rubric to define the facility’s specific desired endpoints (for example, a Year 2 priority might be to achieve strong clarity of purpose, whereas strong sustained incentives for performance might not be expected until Year 4 or beyond). The rubric content would then guide multiple data collection methods as part of the facility’s M&E system (e.g. surveys and interviews of partnership personnel, and associated document review). Periodic reflection and judgments about standards of performance would be informed by this data, preferably validated by well-informed ‘critical friends’. Refinements to the rubric would be made based on new insights or agreements, and the cycle would continue.
In reality, of course, the process would be messier than this, but you get the picture.
How is this different to current practice?
For those wondering how this is different to current facility M&E practice, Table 2 gives you an overview. Mostly rubric-based approaches would enhance rather than replace what is already happening.
Table 2: How might a rubric-based approach enhance existing facility M&E practice?
|M&E step||Existing practice||Proposed enhancement|
|Objective setting||Facility development outcomes are described in broad terms, to capture the thematic boundaries of the facility.
Specific desired endpoints unclear.
|Facility outcomes also describe expected dividends of flexibility e.g. responsiveness, partnership
Rubrics help to define what standard of performance is expected, by when, for each of these dividends
|Focus of M&E data||M&E data focuses on development results e.g. what did we achieve within our thematic boundaries?||M&E data also focuses on expected facility dividends e.g. are partnerships deepening?|
|Judging overall progress||No desired endpoints – for facility as a whole – to compare actual results against||Rubric enables transparent judgment of whether – for the facility as a whole – actual flexibility dividends met expectations (for each agreed criterion, to desired standards).|
Not a silver bullet – but worth a try?
To ward off allegations of rubric evangelism, it is important to note that rubrics could probably do more harm than good if they are not used well. Pitfalls to look out for include:
- Bias: it is important that facility managers and funders are involved in making and owning judgments about facility performance, but this presents obvious threats to impartiality – reinforcing the role of external ‘critical friends’.
- Over-simplification: a good rubric will be ruthlessly simple but not simplistic. Sound facilitation and guidance from an M&E specialist will help. DFAT centrally might also consider development of research-informed generic rubrics for typical flexibility dividends like partnership, which can then be tailored to each facility’s context.
- Baseless judgments: by their nature, rubrics deal with multi-dimensional constructs. Thus, gathering enough data to ensure well-informed judgments is a challenge. Keeping the rubric as focused as possible will help, as will getting the right people in the room during deliberation, to draw on their tacit knowledge if needed (noting added risks of bias!).
- Getting lost in the weeds: this can occur if the rubric has too many criteria, or if participants are not facilitated to focus on what’s most important – and minimise trivial debates.
If these pitfalls are minimised, the promise of rubrics lies in their potential to enable more:
- Time and space for strategic dialogue amongst those who manage, oversee and fund facilities.
- Consistent strategic direction, including in the event of staff turnover.
- Transparent judgments and reporting about the adequacy of facility performance.
Rubrics are only ever going to be one piece of the complicated facility M&E puzzle. But used well, they might just contribute to improved facility performance and – who knows – may produce surprising evidence to inform broader debates on facility effectiveness, something which shows no sign of abating any time soon.
For details on the author, please click on the blog title immediately above, which will redirect you to the Devpolicy Blog.
Other recent articles on aid and development from devpolicy.org
The fate of leadership which aspires to leave no-one behind
Launch of the Development Studies Association of Australia
Managing the transition from aid: lessons for donors and recipients
Market systems and social protection approaches to sustained exits from poverty: can we combine the best of both?
Aid and the Pacific in the Coalition’s third term
This blog forms Part 2 of a 4 part blog series outlining the learnings and reflections of my two colleagues and I who were fortunate enough to attend the 2019 Indigenous Peoples Evaluation conference in Rorotura, New Zealand.
The conference was an inspiring and transformative experience for me, as a non-Indigenous person and evaluator. Having worked as a youth worker prior to entering into evaluation, I particularly resonated with the keynote speech given by Marcus Akuhata-Brown, who among many things, works with at-risk youth and young offenders. Marcus explored the topic of connection and its role in supporting young people and others when crisis occurs. The following is my short reflection, as a non-Indigenous person, on Marcus’s talk and it’s implications for evaluation and our society.
Stepping into the forest: Connection in community
Marcus used the analogy of a forest to explore the importance of connection in our society. He described the sensation of standing on the edge of a natural grown forest and stepping into it; feeling the sudden sensation of humidity and experiencing smells and sounds that you could not hear from the outside. This sensation can only occur as a result of the many interdependent and connected species surviving together and supporting each other as a whole. In contrast, Marcus compared this to stepping into a manufactured pine forest; with only a single species surviving neatly and independently. In the second instance you can smell a strong scent of pine but will not feel the same level of sensation that occurred in the first. Although the first forest is very complex and harder to understand when you look at the whole, you can appreciate a self-sustaining eco-system, which will survive for an incredibly long time with no human intervention.
This analogy is very powerful when reflecting on our own societies and the tendency, particularly in Western-dominated culture, to pursue objectivity, independency and scientific rationalism. Our desire to simplify, neaten things up and search for the absolute and independent truth results in a narrow understanding of communities and a separation of people from one another. Humans are complicated and communities are complex but what results is not something to shy away from; instead, it is something to embrace and work with to better our societies as a whole.
Connection as a support system
Marcus also reflected on his work with young people who have fallen into crisis. In situations where the young person is well connected to their community or ancestors, they have a support system and somewhere to go to heal and get back on the feet. I know from my own personal experience of working with young people experiencing homelessness that in the case of a young person who does not have this level of connection, they can easily fall through the gaps in society, become isolated and lack an attachment to life. The vast majority of the young people I worked with at the homelessness crisis accommodation were there due to a lack of family and community support. When things went wrong for them they had to cope with this alone and were not ready.
Marcus therefore urged the audience and our society to, “separate yourself from what separates you from others,” and to let go of the things that don’t allow us to connect to place, such as phones, the internet and televisions etc.