Data privacy and the law

10 things that evaluators (and anyone in the not-for-profit sector) needs to know.

As an evaluator (or, frankly, anyone on the frontlines of the not-for-profit sector), our job is to know a thing or two about data privacy. After all, we’re increasingly required to collect impact data for donors and funders about how well our organisations, programs and individuals are performing.

Five messages for the new Australian aid performance framework

At Clear Horizon, we recognised and advocate the importance of the ‘L’ (Learning) in MEL – it is something we are passionate about, working with our clients (user-focused) to use measurement to inform practice. Dave Green and Damien Sweeney, Principal Consultants in Clear Horizon’s international team, contributed to a recent blog along with a number of their peers, identifying key messages to improve the performance framework of Australia’s development assistance.

Read the full blog here!

Let’s bridge the cultural divide: why evaluation and anthropology need each other

Research Analyst Alessandra Prunotto reflects on the Australian Anthropological Society conference, 2-5 December 2019.

This year’s Australian Anthropological Society conference in Canberra was themed “Values in anthropology, values of anthropology”. With this focus on values, you might assume that there would be a panel on evaluation, or at least a paper or two.

But there wasn’t.

How is this possible? I thought. After all, anthropology and evaluation are disciplines that overlap significantly.

Socio-cultural anthropology is the study of human societies in their environments. And evaluation is the practice of ascribing value to an initiative that’s creating change within a socio-environmental system. So at an anthropology conference on values, some debates on the thorny issues in evaluation practice seemed like par for the course.

But this was not the case. Instead, the undercurrent of concern that permeated the conference was the value that anthropology as a discipline has to those outside academia.

I suspect the conference theme of “value” manifested in this way because the majority of attendees were academics or PhD students. I believe that some of these intellectuals are struggling to understand the meaning of their work in a world where the pursuit of knowledge for its own sake is seen as retreating into an ivory tower.

As I listened to the various discussions circulating around “applied” and “public” anthropology, I noted down the themes emerging around the principles and skills that anthropology can offer the world beyond academia.

Looking around at the work we’re doing at Clear Horizon, I believe anthropology has much to offer evaluation.

In what follows, I expand on the ways that anthropological perspectives can help us navigate emerging trends in evaluation practice, as we see change-making initiatives move towards systems-change approaches and as cultural safety becomes essential for evaluation in First Nations contexts.

Anthropology’s superpowers

What are anthropology’s strengths, in comparison with other social sciences? At the conference, I noticed several themes emerging that painted a picture of applied anthropology done well.

Understanding of the other. As the comparative study of human societies, anthropologists intend to understand other people’s perspectives, no matter how radically different. This requires you to listen deeply and attentively, with humility. You aim to learn and empathise before reflecting critically.

Challenging assumptions. With an awareness that your way of being and thinking is one of many, anthropology compels you to question your own assumptions, as well as that of others. Anthropology is a critical discipline that aims to surface the most fundamental aspects of different worldviews and value systems.

Bringing together different worlds. As the goal of anthropology is to bridge cultural divides, it gives you the foundation to bring together different stakeholders, creating a space for different voices and perspectives to come together. It helps you to find zones where mutual understanding can flourish.

Reflexivity and awareness of power dynamics. Sharpening your observation skills and handing you a toolkit of theorists from Fanon to Foucault, anthropology enables you to identify power dynamics and maintain an awareness of your own position within them.

Contextualisation and comfort with complexity. Anthropology aims to understand human behaviour and beliefs within a broader social, historical, environmental and political context. It asks you to embrace messiness and complexity, and avoid reductive explanations.

Anthropology in systems-change evaluation

Across the change-making sector, we’re seeing a move away from programs and towards systems-change initiatives, such as place-based and collective impact approaches.

A number of elements that characterise these approaches heighten the complexity required of evaluation practice. These elements include the increased number and diversity of stakeholders involved, long timeframes, the range of changes initiated and the difficulties in identifying how initiatives contribute to certain outcomes.

As we can see from the list above, anthropological training leaves you well-placed to tackle these kinds of evaluations.

It allows you to bring together diverse stakeholders and facilitate the negotiations between different value systems and worldviews. It helps you to understand and manage the power relations that might exist between these different groups. And sitting with the complexity of the work is a sign that you’re dealing with the reality of change, not boxing it down into something that is both manageable and fictional.

Anthropology in First Nations evaluation

Attitudes in relation to First Nations evaluation are changing. At Clear Horizon, we’re hearing more compelling demands for decolonising evaluation and power-sharing. A merely participatory and inclusive approach to evaluating with First Nations peoples is no longer acceptable – instead, evaluation needs to be conducted by First Nations peoples “as” First Nations peoples, a principle that is underpinned by the rights of Indigenous peoples in the UNDRIP.

Anthropological perspectives have much to offer to help non-Indigenous evaluators collaborating with First Nations peoples navigate these issues sensitively and appropriately.

Most importantly, anthropology can provide a means to understand that hegemonic practices in evaluation and change-making are not the only ways of working, nor are they necessarily the right ones. Furthermore, anthropological perspectives can foreground how histories of colonisation can continue to manifest in power relations and structural disadvantage today. These aspects of anthropology can lend the humility needed to truly shift evaluative practice back to First Nations peoples and communities who have the expertise to evaluate in their own context.

Bridging the divide

It’s clear that anthropology can bring important skills and perspectives to the emerging trends in evaluation practice.

But it seems that evaluation is not on the radar as a potential career for anthropology students. As I’ve highlighted, it was not even mentioned at the most recent annual conference, despite the focus on “value” and a more public anthropology. And from my own experience, having graduated from an anthropology degree just last year, I’d never heard evaluation mentioned as a potential career path.

What can we do to bring evaluation and anthropology together?

One option might be a partnership, or at least more dialogue, between the Australian Evaluation Society and anthropology departments. Local chapters of the AES might have the potential to run careers information talks, mentorships, or even graduate placement programs.

Perhaps advertisements for evaluation positions could specifically mention that anthropology graduates are welcome in the list of disciplinary backgrounds accepted.

But the skills and ideas of anthropology should not just be limited to those who studied it at university. There could also be an opportunity for anthropologists to run training for evaluators who come from disciplines more removed from anthropology, to upskill them for these emerging areas of work.

Evaluation and anthropology have the potential to achieve more impact together. Evaluation can benefit from the nuanced, critical perspectives anthropology can bring. And anthropologists are seeking ways to make themselves useful outside universities.

Let’s reach out.

Resilient Sydney

Clear Horizon’s Sustainable Futures team are working with the Resilient Sydney Office to develop an M&E Framework for the five-year Resilient Sydney Strategy.

The Strategy aims to strengthen Sydney’s capacity to prepare for, respond to and recover from disaster, whilst ensuring all of Sydney’s communities can access opportunities to thrive. The Strategy aims to effect change across the systems of the city to achieve these objectives, and is being delivered through collaborative initiatives, underpinned by a collective impact model for systemic change.

With the Strategy’s focus on systemic change, collaboration and collective impact, the Sustainable Futures team have been developing an M&E Framework informed in part by the Place-based Evaluation Framework (Dart, 2018)  This will ensure the Strategy’s M&E will work with the phased nature of systems change, and across the different scales of change.  In addition, to align with the collective impact model used, the Framework distinguishes between the work and outcomes of the backbone organisation (i.e. the Resilient Sydney Office) and those of the broader partnership.

Working with the Resilient Sydney Office on this M&E Framework has been a really exciting opportunity for our team for a number of reasons.  The first is the clear alignment in the passion and vision of our team for driving real and positive change.  The second is that the complexity that the Strategy is dealing with demands that we continue to innovate, test and refine our M&E approaches, to ensure they remain useful and fit-for-purpose, and can meaningfully engage with the complexity of evaluating influence on systems change.  We are thoroughly enjoying the challenges this project has thrown at us and are excited to see where it goes next!

Reflections on transferring to the international evaluation space

Kaisha Crupi is a consultant at Clear Horizon and has recently made the move from the domestic Sustainable Futures team into Clear Horizon International. Below is a reflection piece from Kaisha, discussing her learnings and observations in her new role.

Before joining Clear Horizon 18 months ago, I had only a small taste of the international working world. However, since I made the full transition into ‘Clear Horizon International’, affectionately known as CHI (pronounced chai), I feel as if I have jumped into the deep end with each team member supporting me to learn how to swim, and to swim fast. I also feel that the rest of the organisation is on the sideline cheering me on. Below are my reflections and learnings from the last few months of being part of the CHI team.

Working in the international space is tricky

When I was in the domestic team I was exposed to many different industries, types of work and ways of working and met so many people who are passionate about their work through workshops, interviews and product development. Now starting to work in the international space, I am learning about things outside my home country, and in countries that I am working in (and not necessarily living in). Trying to understand different cultural customs, to work around language barriers, across different time zones and to understand different political systems and social contexts is proving to be quite tricky. I am learning a lot more, am asking a lot of questions and reading wider than my easily accessible news. I am also being kind to myself – I know that I am not expected to know everything and am not putting pressure on myself to do so, especially in a short amount of time!

The work is the same, but different

When I first joined the CHI team, I thought this would be great to learn a whole new skill set and challenge myself even further by learning something different and working in a different way. To my surprise, my first task when I joined the team was to conduct a document review against the monitoring and key evaluation questions for an upcoming workshop, which is something that I had finished doing for a domestic project a week earlier to feed into a report! The questions were similar and the way to go about it was the same. The only thing (which was quite a big thing, mind) was that the language and jargon was different, and instead of talking about an area or region, the project was focusing on a whole country! My biggest challenge in joining the team so far is getting used to all the acronyms in reports and discussions with my peers and our clients. I am slowly getting there; though someone should quiz me in the early stages of next year.

Understanding complex challenges

By going to a destination for an international holiday versus going for work, you learn about a country in a very different way. There is the saying that you should not discuss politics when you are in polite company – this is very different in the international working space, particularly when working in design, monitoring and evaluation. You learn about a country’s context on a more granular level, ask the difficult political questions and try to understand the country as much as you can, as fast as you can, especially whilst in-country. I have been exposed to the complex ways of working, what the client deals with and small spot fires they must put out on a day-to-day basis (which are quite different than in the domestic space). These issues also do not have quick-fix solutions. There is at time a feeling of helplessness – now that you know about this information, what are you going to do with it? I believe that doing design, monitoring and evaluation work helps with this, as knowledge is power and can be a communication tool to change someone’s life for the better.

I feel very fortunate to have landed where I am today. Not many people can say that they have ended up with their dream job, especially in such a great values-driven organisation in a very niche area. I have a great team of people supporting me and putting up with my insurmountable amount of questions and reflections, whilst also looking back fondly at my time in the domestic team, where I was able to build my strong foundational knowledge and be supported in every direction. I am looking forward to continuing to swim out toward the horizon and reflecting on the successes and challenges that are happening in such a complex world.

2019 Changefest: Evaluation Tools session

Evaluation Tools session, with Monique Perusco (Jesuit Social Services and Working Together in Willmot), Skye Trudgett and Ellise Barkley (Clear Horizon)

Our session started with a contextual summary of the work and characteristics of ‘Together in Willmot’, a collaborative social change effort in Mt Druitt involving The Hive, Jesuit Social Services, service providers, schools and many other partners. Clear Horizon is working with Together in Willmot as an evaluation partner. Our shared approach to learning and evaluation responds to the challenges of evaluating systems change and place-based approaches, and is tailored to the phase, pace and strengths of the collaboration. We introduced the process for evaluation we are undertaking, which has involved training in Most Significant Change Technique and local data collection which will feed into building a theory of change and then an evaluation plan. We are planning next year to do co-evaluation focused on the efforts and outcomes to date.

During the session we looked at examples of some Most Significant Change stories so far collected as part of this work.

Most Significant Change (MSC) technique was developed by Jess Dart and Rick Davies. Together Jess (Clear Horizon’s founder and CEO) and Rick authored the User Guide in 2005, and MSC is now applied to innumerable contexts worldwide. MSC story based method that can be used for participatory monitoring and evaluation. The process follows a simple interview structure that can generate a one page change story. It is participatory because many stakeholders are involved both in deciding the sorts of change to be recorded, and in analysing the data. MSC uses stories as a form of data collection. Stories are collected from those most directly involved, such as project participants and field staff. Stories are usually collected by asking a simple question such as ‘during the past year, what in your opinion, has the been the most significant change for participants as a result of this program? Stories are then collected, and stakeholders sit together to analyse the stories, at this time, participants are asked to select the story that represents the most significant change for them. The process of selecting the most significant story allows for dialogue from project stakeholders about what is most important. This dialogue is then used as evaluation data to create knowledge about the project and what it is achieving. 

We also covered concepts and tools for evaluating systems change and place-based approaches from the Place-based Evaluation Framework  and Place-based Evaluation Toolkit, which was commissioned by Commonwealth and Queensland governments last year and is a leading guide for evaluation in this context. We introduced the generic theory of change for place-based approaches and ‘the concept cube’ that shows the multiple dimensions of evaluation in this context. Clear Horizon worked with TACSI and CSIA to lead the co-design of the framework and have been working with government, community, philanthropy and non-government partners to test, apply and progress these learning, measurement and evaluation approaches.

AES Conference 2019 – for a first-time attendee & emerging Evaluator

By now I’ve had a few weeks to reflect on my first Australian Evaluation Society conference, where I was exposed to an amazing variety of evolving and inspiring ideas from within the Australasian evaluation community.

On the first day I found myself inspired by Dr Tracey Westerman, who raised the dubiousness of using or adapting Western-focused data collection tools in a First Nations context. For me, this highlighted the necessity of remaining reflective and adaptive in our approach, as well as tailoring every evaluation’s approach and methods to the context, preferably in partnership with community. This in turn made me reflect on the work of one of my Clear Horizon colleagues, Skye Trudgett, a proud Aboriginal woman who is passionately pursuing Indigenous Data Sovereignty and is leading a program to build First Nations young women’s evaluation capacity in remote Australia.

Following Dr Tracey Westerman’s plenary, I attended a Systems Evaluation Theory Application presentation by Lewis Atkinson, Brian Keogh and Ralph Renger, which helped frame my thinking regarding complexity. I found myself identifying with the concept of cascading success or failure as a neat way to consider the upstream challenges which produce a downstream issue. I could see the concept’s similarity to framing challenges through approaches such as the Problem Tree and found it a resonating concept in which to couch my thinking on casual pathways.

My third and final reflection was on the space for emerging evaluators. The conference provided a valuable sounding board for ideas and challenges facing those new to the profession and was highlighted on the final day by Eunice Sotelo and Francesca Demetriou, who are conducting research on these experiences. I found myself identifying closely with the key findings, and introspectively, the session highlighted a need to establish a community of support and practice for emerging evaluators. Personally, I will be seeking to develop an informal network, but I believe that an AES-associated group would be invaluable in attracting, retaining and developing those new to the space. Young people are not only the future (inevitably), but have the potential to bring new ideas, perspectives and approaches to evaluation. I feel that this potential influx of creativity and vigour should be encouraged and enhanced through more formal arrangements.

As Kailash Satyarthi is credited as saying, ‘No segment in the society can match the power, idealism, enthusiasm and courage of the young people’, or in this context, young (not necessarily age-specific) evaluators.

Big promises and digital staff

Reflections on the first Digital Transformation Summit by Jen Riley, CIO at Clear Horizon

I arrived 2 hours late to the Digital Transformation Summit in Canberra because something had gone wrong with Microsoft which caused havoc with aviation booking systems. The incident was a salient reminder of how dependent we are on technology. The embracing of digital transformation is well underway in our Federal and State Governments according to all the t-shirts, slogans and senior government bureaucrats that used the inaugural summit to demonstrate how well government is embracing the 4th industrial revolution.

The message was hopeful. We will digitalise. We will automate.  We will be client-centred and provide a more personalised intuitive experience for citizens. Citizens will be able to self-serve and have the same seamless experience we have learnt to expect from the business sector, and if you live in Queensland, this will be delivered by the first digital staff member ‘Russell’. Government representatives were quick to add though that this did not mean everything would be delivered online rather restricted to that which can be automated and available through chat bots and the like will be made available to those with the capability and capacity to access digital options.  Those with more complex cases or people without the capability or capacity to access information/services through the online medium would be able to access a person this multimodal or omnichannel approach has proven successful for the banks and Telcos.

What about those experiencing vulnerability? Will they be left behind? Renée Leon, Secretary of the Department of Human Services, cited the example of remote Aboriginal communities being one of the biggest users of Department of Human Services online services who would normally have to drive 100s of kilometres to process Medicare claims.

But the single biggest question from the (mostly government) audience was in relation to how to deal with resistance from staff/management experiencing digital transformation, indicating that the change management within government is not smooth sailing. Ron Gauci, CEO of the Australian Information Industry Association challenged the term resistance and argued what most people found was more likened to intimidation and reluctance – who wants to be shown up as a luddite? Half of the workforce is digitally savvy, they had the internet at school, learnt about programming, the other half (i.e. everyone over 40, myself included) did not grow up in a tech environment, we all remember posting letters, using index cards (they had a use before Agile) in libraries and handwriting essays. The massive upskilling required was a key theme throughout the day and the Digital Transformation Agency capability building projects are to be applauded. But as Renee Leon pointed out, citizen expectations of technology and the speed of technological change is far surpassing the capability of non-technical people in government.

Dr Julia Powles, Associate Professor of Law and Technology, University of Western Australia, warned of the ‘shiny object syndrome’ and ‘deferential’ treatment of technologies such as Artificial Intelligence (AI), Machine Learning (ML) and data analytics. She also cautioned the audience to be weary of vendors over selling and under delivering. There are a lot of white horses and unicorns running around offering all sorts of magical solutions that will solve all our needs and automate everything – when has a digital solution resulted in less work, someone, somewhere, is doing a lot tagging and scraping. Powles also reminded us of the diversity of human experience and the real danger of that being flattened with pattern recognition and aggregated user mapping experiences.

The session on AI and ML was a little underwhelming but showcased the variety of AI applications such as predictive modelling on who wasn’t going to pay their land tax, Olli the South Australian Autonomous Car and the application of Machine Learning in the Human Rights Commission record processing. Olli was fun looked a bit like a large toaster and I loved that it was addressing the first mile/last mile issue of public transport and supporting the elderly and disabled to get out of the home more. It still is really early days for the AI, ML world and there were lots of questions about ethics, who makes the decisions about the algorithms, how will these opaque decision-making processes be accountable, with few answers coming at the moment.

An interesting panel was on data sharing – an incredibly vexed issue for government. Government is committed to the privacy of individual data and grappling with the issue of informed consent and they want to share and consolidate data to make better policy decisions for the common good. Currently, all data requests between and within departments are considered on a case by case basis.  Philip Gould, Assistant Secretary, Office of the National Data Commissioner expressed that releasing data is not a check box activity but rather, a subjective assessment about the implications of sharing and joining data. Data has immense value, but it’s a high stakes game – it’s not just the ‘insurance company’ worry, it’s items such as the location of endangered species.

What I did love about yesterday was walking the walls of ‘life events’, where GovX have mapped citizens experience with government services across the life events of ‘birth of a child’ to ‘finding a job’ and dealing with the death of a loved one. The voice of community has been amplified on real pain points:

–       Lack of proactive engagement.

–       Inconsistent and inaccessible content.

–       No single source of truth across government.

–       Not meeting service delivery expectations.

This is exciting. Digital systems are fallible, as anyone on a plane experienced yesterday. However, an obsession with people’s genuine challenges with services and elevating their experience as central to addressing those challenges, is the true transformation.

AES International Evaluation conference Day 2!

It’s been an amazing AES conference so far – lots of interesting topics and great conversations. Like Jess, the highlight for me so far has been the key note speaker from day one, Dr Tracey Westerman – an Aboriginal woman from the Pilbara in WA. She has been a trail blazer in Aboriginal mental health. The key take away message for me was that measurement matters – but even more importantly, the right measures matter. She described that in many cases of Aboriginal youth suicide, there was been no prior mental health assessment. But when assessment tools are used, they are western based and not culturally appropriate. This can lead to misdiagnosis. For over 20 years, Tracey has argued that it is not appropriate to ‘modify’ existing measures because of their inherent racism. The only way is to develop new tools from the ground up. Tracey has developed seven tools specifically for Aboriginal youth mental health with not a lot of funding – no easy feat. It was a truly inspiring presentation from an amazingly passionate and optimistic woman who really cares about her people.

A highlights from day 2 was a panel of designers and evaluators from Australia and New Zealand: Jess Dart, Kate McKegg, Adian Field, Jenny Riley and Jacqueline (Jax) Wechsler, who explored how we might move out of the traditional box of program evaluation, to make a bigger difference. They discussed the role of evaluators in supporting people to move beyond measuring to think though whether we are doing the right things and whether we are really making a difference across complex systems. Questions were covered such as where can evaluators add value in a co-design process, does evaluation get in the way and slow things down, do evaluators need new skills to help analyse and make sense of big data? Jenny reminded us that evaluators are learners and we are curious, and that we need to get on board with the digital revolution.

One really interesting concurrent session I attended was on the use of Rubrics by Julian King, Kate McKegg, Judy Oakden and Adrian Field. They presented the basics of rubric and then described how rubrics can be a tool for democratising evaluative reasoning, stakeholder engagement and communicating of results. They presented two very different examples– one in a developmental evaluation and the other was using rubrics to evaluate the value for money of an agricultural funding program. I found the second example particularly interesting having experienced the challenges of answering the value for money question. Using a rubric in this way is great for balancing the multiple dimensions of value from different perspectives.

Another memorable moment was at an ignite session (which is a really short presentation). Damien Sweeny and Dave Green from Clear Horizon did a great job at presenting a rather convincing argument for placing more emphasis on monitoring over evaluation – Big M vs small e as they call it. And they cheekily suggested changing the name of the AES to AMeS. An interesting thought.

The day finished with a second keynote speaker, Gary VanLandingham, from the Askey School of Public Administration and Policy. He reminded us of the vast amount of evaluative information available through ‘What Works’ warehouses. They are a good place to start when starting an evaluation, but there are warnings. The main caution for me is that they privilege certain types of data over others, and they don’t include what doesn’t work or things not measured using experimental approach (such as randomised control trials, and quasi-experimental methods).

The day was topped off by a short ferry ride to Luna Park where we had a lovely conference dinner overlooking the opera house. Sydney is a very beautiful city and a great setting for a wonderful conference.

Now for day three….

Have you visited our booth at the conference?

AES International Evaluation Conference Day 1!

Dr Tracey Westerman, proud Aboriginal woman, had me totally gripped throughout her keynote presentation on Day 1 at the AES International Evaluation Conference. She began with the statistic that Australia has the highest rate of child suicide in the word. But she cautioned us to be optimistic and focus also on the positive outcomes that are occurring, such as the six young Aboriginal people graduating from medicine in WA this year alone. She stressed that education is the most powerful solution in the world and described how in one generation her family ‘closed the gap’ – she gained a doctorate despite living very remotely, with a family background of very limited formal education.

She walked us through the importance of developing assessment tools that are culturally sensitive and to avoid confusing causes with risk factors. It seems many tools in existence today are culture-blind and can lead to stereotyping and actual discrimination. She has developed a whole range of specific assessment tools that are culturally sensitive, including an assessment tool for work-based culture. She made the case that there hasn’t been the right sort of research with Aboriginal people, that the causes are different, and need to be assessed in culturally sensitive ways.

She’s working on ‘building an army’ of Indigenous psychologists across the country to address child suicide and influence child protection. She ended with the note that there is nothing that can’t be achieved by Aboriginal people if they believe in themselves.

After this I had the privilege of moderating a panel about the client-consultant relationship, a topic dear to my heart and my business! The panel was from the Victorian Department of Education and training (DEET), as well as consultants from Melbourne Uni and Deloitte Access Economics. DEET have set up a ‘state of the art’ supplier panel with over 30 suppliers on it, and are working to deepen the partnership between evaluation suppliers and commissioners, as well as embedding a culture of evaluation. They were generous in sharing their challenges, including lots of tricky moments around data sharing and IP.

Just before lunch I had the pleasure of indulging in a session of evaluation theory led by Brad Astbury and Andrew Hawkins from (ARTD). The explored the seven great thinkers of evaluation, who laid the foundations of our seven-decade-long journey of building our theoretical foundations. So lovely to wallow in theory, I remember savouring that learning when studying for my PhD. Their conversation was framed around Foundations of Program Evaluation by ShadishCook and Leviton (1991) – it was my evaluation textbook back then and good to see it’s still valued!

It was a busy day for me, and I also convened a fun panel on digital disruption. We had a great spread of panellists, with a designer from Paper Giant, Ruben Stanton; a data scientist, Kristi Mansfield; a social procurement expert, Chris Newman; as well as our very own Chief Innovation Officer, Jenny Riley. We explored the trends, the opportunities and the scary stuff that might come with the fourth industrial revolution – yes, the robots are here! I saw a few people jump in their seats when Jen described how Survey Monkey stores data overseas and is not bound by the same data security regulations. We also looked into the amazing opportunities for evaluators to be the sensemakers of Big Data. When the firehose of data hits you in the face, maybe the evaluators will be there to bring you calmly back to the most important questions. We also explored the opportunities for evaluators to get involved in evaluating new technology and innovation, and to help consider how ready the system is to receive these innovations. I didn’t want it to end!

The day was topped off with a keynote from David Fetterman on empowerment evaluation. Empowerment evaluation is now 26 years old! David explained how empowerment evaluation is a self-evaluation approach designed to help people help themselves. It felt familiar!