
doi: 10.14264/f28fac8
Social impact evaluation is an increasingly common practice among charitable organisations that provide social services. As both an empirical and managerial effort, these evaluations aim to inform organisations and their stakeholders about whether their services are effective, and what can be done to enhance their effectiveness. However, the process of evaluation and its outputs are far more complex than they may initially appear. Evaluations not only measure the reality of social services but are also influenced by the broader institutional environment, and in turn, they actively shape that environment. This thesis investigates the practice of evaluation in charitable not-for-profit organisations. Situated within the Australian context, it combines qualitative data from a large systematic review of evaluation reports, and interviews with evaluation stakeholders. Unlike previous meta-evaluations which problematise the evaluation findings or the evaluand, this study interrogates the practice of evaluation itself; specifically, the role that the practice of evaluation plays in reinforcing or disrupting social inequalities. The thesis begins by outlining existing scholarly thought on this topic and introducing the critical institutional perspective adopted to guide this line of inquiry. The first empirical paper, “The Inspection of Marketised Models: Audit, Evaluation, and Service Beneficiaries” seeks to understand how the chosen modality of inspection in different social services is influenced by the surrounding institutional environment. By combining public statements of inspection from charitable organisations with qualitative interviews with sector leaders and experts, we reveal a stark divide: audited services operating within quasi-markets, on the one hand and evaluated services relying on block grant-funding, on the other. These differences appear to be underpinned by embedded expectations of whether services should change outcomes or simply provide care, in turn influencing the design, administration, and quality of services. The second paper, “Imprinting and the Evolution of Evaluation: A Descriptive Account of Social Impact Evaluation Methodological Practice” investigates the evolution of evaluation approaches within the charitable sector, emphasising the proliferation of options throughout waves of evaluation diffusion. Drawing on imprinting theory, it describes the persistence of specific practices despite the emergence of alternatives. The study illuminates the enduring impact of theory-driven evaluation, introduced in the 1980s, on current practices. However, it also reveals how this approach has evolved over subsequent evaluation waves, acquiring divergent characteristics. The study highlights the potential consequences of this imprinting, including management-focused evaluations and mission drift. The third paper, titled “Digital-Era Evaluation: Digital-Era Evaluation: Automating and Reconfiguring Evaluation in the Social Service Sector” explores the role of digital technologies in evaluation. It presents both their potential to perpetuate managerial values and their capacity to disrupt power relations among evaluators, commissioners, and users. Drawing on discursive institutionalisation’s conceptualisation of change, discourse analysis of 25 qualitative interview transcripts was conducted. The findings indicate a significant shift towards digital-era evaluation through the automation of data collection, analysis, and reporting, shifting control from external evaluators to in-house functions. Despite these advancements, traditional evaluation elements remain entrenched, potentially limiting the future scope and utility of evaluation for social sector organisations. Finally, the fourth empirical paper titled “Evaluation as Advocacy: Legitimating Community and Sector Perspectives on the Function of the Charitable Ecosystem Through Research” explores how charities leverage evaluation outcomes to advocate for a redefined sector that aligns with their visions of social change. This paper presents a nuanced and complex relationship between these expectations and the ecosystem of the charitable sector. This research offers two key contributions. First, it reveals that charitable organisations are using evaluation as an advocacy tool, specifically by leveraging the legitimacy gained from the evaluation process and findings to speak against systemic barriers to fulfilling their social missions. Second, it provides support for the ecosystem paradigm of charity organisational theory, demonstrating how legitimacy functions interactively within the system in which charities operate. The central contribution of this thesis lies in the insights it provides about the influence of the institutional environment on both the practice and products of evaluation. This environment is deeply shaped by neoliberal values emphasising market practices and liberalism, leading evaluations to define and measure effectiveness in ways that align services and their beneficiaries with these values. While these traits may be considered desirable in certain contexts, they conflict with the core purpose of providing charity and are therefore contested by the charitable organisations involved in these evaluations. In the final concluding chapter, I reflect on the insights drawn about the institutions of charity and evaluation from the combined findings, as well as discussing limitations and potential directions for future research.
Measurement, Charitable Organisations, School of Social Science, 4410 Sociology, Social Impact, Evaluation, 441001 Applied sociology, program evaluation and social impact assessment, Social Services
Measurement, Charitable Organisations, School of Social Science, 4410 Sociology, Social Impact, Evaluation, 441001 Applied sociology, program evaluation and social impact assessment, Social Services
| selected citations These citations are derived from selected sources. This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically). | 0 | |
| popularity This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network. | Average | |
| influence This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically). | Average | |
| impulse This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network. | Average |
