Activity 2018 Impact Assessment - Future Generations Scholarships

Summary

Between 2010 and 2018, the University of Melbourne awarded 129 Future Generations Scholarships to post-graduate students across all faculties. The Joining the Dots Pilot Project undertaken by the eScholarship Research Centre investigated the impact of these scholarships for recipients, and identified a methodology for capturing this in the eScholarship Research Centre's Online Heritage Resource Manager, as part of the emerging University of Melbourne Perpetual Calendar.

Details

As a methodological basis for the Impact Assessment, the project team drew on Dart and Davies (2003) Most Significant Change. Originally developed in commercial networks, this methodology is widely recognised and now used globally for program evaluation by non-government organisations, governments and multilateral organisations. As a method of impact assessment, Most Significant Change had already proven itself to be nimble, effective, grounded and scale-able. It had been previously utilised by Ani Wierenga (2008), one of the pilot project directors, as the basis for an internationally focused Australian Research Council project. Usefully, for the planning of the project team, that work has since caught the attention of United Nations Educational Scientific and Cultural Organization (2014) and has been used in its discussions about scaling community-change-based stories into larger quantitative evaluations.

In Most Significant Change there are two main substantive components which became central features of the pilot and the subsequent model of Impact Assessment:

1.Consultations: gathering narratives of impact; and
2. Workshops: analysis of narratives and synthesis of learning through workshops with key stakeholders.

The Most Significant Change process offers the following strengths: this is built on constructivist and participatory methodologies. Its strength is simplicity, rigor in replicability. Assessment can be done post-hoc with no indicators. The stories, as responses to four replicated questions, are coherent; pattern recognition is possible. This leads to the ability to make sense of diverse program experiences and outcomes 'apples and oranges' post facto. The Most Significant Change process of impact assessment is rigorous in that the steps can be repeated to assess the value of any activity in the life of an organisation or community. This replicability is a strength, in that it can be used to investigate the impact and effectiveness of one activity or a group of activities in series. Thus it holds possibilities for comparison or scaling, as previously tested in a co-authored work by Ani Wierenga (2013), into statistical analysis across topics and themes. The engagement of key stakeholders in a contained and light touch process of analysis means that story selection and synthesis of knowledge always happen in the context of unit or organisational strategic priorities. This selection of stories and synthesis of knowledge through the workshops reflects an active engagement with an organisation's key people, strategy and goals and can then be used strategically to inform future activity.

Impact Assessment Model: The Generic Output

This project piloted a model of impact assessment that could potentially be used anywhere across the life of the University, in conjunction with University of Melbourne Perpetual Calendar. The key outputs (products) from the pilot are:

1. Stories of Impact: The most salient stories of impact from individual participants, with each tagged to the themes that they address, as identified by the key stakeholder groups

2. The Impact Assessment Summary: a synthesis of value of an activity to the University community. This summary draws on the discrete voices of key identified stakeholder groups, from their rapid analysis of the data (stories), in the context of their (strategic) priorities.

For 'The Impact Assessment Summary' and 'Selected Stories of Impact' see published resources below.

Published Resources

Books

  • United Nations Educational Scientific and Cultural Organization, Global Citizenship Education: Preparing Learners for the Challenges of the 21st Century, UNESCO, Paris, 2014. Details

Edited Books

  • Wierenga, Ani and Guevara, Jose Roberto (eds), Education for Global Citizenship: A Youth-led Approach to Learning Through Partnership, Melbourne University Press, Melbourne, 2013. Details

Journal Articles

  • Dart, Jess and Davies, Rick, 'A Dialogical, Story-based Evaluation Tool: The Most Significant Change Technique', American Journal of Evaluation, vol. 24, no. 2, 2003, pp. 137-155. Details

Reports

  • Wierenga, A; Wyn, J; Guevara, JR; Gough A and Dyer, S, Youth-led Learning : Local Connections and Global Citizenship, Australian Research council Linkage Project LP0882159, University of Melbourne, RMIT University and Plan International Australia, 2008. Details

Online Resources

  • McCarthy, Gavan and Wierenda, Ani, 'Selected Stories of Impact', in The Joining the Dots and the University of Melbourne Perpetual Calendar 2018 Project, Pilot Project Report December 2018, 2018. Details
  • McCarthy, Gavan and Wierenda, Ani, 'Impact Assessment Summary', in The Joining the Dots and the University of Melbourne Perpetual Calendar 2018 Project, Pilot Project Report December 2018, 2018. Details

Created: 7 November 2018, Last modified: 4 January 2019