By Mark Brown, James Brunton & Eamon Costello
The evaluation of the impact of teaching and learning initiatives is still a relatively immature field of study. The question is how do we assess whether a specific initiative was successful in terms of both direct and indirect benefits on a sustainable basis and ultimately represents a good return on investment (ROI). 
With this question in mind the Seven Impact Dimensions (7ID) Evaluation Framework described below, and presented in the table as a template for conducting regular evaluations, was adapted from two sources. Each source was originally proposed to help more thoroughly evaluate the impact of teaching and learning-related research and development projects. The 7ID Evaluation Framework arose from our experience during an externally funded project to develop a Student Success Toolbox for Flexible Learners (Brunton, Brown, Costello, Delaney & Fox, 2016) and more specifically growing concerns regarding the depth and validity of the evaluation process, which involved twice yearly brief “snapshot” presentations to an international expert panel.
There are seven dimensions to the 7ID Evaluation Framework that are assessed on a five-point scale for both current and projected impact. The following quote explains the first five dimensions:
“Impact can be understood from a personal perspective, in the way that each individual academic [or institution in partner projects] thinks about their teaching, their career, their place in the institution and their relationships with their colleagues and their students. It can be thought about from a teaching perspective more broadly, as a change in thinking about teaching or teaching practices. It can be thought about from a student perspective – as a change in learning or from a more superficial angle, as student satisfaction. It can be thought about from a management [institutional] perspective – are staff in a school, department or unit changing in ways that meet needs perceived to be important (defined in various ways), it can be thought about from a strategic perspective – does the program further institutional objectives in some identifiable way.” (Jones, Lygo-Baker, Markless, Rienties & Di Napoli, 2016, p.10).
The sixth dimension to the framework recognises that teaching and learning related projects can contribute to outreach activities, such as publications and presentations, in the tradition of the Scholarship of Teaching and Learning (SoTL). Other examples of outreach include recognition of the initiative through external awards and coverage in media stories and professional magazines.
The final dimension recognises that a teaching and learning project can have important internal benefits for the people involved in designing, implementing and leading the initiative. Examples include individual team members being promoted partly due to their efforts associated with the project or other people deciding to complete masters or doctoral degrees in the specific area. These two additional dimensions are taken from the framework for evaluating teaching development projects first developed by Ako Aotearoa (National Centre for Tertiary Teaching Excellence) in New Zealand and adopted by the Australian Office for Learning and Teaching (Coolbear & Hinton, 2013).
An important principle of the 7ID Evaluation Framework is that impact needs to be deliberately planned for before a teaching and learning project gets underway, monitored at key milestones during the life of the initiative, and then assessed at least one or two-years after completion of the work to truly understand the extent of impact and sustainability of the innovation. Another feature of the 7ID Evaluation Framework is that unlike most other efforts at evaluating impact the five points on the assessment scale recognises that impact can be both positive and negative or have unforeseen consequences. We aim to pilot this framework in some of our NIDL projects to better understand the impact of our work.
References
Brunton, J., Brown, M., Costello, E., Delaney, L., & Fox, S. (2016). Creating a toolbox of OERs to facilitate flexible learner transition into higher education: A collaborative project between four Irish higher education institutions. Paper at Open Education Global Conference, Krakow, 12th April.
Coolbear, P., & Hinton, T. (2013). Evaluating the impact of research projects in tertiary learning and teaching: Exploring the geography of change. Plenary address HERDSA Conference, Auckland, New Zealand, 2nd July.
Jones, A., Lygo-Baker, S., Markless, S., Rienties, B., & Di Napoli, R. (2016): Conceptualizing impact in academic development: finding a way through. Higher Education Research & Development, DOI: 10.1080/07294360.2016.1176997
Digital Learning Dispatches 