Over 2017 we have been keeping a record of particularly interesting articles in the general areas of Blended, Online and Digital (BOLD) Education. More specifically, since March we have maintained a folder of articles published in open access journals that might qualify later in the year for our 2017 list of “top 10” widely accessible reads. Our list of top reads will be shared progressively via Twitter over the next three weeks as we countdown to our No. 1 article.
Interesting Questions
It follows that this exercise, which builds on last year’s experience, raises some interesting questions:
- What selection criteria do you adopt to help identify a really good (open access) journal article?
- What selection methodology do you use to help identify the top 10 open access journal articles for the year?
- Who do you involve in the selection process to help enhance the validity of the list of top 10 (open access) journal articles?
With these questions in mind the following comments are intended to help explain and frame this year’s selection.
Our Selection Criteria
When we first began this task we didn’t have any clearly defined selection criteria. This was partly in the interests of promoting inclusion and recognition of the fact that we all have different interests and perspectives. However, as we began the wider nomination process and refined the short-listed articles we recognised the need to more explicitly anchor our selection in some guiding criteria. Accordingly, the final selection of the top 10 open access journal articles for 2017 were loosely guided or informed by the following criteria:
• Published in open access journals listed on NIDL website
• Restricted to Higher Education (inclusive of teacher education)
• Strong preference to journal articles with international focus or relevance
• Minor preference to journal articles published by professional associations
• Strong preference to journal articles offering major literature reviews
• Strong preference to journal articles addressing major gaps in the literature
• Minor preference to journal articles exploring new and emerging topics
• Strong preference to journal articles which challenge conventional thinking
• Minor preference to journal articles relevant to current NIDL’s projects
• Overall selection of top 10 journal articles reflects a mix of gender, cultural and geographical diversity
Our Selection Methodology
The formal methodology initially involved a nomination process open to 20 members of the NIDL team. A related internal objective of the task was to raise awareness and encourage team members to more deeply engage with the published literature. While a shared folder for the collection of top journal articles was available from the beginning of March 2017, the wider open nomination process didn’t begin until the start of November.
In addition to this open nomination methodology and the collection of interesting journal articles in a shared folder over the course of the year, the NIDL Director systematically went through the full list of open access journals available on our website to help identify specific journal articles published in 2017 which might qualify for inclusion in the short-list. Notably, although extensive, this list of journals does not include all of the 270 publications identified last year by Perkins and Lowenthal (2016) in their comprehensive analysis of open access journals. From this more selective list taken from our website a draft collection of top articles for the year was created using a shared folder in Google Drive, which contained copies of the original articles.
Towards the end of November this list was refined to a draft list of 10 articles, which the aforementioned NIDL team members were then invited to rank in order of merit and wider readership value. During this time the draft top 10 list was relatively dynamic as the ranking process threw up additional articles and a handful of new journal issues were published during the month (e.g., AJET, IRRODL & EDUCAUSE Review). As a consequence, a handful of articles on the original list were replaced with late additions, which posed some challenges in the ranking process but we believe is evidence of the inclusive nature of our overall methodology.
Lessons from the Selection Process
Before we share (initially via Twitter) our final selection of top 10 open access journal articles for 2017 it is useful to reflect on several lessons arising from the process.
1. Blurring of boundaries
Firstly, there is an increasing blurring of boundaries between open and closed publications. During the selection process the question arose what constitutes an open access article?
One of our top 10 articles, for example, appears in a highly ranked closed journal published by Taylor & Francis which is managed by a professional association. The publisher now provides an open select service where the author(s) have the option of paying a fee to ensure downloads of their article are freely available. In this case we decided to include such publications for consideration in our list of top 10 as we wanted to recognise the authors commitment to openness and more widely disseminating their work.
The question of what constitutes an open access publication also arose with pre-print uploads of articles by authors to institutional repositories and websites such as Research Gate, especially when the published article also appears in a closed journal.
More specifically, this issue came up when we considered publications such as George Veletsianos’ article on who participates on MOOC hashtags and in what ways in trying to develop a generalizable understanding of Twitter and social media use. Although a pre-print version of the article is openly available from ResearchGate and George’s personal blog, it is formally published in the Journal of Computing in Higher Education which is a closed publication.
Similarly, on the theme of MOOCs, we had to consider how to handle a useful publication on designing Massive Open Online Courses to take account of participant motivations and expectations, which is available as a pre-print version on Gilly Salmon’s personal blog, when the final version of this article appears in the British Journal of Educational Technology.
In the end, after carefully reflecting on this issue, we decided to exclude such articles from our list; but the wider lesson is the way that some authors are strategically navigating and intentionally managing both open and closed spaces to help more widely disseminate their work.
2. Growth of review articles
Secondly, there appears to be a growing trend and increasing popularity towards the publication of review articles on topical issues following a systematic review methodology. For example, amongst the list of nominations we considered Krull and Duart’s (2017) article reporting a systematic review of research on mobile learning in higher education. Similarly, we also considered Liyanagunawardena and colleagues’ article reporting a systematic review of literature on open badges published in the European Journal of Open and Distance Learning. In addition, Mnkandla and Minnaar’s (2017) metasynthesis of the literature on the use of social media in e-learning was considered for inclusion given our preference for major review articles.
Also, with an interesting focus on how authors collaborate in written publications in the area of e-learning, we reviewed Mohammadi, Asadzandi and Malgard’s paper in the International Review of Research in Open and Distributed Learning analyzing articles in the Web of Science over the 10-year period.
While all of the above publications explore important topics and our final selection of top 10 articles for 2017 include a number of major literature reviews, the standout observation from evaluating this type of work is that not all review articles are created equally. Polanin, Maynard and Dell (2017) support this observation in their critical analysis of this line of research recently published in the Review of Educational Research where they report:
Despite their popularity, few guidelines exist and the state of the field in education is unclear (p. 172).
They go on to observe that such ‘overviews are a relatively nascent and undeveloped synthesis method that pose unique methodological challenges and may be problematic’ (Polanin, Maynard & Dell, 2017, p. 173). Building on this concern the literature is never neutral and the challenge is to critically interpret the body of published work and methodologies adopted based on explicit theoretical frameworks, which go beyond closet positivist methods simply describing what has been published. Not all of the review articles we include in our top 10 list fully address this point and we encourage readers to be critical of such publications.
3. Value of closed publications
Thirdly, despite the focus of this exercise being on open access publications, which is both philosophical and pragmatic as we want as many people as possible to read the articles we select, many of the so-called best articles (depending on your personal selection criteria) continue to feature in more traditional closed journals. Put another way, our list of top 10 reads for the year would be very different if we adopted an hybrid sample of both open and closed publications. This point begs the question, what might we have included or at least considered in our selection from a wider sample of more traditional closed or restricted journals?
This is a difficult question to answer without repeating a systematic selection methodology; however, for your interest we have listed below another 10 publications that we may have considered for this list, although it needs to be stressed that there are many other journal articles worthy of consideration and further evaluation depending on your specific interests:
- TechTrends 2010–2015: A content analysis
- The goals and methods of educational technology research over a quarter century (1989–2014)
- Systematic review of two decades (1995 to 2014) of research on synchronous online learning
- What works and why? Student perceptions of ‘useful’ digital technology in university teaching and learning
- Giving up technology and social media: why university lecturers stop using technology in teaching
- The innovation paradox: a selective review of the literature on action learning and innovation
- Beyond change blindness: embracing the technology revolution in higher education
- Change levers for unifying top-down and bottom up approaches to the adoption and diffusion of elearning in higher education
- Learning analytics: challenges and limitations
- Big data and data science: A critical review of issues for educational research
Final Comment
Our final list of top 10 open access articles for 2017 come from just five well known journals. Partly by design, professional associations, with one notable exception, manage the majority of the journals. This exception is the International Review of Research in Open and Distributed Learning which continues to be ranked as one of the top journals in the field. After we have announced the full list of top 10 publications we intend to post to our NIDL blog a brief explanation for our selection of each article and further remarks on the publications missing from this list. Importantly, in the end the final selection is not intended to be a definitive list of the “top 10 reads” for 2017, as this very much depends on the eyes and interests of the reader. What we hope is that our list is helpful in confirming your own selections, challenges you to reflect on what is missing and/or alerts you to something valuable that you may have missed in your professional reading over the year.
Digital Learning Dispatches 