Next week sees the launch of new research into hybrid working, a new report commissioned by the National Digital Leadership Network in Ireland.
Global Trends Towards Flexible, Hybrid Working and its Impact for Digital Leadership in Higher Education
This strategic report will focus on the global trend towards flexible, hybrid working and its impact for Digital Leadership in Higher Education in Ireland and beyond. We examine the wider implications of hybrid learning and working for Higher Education and situate this current globalised trend in its historical context. We also explore best practice exam-ples drawn from industry and education sectors and highlight the role of effective Digital Leadership in meeting the challenges of changing needs of (lifelong) learner populations in a world impacted by environmental and technological factors and draw together strate-gic recommendations for leaders in Higher Education seeking to support student success from Higher Education into a fast moving, competitive job market that is under threat from automation and AI .
Hosted by the Department of Technology Enhanced Learning (TEL) at Munster Technological University (MTU), this event will showcase the NDLN’s mission to advance digital leadership in higher education across Ireland.
I am really looking forward to speaking an next week’s World of Learning Summit. In preparation for the session and to offer an insight into what my session will be about, I contributed an article to the Learning Magazine: you can read the article freely online.
Screenshot of the article with the caption: Are you returning to the office, five days a week?
My session at the summit is on Wednesday, and focused on offering an interactive discussion forum for leaders. We’ll explore hybrid working recruitment and retention, new trends in hybrid working and also challenges that arise over time from recruitment to long term retention.
Look out for the slides of the session, which I’ll be sharing via my linktree.
Don’t miss a post. Subscribe to my free newsletter and get a summary of everything once a month to your inbox.
Today is a big day! Together with colleagues from all across the UK, I am heading to London. It’s the Week of VocTech, and the annual showcase of everything that’s happening across the sector.
I feel fortunate to be continuing my work as Strategic Lead for the partnership between ALT and the Ufi VocTech Trust even though I stepped down as ALT’s CEO earlier this year. There’re a couple of reasons why: from a personal perspective, I love working in Vocational and Further Education because that’s where I took my first steps from alienated high school drop out to finding my way into further learning. Next, I love being part of a partnership where so much actively happens (and we have achieved much in the last few years). The biggest reason however is AmplifyFE. Yes, it’s a project, yes, it’s working in partnership, yes, there are awesome people involved, and it is also so much more than that: it’s a grassroots community of practice that is connecting the dots between individual practitioners, innovators and industry for the benefit of the learners furthest from opportunity. Since it’s launch in October 2020 (remember that time?) our community has grown to connect 2,700 practitioners.
Over the past twenty years I have worked with lots of projects that tried to achieve what we have done, and my bookmarks bar is a graveyard of moribund communities of practice in which tumbleweeds flourish. So what is the secret sauce of our success?
As we enter our fourth year, here are my reflections:
We connect, we don’t compete
When we first started setting up AmplifyFE, we were very clear that we were building a network of networks, not a new network. This distinction is key to our success: we don’t aim to replace or take over existing networks, we act as a meta network, an overarching connection that links one specialist group with another, one sector event with practitioners who don’t know where to start. We don’t wait for you to come to us. We try to find and come to you.
Part of our mission is to help those who are isolated within their institution or specialist subject, those EdTech or IT teams of one, to connect with people who speak their language and share their concerns.
We put everyone on the map
All of our artwork is created with Bryan Mathers via a Visual Thinkery process. Our community map was one of the first things we developed, and it was born out of a desire to show the communities and conversations that we already knew about and to allow others to (literally) put themselves on the map.
That’s a very powerful message to send. We say: put yourself on the map, and we will help others find you, join in and contribute. We want to map where the conversations about practice are really happening, no matter how informal or niche your interest. In a sector in which everyone is under pressure, short of funding and faced with mounting challenges, let’s connect you with the people who are in the same boat. We are not gatekeepers, we are map makers.
We reflect our community in our research and case studies
We also take a deeper diver into how learning technology and digital tools can be designed and deployed to support vocational learners most impacted by the ‘digital divide’. We explore what key advice can we give to those beginning or continuing an innovation journey in VocTech.
Our Insights Report shares fresh research and case studies that show what makes for truly successful co-creation between vocational educators and digital design project teams. We unpack how we can create effective learning solutions working in partnership with vocational learners to help narrow the ‘digital divide’ for those most in need of workplace skills development. Basically, we find practical examples that show what is the difference between what really works and what doesn’t deliver.
We learn from others
One of the phrases we think a lot about is ‘by the sector, for the sector’. I am always curious about what people mean when they say it. How much do you know about ‘the sector’ to begin with? How do you take part? How do you contribute? And what are you asking in return?
We work with a community that suffers greatly from the ever-changing policy and funding changes starting at top-level government policy, to sector bodies and then filtering down to the local level and to individual institutions. Excellent programmes are launched, and then disappear again 3 years later. Or funding is cut for frameworks and skills training that had been invested in for years.
From the outset, our work with AmplifyFE was designed to learn from that. We wanted to find the data that shows how much the constant changes impact on professional development and innovation. Since 2020 we have collected and analysed this data annually, and published the results in our sector audit.
It’s a fantastic piece of work that really shines a spotlight on the scale of change that Further and Vocational Education has to weather. We learn from that, and we aim to change that.
We use our funding to stay free and open
Thanks to the funding from the Ufi VocTech Trust and the partnership work between the trust and ALT, we benefit from three decades of expertise in how to run an independent community of practice and we can stay free and open to all practitioners to join.
And that brings me to the biggest lesson I learnt over the past three years:
We are here to stay
Building trust is key to our success. Our team (the fabulous Emma, Lynne, Chloe as well as Lou and also others) includes only people who really understand the sectors we serve, who have worked in Further and Vocational Education and often they still do. We are transparent about our aims and also what we won’t do (i.e. commercialise our community) .
Active communities of practice that grow are gold dust. We know from our sector audit how many communities disappear each year, either because no one is active or because of funding or policy changes. Or sometimes simply because a certain social network is taken over by madness. Remember the sector forums that suddenly closed? Or the demonstrator programme that stopped? Or the skills training that was cut? Well, we do, too. And that is why we are here to stay, actively supporting our community and amplifying what is happening across sectors as an independent network of network.
This is the second post about my current work on researching alternative ways of measuring impact in Learning Technology. Go back to the first post in which I have set out the context of my work and what I am particularly focused on.
Alongside the practical work with the ALT Journal Strategic Working Group, I am pleased that my proposal of a short session ‘The quality of metrics matters: how we measure the impact of research in Learning Technology’ has been accepted for ILTA’s Annual Conference in Carlow, Ireland later this month.
In the meantime, I have been doing more reading and research into innovative ways of measuring impact and this time my work has come up against some very practical questions, not least because as a UK-based publisher we are in the process of ensuring the the journal’s operations comply with the incoming GDPR legislation. Open Source journal systems are not at the forefront of compliance and like other independent publishers we work as part of the community to move towards compliance.
At first glance factors like GDPR may not seem to be closely related to how impact is measured, but my thinking links them closely as a lot of the opportunities around developing the journal are dependent on technical solutions that have data processing implications:
A convincing alternative
Discussing how important having an impact factor is quickly runs into the question of what the alternative looks like. As well as the technical challenges in implementing innovative tools or mechanism for measuring impact (to which the new GDPR legislation adds another level of complexity), the sustainability and longevity of both tool and data storage need to be examined. For example, introducing a tool like Altmetrics requires us to educate all stakeholders and ensure that the level of digital literacy required is not a barrier to making the tool useful. The user interface and experience needs to be robust and practical, building confidence in alternative or innovative ways of measuring impact. With new tools and platforms being created all the time there is a certain amount of churn and in order to really build a convincing alternative there needs to be a certain level of consistency.
Scrutiny of new vs. established ways of measuring impact
The kind of scrutiny with which we are examining alternative ways of measuring impact isn’t easily applied to the established method. There is a critical discourse, for example this recent blog post on the LSE impact blog, which argues:
Many research evaluation systems continue to take a narrow view of excellence, judging the value of work based on the journal in which it is published. Recent research by Diego Chavarro, Ismael Ràfols and colleagues shows how such systems underestimate and prove detrimental to the production of research relevant to important social, economic, and environmental issues. These systems also reflect the biases of journal citation databases which focus heavily on English-language research from the USA and north and western Europe. Moreover, topics covered by these databases often relate to the interests of industrial stakeholders rather than those of local communities. More inclusive research assessments are needed to overcome the ongoing marginalisation of some peoples, languages, and disciplines and promote engagement rather than elitism.
It’s really helpful to read this kind of perspective, but in my experience there is a strong sense that institutions and senior management place much importance on the established value of the impact factor. We have decided to carry out consultation with stakeholders, but in the absence of a convincing alternative (which in our case we simply haven’t had time to implement as yet) I am not sure what we would be asking our stakeholders to compare or comment on. There is such a range of options being implemented by Open Access publishers, that we can a learn a lot from their example and work towards putting in place improvements that will help establish what might be an alternative or a complimentary perspective to the traditional impact factor.
Measuring beyond impact: peer review
Through our Editorial Board, the working group has now also begun to look at platforms like Publons, which promises to ‘integrate into the reviewer workflow so academics can track and verify every review and editorial contribution on the fly and in complete compliance with journal review policies’ (read more). It’s clearly a widely-used platform and some colleagues seem to be enthusiastic users, so it’s made me consider what this kind of platform could add to the user experience alongside innovative tools to measure impact. As a journal that does not charge any APCs, the value proposition for authors is clear, but resources to improve the experience of reviewers are limited. More work is needed for us in this area to examine whether we can compliment our efforts to improve the ways in which the impact is measuring could be complimented by enhancing the experience of peer review.
Read more (with thanks to everyone who’s sent me comments or links):
Information for publishers from DOAJ:
DOAJ does not believe in the value of impact factors, does not condone their use on journal web sites, does not recognise partial impact factors, and advocates any official, alternative measure of use, such as article level metrics.
There is only one official, universally recognised impact factor that is generated by Thomson Reuters; it is a proprietary measure run by a profit-making organisation. This runs against the ethics and principles of open access and DOAJ is impact-factor agnostic. DOAJ does not collect metadata on impact factors. Displaying impact factors on a home page is strongly discouraged and DOAJ perceives this as an attempt to lure authors in a dishonest way.
If you have been following the reporting on the gender pay gap in the UK, then this has been a sobering week indeed. You can search for the reports from different employers here. I have had a look through many of the education providers and sector bodies that I work with and the scale of the ‘gaps’ highlighted in some of the reports is staggering. Not a surprise, given my day to day experience of the sector, but still – staggering.
As a chief executive I have reflected much during this week on how we can change things across the system. There are so many aspects to the problem that there is definitely no shortage of things we should tackle and there is much to do in relation to the professionalisation of Learning Technology.
But on a more personal note this has also reminded me of how important it is that we continue to work towards achieving greater equality – in all its forms. So with a large international conference on openness in education just around the corner I hope that there’ll be much to learn and discuss from different, global perspectives. I also want to help give a voice to this conversation together with colleagues, and make sure that we consider equality in the context of openness.
Powerfully, Catherine Cronin spoke of criticality, equality and social justice at OER17 in London last year. In the closing plenary we were asked to respond to a call to action… #Iwill #OER17 and many participants in the room and on social media joined in, making their voices heard and sharing their aspirations, making a commitment to taking action. I think it’s time to renew our vows to take action #OER18.
Last year I worked on finding a sustainable new home for the Open Access journal Research in Learning Technology. As part of my work for ALT, this was the third transition I have worked on since 2008 and during this period I have contributed to the thinking around Open Access publishing in Learning Technology, often through ALT’s contribution to initiatives such as the 2012/3 ‘Gold Open Access Project‘. This year I will be working with a new group set up by ALT to steer the future development of the journal:
A new Strategic Journal Working Group to help steer the development of the journal now being published by ALT in partnership with Open Academia has been established and we are grateful that representatives from other scholarly bodies who are publishing in a similar model, have agreed to join the group to share best practice and support each other. The group is chaired by Prof Neil Morris, who also chairs the Editorial Board, and we are delighted to welcome colleagues from ascilite, ILTA and the OLC alongside our Editors.
As well as learning from each other, the group is going to be examining alternative ways of measuring impact (alternative to the established impact factor, which the journal has not been awarded to date). This is an area I am particularly interested in for three reasons:
Knowledge exchange happens elsewhere
Firstly, much of the most cutting edge research and practice in Learning Technology is not published in formal journals. Even the most responsive Open Access peer-review system can’t necessarily keep pace with the quickly changing technology landscape we work in and so less formal ways of knowledge exchange on blogs, on social media or in project reports is often more important and useful.
Different media Secondly, a lot of the most interesting ideas may be shared as videos, drawings, data visualisations and so on, in short, they may not easily fit into the traditional formats measures like an impact factor were designed for. What we cite and where we link to can be harder to track. As we use new technologies to communicate and share information the way in which we cite/link to sources needs to adapt.
Crossing boundaries
Another aspect of what makes measuring impact interesting in Learning Technology is the way we cross boundaries of disciplines in research, policy and practice. Coming from a discipline like Anthropology, which has a hugely broad frame of reference depending on what you specialise in, it still seems challenging to what extent the work of Learning Technologists crosses boundaries.
So, keeping all this in mind, here is where I am in my work to research alternative ways of measuring impact…
I started with a a blog post DOAJ LAUNCHES THE DOAJ BEST PRACTICE GUIDE which I came across as Research in Learning Technology was recently awarded the DOAJ best practice seal. It’s a useful new guide that provides a lot of helpful information to publishers, authors and policy makers interested in Open Access publishing. One of the resources it referred me to was a tool for authors called ThinkCheckSubmit. Whilst not specifically talking about how the impact of the journal is measured, it does ask authors to check the publisher’s information for example how the journal is indexed or whether the publisher is a member of OASPA or COPE.
Altmetrics are metrics and qualitative data that are complementary to traditional, citation-based metrics. They can include (but are not limited to) peer reviews on Faculty of 1000, citations on Wikipedia and in public policy documents, discussions on research blogs, mainstream media coverage, bookmarks on reference managers like Mendeley, and mentions on social networks such as Twitter.
Sourced from the Web, altmetrics can tell you a lot about how often journal articles and other scholarly outputs like datasets are discussed and used around the world. For that reason, altmetrics have been incorporated into researchers’ websites, institutional repositories, journal websites, and more.
Whilst I have been familiar with altmetrics for some time, I hadn’t actually come across the history in much detail and I found it really helpful to visit http://altmetrics.org/manifesto/ and read up on some of the older posts. It gave a me a better insight into the thinking that informed the development of the tools and policies involved. It also reminded me of the 2014/5 HEFCE publication called “The Metric Tide” which includes an executive summary, literature review and correlation analysis. As part of the recommendations the report featured, it states:
These recommendations are underpinned by the notion of ‘responsible metrics’ as a way of framing appropriate uses of quantitative indicators in the governance, management and assessment of research. Responsible metrics can be understood in terms of the following dimensions:
Robustness: basing metrics on the best possible data in terms of accuracy and scope
Humility: recognising that quantitative evaluation should support – but not supplant – qualitative, expert assessment
Transparency: keeping data collection and analytical processes open and transparent, so that those being evaluated can test and verify the results
Diversity: accounting for variation by field, and using a range of indicators to reflect and support a plurality of research and researcher career paths across the system
Reflexivity: recognising and anticipating the systemic and potential effects of indicators, and updating them in response.
The recommendations outlined in the report apply mostly to HEIs, funders and government bodies. There are some however that are directly aimed at publishers. These are:
Publishers should reduce emphasis on journal impact factors as a promotional tool, and only use them in the context of a variety of journal-based metrics that provide a richer view of performance. As suggested by DORA, this broader indicator set could include 5-year impact factor, EigenFactor, SCImago, editorial and publication times. Publishers, with the aid of Committee on Publication Ethics (COPE), should encourage responsible authorship practices and the provision of more detailed information about the specific contributions of each author. Publishers should also make available a range of article-level metrics to encourage a shift toward assessment based on the academic quality of an article rather than JIFs. (Publishers)
Publishers should mandate ORCID iDs and ISNIs and funder grant references for article submission, and retain this metadata throughout the publication lifecycle. This will facilitate exchange of information on research activity, and help deliver data and metrics at minimal burden to researchers and administrators. (Publishers and data providers)
Interestingly there are a number of recommendations for HEFCE and future REF exercises, that as far as I can tell seem to have not necessarily been picked up given the recent closure of HEFCE. Still, it is useful to revisit this report and its recommendations within the wider context of thinking about alternative ways of measuring impact.
I also came across an ebook that is new to me, published by altmetrics and Scholastica, entitled “The Evolution of Impact Indicators”. The whole publications looks extremely useful and has a lot of references that are relevant to my work, but the chapter that I am particularly interested in is called “Beyond the Impact Factor”. It discusses a number of alternatives to the impact factor including the EigenFactor and the H-Index. The H-index is probably the one I am most familiar with, but it’s also useful to remind myself of how this is tracked:
Google Scholar: Google Scholar provides the h index for authors who have created a profile.
Publish or Perish: Publish or Perish is a software program that retrieves and analyzes academic citations from Google Scholar and provides the h index among other metrics. Publish or Perish is handy for obtaining the h index for authors who do not have a Google Scholar profile.
Scopus: Scopus provides a Citation Tracker feature that allows for generation of a Citation Overview chart to generate a h index for publications and citations from 1970 to current. The feature also allows for removal of self-citations from the overall citation counts.
Web of Science: Web of Science allows for generation of the h index for publications and citations from 1970 to current using the “Create Citation Report” feature.
Now that I have started to refresh my memory of some recent developments, my next step will be to take this back to my desk, do some work on the journal itself and compare notes with my colleagues from the other publishers.
