Author: Maren Deepwell

  • The quality of metrics matters: looking ahead to #EdTechIE18

    The quality of metrics matters: looking ahead to #EdTechIE18

    I am really looking forward to taking part in ILTA’s upcoming conference, TEL Quality Matters – People, Policies and Practices, 31 May – 1 June at IT Carlow. You can see the full programme here http://programme.exordo.com/edtech2018/ .

    Working with ALT’s new strategic working group for the development of the Open Access journal Research in Learning Technology, I have been working on understanding more about alternatives to the established Impact Factor for independent Open Access journals generally and more specifically for researchers in Learning Technology (revisit my first and second post on this subject) and next week’s conference is a valuable opportunity for me to meet some of the colleagues who are part of the group and find out more about their experiences.

    Meanwhile I have come across two new interesting articles (thanks to Neil Morris for making me aware of the blog post).

    The first is another interesting post from the LSE Impact BlogThe academic papers researchers regard as significant are not those that are highly cited . The authors describe the current perspective as follows:

    Citations, JIF, and h-index have served as the triumvirate of impact evaluation for many years, particularly in STEM fields, where journal articles are frequently published. Many studies have pointed out various flaws with reliance on these metrics, and over time, a plethora of complementary citation-based metrics have been created to try and address various proficiencies. At the same time, we see altmetrics emerging as a potential alternative or complement to citations, where we can collect different data about the ways in which research is viewed, saved, and shared online.

    The authors share useful insights from their work surveying chemistry researchers to “gauge their perceptions of significance, importance, and highly cited materials. The results, while not truly startling, were nevertheless a stark illustration of how different these concepts are.”

    The post ends with reflecting on how meaningful assessment of research can be developed within individual disciplines and I think this is a useful approach for Learning Technology in particular, given the diversity of research and research active professionals as well as the broad range of media and technologies we utilise.

    Another interesting post I have come across is this from the Scholarly Kitchen which examines the role of preprint repositories and their impact on journal citation rates https://scholarlykitchen.sspnet.org/2018/05/21/journals-lose-citations-preprint-servers-repositories/ . The article in itself is interesting, but in particular I found it helpful to look at how some of the practices in Open Access publishing have developed. What I am particularly interested in is the messiness that results in different versions of the same research being linked to and cited on different platforms and how this impacts on citation rates and access to research over time.

    One of the questions in the post is why people keep linking to and citing pre-prints even when the published version of an article is available. There is no tidy way of making everything link up in one place and so whichever platform is used most becomes the most useful even though the life cycle of publication is designed with a different aim in mind. The post uses the example of bioRxiv to examine how this, but this can easily be applied to other, similar platforms formal and informal.

    The post concludes with this:

    A citation is much more than a directional link to the source of a document. It is the basis for a system of rewarding those who make significant contributions to public science. Redirecting citations to preprint servers not only harms journals, which lose public recognition for publishing important work, but to the authors themselves, who may find it difficult to aggregate public acknowledgements to their work.

    I am looking forward to exploring these and related questions next week. You’ll be able to access all details of my presentation via the conference website http://programme.exordo.com/edtech2018/delegates/presentation/1/ . I’d also like to add a note of thanks to the wonderful Bryan Mathers for making his Visual Thinkery available under Creative Commons licences and thus enabling me to use (and credit) them in my presentation. Thank you.

  • Weekend post: leadership lesson from my favourite starship captain

    Weekend post: leadership lesson from my favourite starship captain

    As you may have gathered, I like Star Trek and Captain Picard is my favourite captain in that particular universe. I don’t agree with every aspect of his approach to leadership, but I like to think that we share a lot of values and every now and then I come across something he says or does that really resonates with me.

    I was watching an old episode of The Next Generation recently (fellow nerds: this is the one in which android Data loses his confidence after someone beats him in a futuristic strategy game), in which Picard says to Data that it is possible not to commit any mistakes and still loose. I like the episode in general, but that particular bit of dialogue stuck in my head, because it’s related to things I have been thinking about recently.  Here’s why it resonates with me:

    Much of the time following all the right steps and doing all the right things does result in the desired outcome. At least when things are reasonably within your sphere of influence. So, when something goes wrong or something bad happens in response to what we do, we seek to establish what we did wrong, where we made a mistake – same as Data does in the Star Trek episode. He thinks he must have made a mistake because he lost the game. It’s logical, and it’s comforting to have a sense of control, to believe that doing the correct thing will result in the correct outcome.

    But, in many instances, that’s not the case. That’s not the case in the well-ordered universe of the Starship Enterprise and certainly not the case in the chaotic reality we call home.

    The advice from Captain Picard cheered me. I thought it was a useful lesson to revisit because it reminded me that not everything that goes wrong is my responsibility or within my power to change. It made me reflect on the fact that even when things go wrong, often you can still get to your overall aim in other ways. And it made me think about how valuable it is to make mistakes, to learn how to cope with making them, learn from them and move on with your confidence still intact. 

    In a high-performing environment, like Starfleet in this instance, it’s doubly important to be considerate about that part of your working culture. To be able to share when things go wrong, to make the best of them and to move on. And that a little kindness, a moment of listening or recognition can go a long way to helping oneself and others weather difficult moments like that. 

    Back in the Star Trek story, a cheered Data returns to duty with the Captain’s advice fresh on his mind, but he also returns to face his opponent at the game table. This time, Data plays not to win, but for a stalemate, and he succeeds in not losing. He changes his winning conditions, what he wants to achieve to succeed. It’s a great ending to the episode and a good example to follow.

  • My week as guest curator of @femedtech

    My week as guest curator of @femedtech

    This week I’ve been volunteering as a guest curator of @femedtech. I actively work to promote equality in everything I do, so when the opportunity came up to support this growing network as a Twitter curator, I took the opportunity to help out gladly – but I wasn’t really sure what it would be like, taking over the voice of this kind of account. So one week in and with one more week to go, here is my personal reflection on how it’s going.

    Fortunately I had guidance, encouragement and advice from Frances Bell and Helen Beetham from the outset and they shared their experience and advice generously. I liked their suggestion to add my own identity to the account, which I did, and then I started tweeting and re-tweeting and following and.. a few days in everything started to feel a bit easier.

    I also set up a TAGS Twitter archive and TAGSExplorer for the #femedtech hashtag, to help chart and visualise the conversation as it evolves:

    As you can see, it’s still a fairly small conversation, but it is starting to grow and I found it really interesting to be able to see what is going on.

    Another approach I tried was to find and follow accounts with similar values or related ideas, such as @uncommon_women and the @feministintrnet for example and to welcome new followers or volunteers as they arrived. Hello to you all!

    With another week ahead, I hope I’ll get a chance to generate a bit more of a conversation and prompt more contributions (tagged as #femedtech) and have the chance to contribute to the development of information for future guest curators with Helen and Frances. My long term hope is that we can help draw together some of the existing initiatives, amplify the voices of those actively working to promote feminism and equality in Learning Technology and connect more people with each other. There are certainly enough challenges out there to be tackled, so let’s hope that we can help connect the dots to make a stronger movement.

    Note: this post was updated with a better TAGSExplorer view and links (thanks to @mhawksey)

  • Which direction to take… researching alternative ways of measuring impact in Learning Technology

    Which direction to take… researching alternative ways of measuring impact in Learning Technology

    This is the second post about my current work on researching alternative ways of measuring impact in Learning Technology. Go back to the first post in which I have set out the context of my work and what I am particularly focused on.

    Alongside the practical work with the ALT Journal Strategic Working Group, I am pleased that my proposal of a short session The quality of metrics matters: how we measure the impact of research in Learning Technology’ has been accepted for ILTA’s Annual Conference in Carlow, Ireland later this month. 

    In the meantime, I have been doing more reading and research into innovative ways of measuring impact and this time my work has come up against some very practical questions, not least because as a UK-based publisher we are in the process of ensuring the the journal’s operations comply with the incoming GDPR legislation. Open Source journal systems are not at the forefront of compliance and like other independent publishers we work as part of the community to move towards compliance.

    At first glance factors like GDPR may not seem to be closely related to how impact is measured, but my thinking links them closely as a lot of the opportunities around developing the journal are dependent on technical solutions that have data processing implications:

    A convincing alternative
    Discussing how important having an impact factor is quickly runs into the question of what the alternative looks like. As well as the technical challenges in implementing innovative tools or mechanism for measuring impact (to which the new GDPR legislation adds another level of complexity), the sustainability and longevity of both tool and data storage need to be examined. For example, introducing a tool like Altmetrics requires us to educate all stakeholders and ensure that the level of digital literacy required is not a barrier to making the tool useful. The user interface and experience needs to be robust and practical, building confidence in alternative or innovative ways of measuring impact. With new tools and platforms being created all the time there is a certain amount of churn and in order to really build a convincing alternative there needs to be a certain level of consistency.

    Scrutiny of new vs. established ways of measuring impact
    The kind of scrutiny with which we are examining alternative ways of measuring impact isn’t easily applied to the established method. There is a critical discourse, for example this recent blog post on the LSE impact blog, which argues:

    Many research evaluation systems continue to take a narrow view of excellence, judging the value of work based on the journal in which it is published. Recent research by Diego ChavarroIsmael Ràfols and colleagues shows how such systems underestimate and prove detrimental to the production of research relevant to important social, economic, and environmental issues. These systems also reflect the biases of journal citation databases which focus heavily on English-language research from the USA and north and western Europe. Moreover, topics covered by these databases often relate to the interests of industrial stakeholders rather than those of local communities. More inclusive research assessments are needed to overcome the ongoing marginalisation of some peoples, languages, and disciplines and promote engagement rather than elitism.

    It’s really helpful to read this kind of perspective, but in my experience there is a strong sense that institutions and senior management place much importance on the established value of the impact factor. We have decided to carry out consultation with stakeholders, but in the absence of a convincing alternative (which in our case we simply haven’t had time to implement as yet) I am not sure what we would be asking our stakeholders to compare or comment on. There is such a range of options being implemented by Open Access publishers, that we can a learn a lot from their example and work towards putting in place improvements that will help establish what might be an alternative or a complimentary perspective to the traditional impact factor.

    Measuring beyond impact: peer review
    Through our Editorial Board, the working group has now also begun to look at platforms like Publons, which promises to ‘integrate into the reviewer workflow so academics can track and verify every review and editorial contribution on the fly and in complete compliance with journal review policies’ (read more). It’s clearly a widely-used platform and some colleagues seem to be enthusiastic users, so it’s made me consider what this kind of platform could add to the user experience alongside innovative tools to measure impact. As a journal that does not charge any APCs, the value proposition for authors is clear, but resources to improve the experience of reviewers are limited. More work is needed for us in this area to examine whether we can compliment our efforts to improve the ways in which the impact is measuring could be complimented by enhancing the experience of peer review.

     

    Read more (with thanks to everyone who’s sent me comments or links):

    Information for publishers from DOAJ: 
    DOAJ does not believe in the value of impact factors, does not condone their use on journal web sites, does not recognise partial impact factors, and advocates any official, alternative measure of use, such as article level metrics.

    There is only one official, universally recognised impact factor that is generated by Thomson Reuters; it is a proprietary measure run by a profit-making organisation. This runs against the ethics and principles of open access and DOAJ is impact-factor agnostic. DOAJ does not collect metadata on impact factors. Displaying impact factors on a home page is strongly discouraged and DOAJ perceives this as an attempt to lure authors in a dishonest way.

    Full information here.

  • Re-post #altc: My Chief Executive Officer’s Report, May 2018

    Re-post #altc: My Chief Executive Officer’s Report, May 2018

    This is my report to Members of ALT for May 2018, originally published on the #altc blog here.

    Dear Members

    As I am writing this we are just beginning a particularly busy period for the Association, so my report to you this time will be a whistle-stop tour of what’s happening across our community. I am pleased in particular to welcome new member organisations who have recently joined ALT: Staffordshire University, Ajenta, MyKnowledgeMap, TES, Northern Regional College, University of Dundee and Wrexham Glyndwr University.

    A global perspective on professionalisation in Learning Technology

    On 3 May 2018 I was honoured to join the BOLT project organisers and partners at the Hong Kong Polytechnic University at their blended learning Symposium @PolyU,  a celebration and culmination of the BOLT project. As this 4-year University Grants Committee-funded project draws to a close, the Symposium celebrated at its impact so far – evidenced by its shortlisting for the Reimagine Education 2017 awards – and also looked to the future and its sustainable legacy. A particular highlight for me was being invited to present two members of staff, Seth Neeley and Arinna Nga Ying Lee, with their CMALT Certificates.

    Congratulations to Seth, Arinna and the 20 other individuals who have achieved CMALT accreditation so far this year.  

    Launching a new Award for the Learning Technology Research Project of the Year

    One of our strategic priorities for this year is to enhance recognition for research in Learning Technology and the launch of this year Award helps us achieve this aim. The ALT Learning Technologist of the Year Awards celebrate and reward excellent research and practice and outstanding achievement in Learning Technology. Established in 2007, the Awards have established a benchmark for outstanding achievement in Learning Technology on a national scale and attract competitive entries from the UK and internationally. All entries are reviewed by an independent judging panel chaired by the President of ALT. We gratefully acknowledge the support from our sponsors Catalyst, open course technologists, for supporting the Awards this year. The Awards are now open for entries.

    We seek Member input for UNESCO Recommendation on Open Educational Resources (OER)

    ALT is collating a response for the following UNESCO consultation. Please use this shared doc to provide input https://go.alt.ac.uk/2FsAbBA . If you would like to provide input for the response, please use the heading structure provided. Alternatively you can also email your contribution to maren.deepwell@alt.ac.uk . The deadline for responses is 1 June 2018.

    General Data Protection Regulation (GDPR)

    As I’m sure many of you are aware new data protection regulations come into effect at the end of this month. We welcome these regulations as they will hopefully permit greater transparence in the use of personal data including in learning and teaching. In preparation of the new regulation we have updated ALT’s own Privacy Policy and carried out a number of actions in-line with guidance from the ICO.

    We have also been supporting Members with a series of webinars to raise awareness around GDPR in learning in teaching. As part of this we were delighted to host Martin Dougiamas, Moodle Founder and CEO, along with Gavin Henrick, Moodle Business Development Manager to highlight actions the Moodle community have taken, Stephan Geering, Blackboard Global Privacy Officer and Associate General Counsel, and Mark Glynn, Head of the Teaching Enhancement Unit at DCU. If you missed any of these sessions recording and resources have been added to the event pages accessible from the past events section of our website.

    ALT Annual Survey data & report

    I will conclude my report with a reflecting briefly on the findings from this year’s ALT Annual Survey, the report of which was published in March by my colleague Martin Hawksey. As with previous years the Annual Survey is designed to:

    • understand current and future practice;
    • show how Learning Technology is used across sectors; and
    • help map the ALT strategy to professional practice to better meet the needs of and represent our Members.

    With the survey in its fourth year we are able to record and report and number of changes. This year some of the biggest changes are in the enablers and drivers for use of learning technology. The insights gained go beyond the trends in technology and organisational change, but help us understand the needs of staff enabling students and building a more empowered relationship with Learning Technology. Themes that we can look forward to exploring more at ALT’s Annual Conference this September.

    Maren Deepwell

     Maren Deepwell, Chief Executive of the Association for Learning Technology (ALT), @marendeepwell

    If you enjoyed reading this article we invite you to join the Association for Learning Technology (ALT) as an individual member, and to encourage your own organisation to join ALT as an organisational or sponsoring member

     

  • 3 months in… hitting our first milestones as a virtual team

    3 months in… hitting our first milestones as a virtual team

    This is a joint post with Martin Hawksey (cross-posted here).  If you have missed our earlier posts we encourage you to revisit the beginning of the story of how we, as senior staff, lead our organisation to adopt virtual operations.

    April

    This month we reflect on the first 3 months operating as a virtual team, delivering our first few big milestones and look back at the OER18 conference.

    Maren: It’s been three months now since we transitioned to operating as a virtual team. Looking at the bigger picture, I feel we are on the right track [cue celebratory sounds here]. A lot of what we had planned and prepared for is working and with some of this year’s key deliverables successfully achieved, we have evidence that we are achieving what we need to. As an organisation we changed more than the way we operate over the past three months as we started to employ staff directly at the same time. For me, that part of the transition had to take priority over everything else. Payroll, pensions, tax and HR had to be in steady state, putting staff welfare, support and recruitment first. Those were the things I really worried about. So it’s a relief to have managed some of the biggest risks successfully and for more and more of the way we operate as a new employer to happen in steady state. When I reviewed our progress recently I realised that whilst I was focused on the transition, we have also made a lot of strategic progress – including delivering one of the largest events in our calendar, the OER18 Conference. What’s your perspective now that the event is successfully behind us?

    Martin: For me, I was interested in how the runup to OER18 conference would work out. This was our first big event where the entire team were distributed. With multi-day events I think it’s hard to appreciate just how much goes into them if you haven’t organised one yourself. Even in a digital age various things need to go to print, you’ve got material and equipment that needs to be delivered and there are some practical things like getting conference badges prepared. As part of our distributed organisation we now have distributed resources which needs some extra logistical planning to bring together at the venue. Overall this aspect went well but one of our couriers let us down and failed to pickup a next day delivery. As it happens the pickup was for some extra banners which in the end we could get away with not taking. Fortunately the bulk of material was handled by our regular courier who we have a long standing relationship with. The challenge now for us as a distributed organisation is we need to develop relationships with additional service providers, either out of doing things differently or because we are all not in the same place. This takes time and effort which I think should be factored in if you are thinking about moving your organisation to a distributed or virtual structure. Has OER18 highlighted anything for you we should take a new approach on?   

    Maren: Yes, I think it has and it feels timely. The work on the conference felt like a useful way for us to pull together as a team and work together virtually. It helped highlight which parts of our new virtual set up are working well and it also made me realise how much has changed in a short space of time. Because the change has been strategic and welcome, it’s felt inspiring and positive overall, but we mustn’t forget that change takes time. Our expectations of what we want to achieve are high and sometimes I have to remind myself that it takes a lot of doing to put in place new ways of doing things. Doing something once isn’t enough. In my mind the kinds of considerations you are talking about, like building new relationships with suppliers, adapting what we do and how we do it, are helpful to me now – and to us as a team, but they wouldn’t have been a few months ago because there was no capacity to handle any more novelty. Now, when we evaluate the OER conference, I feel we have the capacity to put what we’ve learnt to good use in the run up for the next event. Another aspect of running an event as a distributed team I am thinking about is meeting up beforehand. For example, we had an evening to get ready this time, but not many hours at the venue. I feel this is particularly important for members of the team who are new and haven’t been part of delivering an event with us. As we are recruiting at the moment, I am considering that with new members of staff joining soon, ahead of a bigger conference I might value more time. What are your thoughts on this?

    Martin: Interesting point about time considerations. Knowing our Annual Conference is our next big event where we have at least 4 times more delegates it’s going to be important to factor in some of the practicality of badge stuffing and conference material gathering. Something that I only considered after OER18 was that we could do more to distribute our printing, both in terms of when and where it is done. For example, as part of our move to a distributed organisation we limited the purchase of printers to just getting one for our Finance Officer. I already use Google Cloud Print at home, which essentially lets you turn any printer into a network printer. Adding our printer to Google Cloud Print would allow us to share the printer with our team. There is still a logistic issue of getting materials we print to events but at least they would be in one location where we also have access to a reliable courier. The occasional printing we do is however only a small part of event delivery and in the bigger picture it would be useful to revisit our conference plan to see what we can prepare earlier to remove some of the end loading, like batching badge production. Whilst the Cloud Printing is a small example I think it reflects what you were saying about a wider change in the organisation. It feels like we have more agility in how we approach and solve problems.  

    Maren: I agree with that. The first few months we had our preparations to build on and then the event to focus on, now we are moving on to the next phase: doing things differently, expanding (admin) support for virtual operations and updating our plans. The conference was a good catalyst to highlight the kind of questions you raise and also our skills and competencies and the gaps in them. One of the problems in Learning Technology I come across again and again is how to build a successful organisational culture when tech, expectations, milestones etc keep shifting. I relate to that in a different way now, because as you point out, we have a lot more scope now to be flexible, to solve problems creatively rather than having to work around them. We’ve committed to being more agile and I’m discovering what that means in practice for me as an individual, for us as a team and as an organisation all over again.

    Martin: Noted computer pioneer Alan Kay uses a quote from ice-hockey player Wayne Gretzky “[a] good hockey player goes to where the puck is, [a] great hockey player goes to where the puck is going to be”. If a system is in steady state everything is predictable, removing this means as an organisation we have more control about deciding where we want to be. By creating an organisational culture where there is scope for not following the puck we become more comfortable in not being in steady state, as a result more confident in finding solutions to problems as they emerge. Something I think required to make this successful is an existing confidence within the team … success breeds success.