Process notes for an evaluation study

As I mentioned yesterday, I have just begun a study to find out how refugees can best learn from MOOCs and other online resources, with the very kind support of the Kiron Higher Education Organisation‘s staff, volunteers and learners, who have agreed to participate as a case study. I will give a bit more background and context to the study here for anyone who may be interested in following it. First though, it’s important to note that Kiron themselves, and several of their Higher Education (HE) partners are already undertaking research in this area, for example by comparing the performance of refugees in MOOCs with that of enrolled students in a parallel mainstream HE course (where both groups take the identical assessment), and mapping the performance outcomes of Kiron’s own learners who have participated in live online tutorials against those who have not. I hope that my research will augment and supplement this work.

I plan to carry out the research openly (following Pitt et al, 2016, and many other friends and colleagues in the open education sector) – maintaining strict anonymity for the refugee participants so as not to compromise them in any way – but sharing my processes, dilemmas and interim findings here under an open licence. The current study will continue until the end of 2017, and I aim to have a paper ready to share in the new year. This is to fit in with the requirements of my PhD programme, which has three modules in the first two years, each one culminating (hopefully!) in a publishable piece. This paper is for the second module, which is on evaluation in HE. (The first was on policy and change in HE – I will be presenting it at the ALT conference in September and blogging about it soon.) My research plan, as described here, has been developed with help from my PhD module tutor, Murray Saunders.

The focus of the research will be on what, in the learners’ perception, has helped them to succeed in learning online. The evaluation will be ‘developmental’ (Patton, 2011), meaning that the findings could be used to trigger change or enhancements to Kiron’s support programme for refugee learners. Further potential uses of the evaluation by other groups might include informing support programmes aimed at inclusivity or widening participation in higher education through open education.

I hope that this study will be the start of a more in-depth investigation into the learning strategies and open education support processes that enable refugees to succeed in higher education. Whilst I am individually responsible for the outputs of my PhD, I will be working in close partnership with the Kiron team and the research participants (a group of Kiron’s learners I met last weekend), and I hope to collaborate with other researchers working in the fields of open education and migration; I will also be sharing work-in-progress with my PhD peers at Lancaster and with fellow members of the GO-GN (Global OER Graduate Network).

IMG_3816

Clarifying points in the brainstorm during discussion in focus group session with Kiron learners on 5/8/2017 (Photo by Donya Zikry)

Next – a few notes about my research methodology and methods for this project. This will be an in-depth qualitative evaluation, with a relatively small number of participants (approximately 15). My methodology will be grounded theory/ categorical analysis. The main idea behind this approach is that the researcher constructs themes from what the research participants have said, and then organises the data accordingly, thereby providing a new lens through which to look at the question or issue being explored.

My data generation (to use a term from Pat Thomson, who makes a well-argued case for not talking about ‘data collection’) methods will involve progressive focusing, meaning that I will have a series of communication events with the participants, each time gathering a bit more information from them, analysing what they have said, and selecting points for more in-depth focus. I will then ask the participants for further information and repeat the analytic process. My data-gathering ‘events’ will include focus groups (which I briefly described in the previous blog post), emails to the participants with prompts for them to reply via email, WhatsApp messages with prompts for them to send me text responses or recorded voice messages, face-to-face interviews (where possible) and online interviews. I will focus some of the research on how these different formats may have furthered (or hindered!) the generation of data. I will audio record all verbal responses and interviews, but will not make full transcripts of these; instead, in the interests of efficiency, I will make notes and will tag specific points in my notes where I want to listen to the recording and transcribe a segment of speech for content analysis.

My analysis of the data will be informed by conceptual frameworks from the literature, particularly Garrison et al’s Community of Inquiry framework (2010), with possible adaptations as recommended by Jaffer et al (2017), and Saunders et al’s (2005) notion of ‘provisional stabilities’. The evaluation will also focus on the usefulness of these conceptual frameworks.

I would love to receive thoughts and questions on this process as I go along, and any links to similar research in progress, so please feel free to comment below.

References

Garrison, D.R., Anderson, T. & Archer, W., 2010. The first decade of the community of inquiry framework: A retrospective. The Internet and Higher Education, 13, pp.5–9. Available at: http://dx.doi.org/10.1016/j.iheduc.2009.10.003.

Jaffer, T., Govender, S. & Brown, C., 2017. “The best part was the contact!”: Understanding postgraduate students ’ experiences of wrapped MOOCs. Open Praxis, 9(2), pp.207–221. Available at: http://openpraxis.org/index.php/OpenPraxis/article/view/565/312.

Patton, M.Q., 2010. Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, New York: Guilford Press.

Pitt, R. et al., 2016. Open Research, Milton Keynes: OER Hub. Available at: http://oro.open.ac.uk/48035/1/OpenResearch.FINAL_.pdf.

Saunders, M., Charlier, B. & Bonamy, J., 2005. Using Evaluation to Create “Provisional Stabilities”: Bridging Innovation in Higher Education Change Processes. Evaluation, 11(1), pp.37–54. Available at: http://evi.sagepub.com/content/11/1/37.abstract.

Posted in Refugees and open education | Tagged , , , | Leave a comment

What helps refugees succeed in learning from MOOCs?

The title of this blog post is a question I would like to find an answer to through my PhD studies. As a first step towards this goal, I was privileged to be able to join a group of 14 refugees who have been studying through MOOCs, supported by Kiron Higher Education Organisation, at a study weekend held in Berlin on 4-6 August. This group, from all over Germany, are amongst the first 30 or so of Kiron’s participants to have met the requirements – fluency in German and participation in a number of recommended MOOCs – to apply for entrance to German universities. (Kiron currently supports over 2,000 refugees in almost 100 countries, with offices in three focus countries; approximately ten learners are already on their way to transfer from the Kiron support programme to universities, leading the way for the group I met on the weekend.) Kiron provides academic support in a university preparatory programme as well as five discipline-related “study tracks”, and last weekend’s event was focused on the Business & Economics and Mechanical Engineering study tracks. Volunteer tutors from Kiron’s partner RWTH Aachen ran tutorials, covering a syllabus that would normally be taught over at least two months in one and a half days. Because the students had already worked through much of the content in the respective MOOCs, they were well prepared and eager to use the opportunity to augment their learning.

20170805_124751 (1)

Kiron students answering the question “What helped you learn in your online studies?”

I was given two slots on the agenda – an hour before lunch with half the group, and an hour after lunch with the other half. In the first focus group I started by asking “What helped you learn in your MOOC studies”, and then quickly modified the question to “What helped you learn in your online studies?”, when I found that two of the learners had not yet completed any MOOCs; however they had learned online from YouTube and Google. To elicit as much information as possible in the short time available, I ran the two sessions like workshops, starting with a “silent brainstorm” in which I asked the students to write their responses to the question on A4 sheets. They then stuck their sheets on the wall, grouping similar ideas together, and we ended with a group discussion around some of the topics that had emerged. All participants consented to me contacting them afterwards for individual, online interviews, to follow up in more detail on the points they had made. Before doing so, I will carry out a preliminary analysis of the focus group discussions and liaise with staff at Kiron to identify themes that they would like me to explore further with the participants, in order to help them enhance the support they provide for their learners. I will also receive some input from my module tutor at Lancaster University, Prof. Murray Saunders, to help me shape the research plan.

It was an immense privilege to be with Kiron’s tireless and upbeat staff (Sophia, Elyzabeth, Mehmet, Florian, Donya, Milena and Laura) and dedicated volunteers (Gui and Lucille), the energetic volunteer tutors (Stefan, Aras, Ibrahim and Mihir), and the students, who were amongst the most motivated adult learners I have ever met. I was struck by the atmosphere of optimism, kindness and gentle humour that pervaded the weekend, and the strong bonds of friendship between all of us that were built in such a short space of time. I am hugely looking forward to learning more from this special community about how open, online learning can be harnessed for the benefit of refugees and the wider society of which they are a part, and will be sharing my learning journey here as I go along. I welcome comments from “Kironistas” and other open education practitioners who are interested in this project.

Many thanks to Sophia Burton and Florian Rampelt for being my “critical friends” for this blog post.

Posted in open learning, recognition | Tagged , , | 4 Comments

German universities prepare for recognition of MOOC-based learning

Continuing with the German theme from my last post… Last week I was honoured to be invited to speak at a workshop in Berlin held by Kiron Open Higher Education for staff from their German partner universities, on the topic of recognition of prior learning through massive open online courses. My slides from the event are here.

Kiron was set up by Vincent Zimmer and Markus Kressler in mid-2014, initially as a Kickstarter start-up, and is now an established non-government organisation, aiming to support refugees to enter the higher education system. With a current student body of over 2,300, the organisation has developed a model in which learners spend two years studying MOOCs, with guidance and support from academic mentors, counsellors and friendly “buddies”, whilst also learning German. Upon successful completion of this phase, the learners based in Germany – approximately 50% of the total “cohort” – will apply to a German university for recognition of their learning equivalent to the first year of a degree programme of their choice (within certain limits – more on that below), enabling them to be accepted into second year. About 40 of us gathered at the stylish BMW Foundation office in Berlin (an indication of corporate philanthropic support for the initiative) to discuss scenarios for this future.

Kiron has already made significant progress towards its goal in terms of developing an educational model that is acceptable to its partner institutions. The curriculum is divided into five subject streams, called “study tracks”, and for each of these subjects a mapping exercise has been done to identify (a) the most common intended learning outcomes in German university curricula, and (b) the MOOCs whose outcomes map most readily onto the local curricula. They have further created a number of so-called “MOOKlets” – booklets which provide information about the selected MOOCs, to help both learners and institutions make decisions about which courses to study/ recognise. I was delighted to hear from Hannes Niedermeier, Kiron’s head of curriculum, that this idea had been inspired by the concept of the “traffic light” model in the OpenCred report that I co-authored for the EU’s Joint Research Centre. (Special acknowledgement is due here to Anne Tannhäuser, the creative power behind this model!)

What struck me most about the gathering was the quiet, determined commitment of the group to the shared vision of integrating refugees into the formal higher education system. Every potential obstacle was treated as a problem to be solved, through a combination of lateral thinking and methodical planning. Amongst the solutions we discussed were the following:

  • Recognition by portfolio – this is often a very labour-intensive process, both for applicants and assessors, but in the Kiron case the concept has been streamlined to simply comprise all the MOOC credentials (certificates or badges) obtained by the learner, along with a short piece of reflective writing on their experiences in acquiring each credential.
  • The use of online proctoring – there was some agreement that this could be as reliable as (if not more reliable than) face-to-face proctoring, and some of Kiron’s partners (e.g. OpenHPI) have already successfully integrated this into their routine delivery of courses.
  • The use of “challenge exams” – one institution, FH Lübeck, had invited a group of 14 Kiron learners who had completed MOOCs to take exams in their chosen subjects, to ascertain whether they had actually achieved the learning outcomes that were required for entry into Year 2 of the respective programmes. The exams were identical to the ones taken by the institution’s own students after a year of study. This “proof of concept” pilot confirmed that most of the refugees in the group had achieved the necessary learning outcomes and could therefore have their learning recognised. In the case of one subject where the learners performed less well, it was noted that the MOOC outcomes did not map very closely onto the institutional programme outcomes, implying that further work needs to be done in terms of either selecting suitable MOOCs, or supporting learners to fill the “gaps” identified between MOOC courses and particular university programmes. (For a flavour of some of the work that FH Luebeck is doing with refugees, see https://integration.oncampus.de.

The workshop day ended on a high note with each person saying what they had got out of it. From everyone’s closing remarks, it was clear that the opportunity to network with others in the Kiron family and address common problems together had been valuable. I will be watching this space closely to see what models are developed at Kiron for the recognition of open learning, and how these might be transferable to other contexts. In the meantime, I will continue my support for Kiron through the Buddy programme and I would encourage any readers who are interested in partnering with a refugee learner to get involved. And no, you don’t need to be a German speaker 🙂

Posted in Uncategorized | 1 Comment

Reflecting on #UKOER at #OER17

Last week I attended #OER17 in London, and am still basking in the warm glow of the shared sense of purpose to open up education for equity and social inclusion. My presentation, which I was privileged to do collaboratively with Airina Volungevičienė and Marius Šadauskas from the ReOPEN project, was entitled Recognition of non-formal, open learning: are we there yet? We looked at findings from the OpenCred study, which sparked the current EU-funded project, ReOPEN (succinctly described by Airina in the videos embedded in the slides).

One of the most striking aspects of this conference, in contrast to previous years, was the prevalence of German participants and presentations about open education in Germany. The reason for this was that the German Ministry of Education has recently provided funding for the establishment of 24 open educational resources (OER)-focused projects, as well as a central OER info-hub, so there is a surge of interest in all things OER in Germany. Listening to many of the talks about plans for the development and implementation of Open Educational Resource projects took me back to the discussions we had during the #UKOER projects.

Chats with German colleagues at the conference led me to think back on what we had achieved, and the challenges we had faced in those projects. There were a few things I can see now that I wished I had known before we started. So, in the spirit of sharing, here are a few tips for colleagues who are starting new OER projects:

  1. While many academics are open to the idea of creating and publishing their teaching materials openly, the fear of unknowingly breaching copyright can be an obstacle. In the UK OER projects we created a robust take-down policy which would kick in within a certain time period (say, 48 hours) of a complaint being made. Here is the one developed by Ale Armellini and the OER team at the University of Leicester, which includes a workflow process diagram. And here is a case study on copyright and policy licensing, produced by Meghan Baxter for Newcastle University. It provides a detailed model of a take-down policy and process.
  2. When reusing other people’s OERs, some folks tend to be forgetful or casual about attributing the creator and adhering to the permissions in the particular CC licence used. The Creative Commons website contains a handy guide to attributing, and BCOpenEd has put out a more detailed guide here. For a discussion about the complexities of attributing OERs, see this blog post by Brenda Padilla.
  3. For an OER initiative to be sustainable (i.e. to live beyond the time period it is funded for), it is essential for academics to take ownership of the OERs they produce. It may seem like a nice idea to have a team that reduces the burden for academics by formatting, copyright checking and uploading materials as OERs, but unless that team is a permanent fixture in the institution and regularly prompts contributors to review and update their OERs, the uploaded resources will soon become out-of-date and will not be refreshed.  An example of good practice in this respect is Humbox, a humanities OER repository created by the University of Southampton in 2008 and still active today. The platform is now open to anyone in the humanities who wants to upload resources. Its strength is that it operates on the basis of a social networking ethos, with academics uploading their own materials, updating them whenever necessary, and also being able to review and reuse each other’s resources.
  4. Being a producer (or a user) of OERs is not a set of technical skills and knowledge (although it certainly includes these), but is rather a mindset. There is a large and growing global community of open education practitioners, who are firmly committed to the agenda of equity in higher education, and are developing open education as a social good. The 2016 report produced for the EU by Adreia Inamorato, Yves Punie & Jonatan Castaño Muñoz on opening up education helps to locate OER work in the wider context of the open educational practices that have been emerging in the higher education sector over the past several years.
  5. Finally, there is a helpful report by Lou McGill et al which provides a synthesis of findings from the #UKOER project, as well as the related SCORE project.

If anyone has any tips to add, or thoughts on what I’ve written here, please do share!

**Update**

Comment via Anna Page on Twitter :

Also talk to other projects who do OEP already and take a look at Becoming an Open Educator

(I most definitely agree – see https://oepscotland.org/)

Posted in Uncategorized | 1 Comment

The secret life of face-to-face learning in MOOCs (Part 2)

Many thanks to the folks who commented on Part 1 of this blog series, and to those people who contacted me privately to tell me about cases where face-to-face learning is playing a significant role for MOOC learners. In this post I will share what I have learned from that communication, as well as briefly summarising some of the literature I have found on the topic. As I am particularly interested in the potential of MOOCs to increase access to higher education in developing countries, that will be my focus in this post. To set the scene, it is worth noting that learners in developing countries tend to face many constraints that are likely to restrict their ability to take MOOCs. To give just one example, my colleague Bernard Nkuyubwatsi, reporting on MOOC usage in Rwanda, points out that limited internet penetration (around 7% in the case of Rwanda) and unreliable broadband connectivity, combined with other issues such as limited awareness of open educational practices and low levels of digital literacy skills, may create almost insurmountable obstacles for potential learners on MOOCs in Rwanda. Where learners do overcome such obstacles in developing countries, I am interested to know whether, and to what extent, face-to-face (F2F) learning is playing a supporting role.

My list of nine questions in Part 1 of this blog series will require more in-depth, longer-term research, and so for today I will just look at three of those questions, with reference to a few initiatives:

  • Who organises the F2F gatherings?
  • What happens in these gatherings? (I’ll combine the answers to this question with the first one, to avoid repetition.)
  • What do participants get out of these gatherings?

Who organises these gatherings and what happens in them?

Some of the F2F gatherings around MOOCs are formal, and are organised by institutions or organisations that provide a facilitated MOOC learning experience for learners. In an example from South Africa, the social enterprise organisation RLabs organises F2F classroom activities based on UCT’s  ‘Becoming a changemaker: Introduction to Social Innovation’ MOOC on Coursera. (For more information, see the article ‘Online course opens doors for changemakers’.)

Another example of a facilitated MOOC comes from India in the form of the TESS-India project, in which the UK’s Open University, with funding from UKAID,  is supporting learning around open educational resources and an EdX MOOC for teacher development.

An example of a slightly less formally organised MOOC F2F gathering comes from the Philippines, where  the US Embassy in Manila hosts occasional MOOC meetups, such as this one on the ‘Art of Poetry’ MOOC. This particular session included a poetry  reading by local participants, and a virtual lecture by one of the course leaders, who joined the meeting from the USA.

At the other end of the continuum, MOOC meetups can be informal, and organised by MOOC learners themselves. In the paper by Bulger, Bright & Cobo (2015), the authors analyse the data from the website meetup.com,  based on invitations to over 4000 MOOC related events, in over 140 countries around the world, over a two-year period. The meet.up website focuses on Coursera-related gatherings, mainly in industrialised countries, but also in some developing countries, including a number of South American countries, Nigeria, Southeast Asia, India and Pakistan. These events tend to be organised by MOOC learners themselves. According to Bulger et al’s analysis, the meet-ups in developing countries are usually more focused on general networking around online learning than specifically looking at particular MOOCs. They speculate that this may be because of the (postulated) small numbers of learners on any given MOOC in the local areas where the meet-ups are being held.

In a related initiative in Jakarta, Indonesia (Firmansyah & Timmis, 2016), some MOOC learners took the initiative to form their own F2F learning community, which they called ‘IDCourserians’. A study was carried out with six group members, over two months in mid-2014. The researchers found that the participants had three aims in common, which were: a) to create a learning place for people taking Coursera MOOCs, b) to promote online learning (in particular, Coursera MOOCs), and c) to localise the content of selected MOOCs for the Indonesian context. At the group meetings, members voluntarily presented about the courses they were studying and facilitated the discussion process following the presentation; they also had some ‘semi-guided discussions’ – general discussions about online learning strategies with reference to a range of different MOOCs; and they also set up some specific study groups based around particular MOOCs.

What do participants get out of these F2F gatherings?

Most of the above-mentioned initiatives do not (yet) have findings reported from the learners’ perspectives; however, the paper by Firmansyah & Timmis (2016) provides some encouraging conclusions. Members of the IDCourserians group reported in this study that they were more motivated to learn new courses, and to finish what they had started – offering indicative evidence that F2F meetups around MOOCs might have a positive impact on MOOC completion rates. These learners also reported that the F2F get-togethers helped them to make the new knowledge they were learning more contextually relevant, as they were able to explain concepts to one another using local frames of reference. They also believed that they had helped one another to overcome difficulties and they experienced greater motivation to complete the MOOC and pursue the certificate because of the achievements of others in the group. The whole shared experience contributed to creating a sense of community and belonging for participants, prompting the authors to describe the IDCourserians group as a Community of Practice (Lave & Wenger, 1991).

As before, comments on this post are welcome. I will be dipping into this theme on and off over the coming months, and will appreciate being able to continue the conversation with others who are interested in exploring these issues.

References

Bulger, M., Bright, J., & Cobo, C. (2015). The real component of virtual learning: motivations for face-to-face MOOC meetings in developing and industrialised countries. Information, Communication & Society, 18(10), 1200–1216.

Firmansyah, M., & Timmis, S. (2016). Making MOOCs meaningful and locally relevant? Investigating IDCourserians—an independent, collaborative, community hub in Indonesia. Research and Practice in Technology Enhanced Learning, 11(11), 1–23.

Lave, J., and E. Wenger. 1991. Situated Learning: Legitimate Peripheral Participation. Cambridge: Cambridge University Press.

Nkuyubwatsi, B., 2013. Evaluation of Massive Open Online Courses (MOOCs) From the Learner’s Perspective. In Proceedings of the International Conference on e-Learning. Prague: IADIS, p. 340. Retrieved from: https://lra.le.ac.uk/handle/2381/28553

Posted in Uncategorized | Leave a comment

The secret life of face-to-face learning in MOOCs (Part 1)

Massive, open, online courses (MOOCs) are generally designed with the intention that learners will learn online. Indeed, the name implies this. And one of the key advantages of MOOCs, from a learning design point of view, is supposed to be that all activity by learners is visible online and therefore available as data for analysis, with the power to help course designers improve the course in the next iteration. Reports on MOOC-based learning usually include quantification of enrolment numbers – often compared with completion numbers – as well as data on average viewing time on any video content, number of clicks indicating resources accessed, and analysis of various kinds of learner activity on the course platform. Data of this nature has been used to explain how learners progress through courses (Perna et al, 2014; Milligan, 2015), and is sometimes claimed to have the capacity to reveal how learning takes place (O’Reilly & Veeramachaneni, 2014).

Some researchers are seeking further insights into the nature of MOOC-based learning by investigating social learning in MOOCs, focusing on interactions taking place within the MOOC discussion forums (for example, Brinton et al, 2014). Other studies have also investigated the nature of social learning in groups set up by MOOC learners outside the MOOC platform, such as on Facebook and other online social networking sites (for example, Alario-Hoyos et al, 2014).

I have a hunch, however, that quite a lot of learning in MOOCs is taking place face-to-face (F2F). I think this is likely to range from spontaneous, informal chats with friends and family members about the MOOC content (I can attest to that happening on the MOOCs that I and my friends have taken), through to regular, planned gatherings of MOOC learners in local meeting places – organised perhaps by a MOOC learner or by a local institution, with varying degrees of formality. I suspect also, that at the latter end of this spectrum, there may be many additional active MOOC participants who have not themselves enrolled on the MOOC, and are therefore invisible to the MOOC organisers.

In order to test my hunch, I did a Google Scholar search using the terms “MOOC” and “study group”/”meet-up”, and found a handful of articles which confirmed that MOOC-related F2F learning is playing an important role in the MOOC experience for some learners in places as diverse as Mexico City, Northern Sweden, Indonesia and Tokyo (Sanchez-Gordon, 2016; Norberg et al., 2015; Firmansyah & Timmis, 2016; Oura et al., 2015). However, the literature in this area seems to be embryonic – and it begs more questions than it answers.

A key question is: what are the effects of F2F interaction between MOOC participants on learning? Linked to that question are several others about the why, the how and the what of F2F learning in MOOCs, such as:

  1. Why do MOOC learners choose to participate in F2F gatherings?
  2. How are F2F gatherings organised? (E.g. via Twitter, Facebook or Meetup.com, by word of mouth, or by someone sticking a notice on a local library/ church hall notice board?)
  3. Who organises the F2F gatherings? (E.g. a learner on the MOOC, a local volunteer teacher, an education institution?)
  4. How often are the F2F gatherings held?
  5. How many people come to the gatherings?
  6. What percentage of the participants at the gatherings are enrolled on the MOOC? (E.g. all, just a few, or just one person who shares content with the rest of the group?)
  7. What happens in these gatherings?
  8. What do participants get out of these gatherings? (E.g. social enjoyment, a meal, drinks, deeper insights or understanding of the MOOC subject matter?)
  9. Do participants have to pay to participate, and if so, where does the money go?

I will soon be summarising my findings from the literature, but in the meantime I would be very grateful to hear from anyone who has experience of F2F learning in a MOOC. I would love to get comments on any or all of the questions above – and any further questions you think I should have asked.

Thank you in advance – I’m looking forward to your comments!

References

Alario-Hoyos, C., Perez-Sanagustin, M., Delgado-Kloos, C., Parada G., H. A., & Munoz-Organero, M. (2014). Delving into participants’ profiles and use of social tools in MOOCs. IEEE Transactions on Learning Technologies, 7(3), 260–266. https://doi.org/10.1109/TLT.2014.2311807

Brinton, C. G., Chiang, M., Jain, S., Lam, H., Liu, Z., & Wong, F. M. F. (2014). Learning about social learning in MOOCs: From statistical analysis to generative model. IEEE Transactions on Learning Technologies, 7(4), 346–359. https://doi.org/10.1109/TLT.2014.2337900

Firmansyah, M., & Timmis, S. (2016). Making MOOCs meaningful and locally relevant? Investigating IDCourserians—an independent, collaborative, community hub in Indonesia. Research and Practice in Technology Enhanced Learning, 11(11), 1–23. https://doi.org/10.1186/s41039-016-0032-6

Milligan, S. (2015). Crowd-sourced learning in MOOCs. In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge – LAK ’15 (pp. 151–155). New York, New York, USA: ACM Press. https://doi.org/10.1145/2723576.2723596

Norberg, A., Händel, Å., & Ödling, P. (2015). Using MOOCs at learning centers in Northern Sweden. International Review of Research in Open and Distance Learning, 16(6), 137–151. https://doi.org/10.19173/irrodl.v16i6.2035

O’Reilly, O.-M., & Veeramachaneni, K. (2014). Technology for mining the big data of MOOCs. Research and Practice in Assessment, 9(Winter), 29–37. Retrieved from http://eric.ed.gov/?id=EJ1062785

Oura, H., Anzai, Y., Fushikida, W., & Yamauchi, Y. (2015). “What Would Experts Say About This?” An Analysis of Student Interactions outside MOOC Platform Methods Post-survey procedure Post-interview procedure Acknowledgments. In O. Lindwall, P. Häkkinen, T. Koschmann, P. Tchounikine, & S. Ludvigsen (Eds.), Exploring the Material Conditions of Learning: The Computer Supported Collaborative Learning (CSCL) Conference 2015, Volume 2 (pp. 711–712). Gothenberg: International Society of the Learning Sciences. Retrieved from https://www.isls.org/cscl2015/papers/CSCL2015ProceedingsVolume2.pdf

Perna, L. W., Ruby, A., Boruch, R. F., Wang, N., Scull, J., Ahmad, S., & Evans, C. (2014). Moving through MOOCs: Understanding the Progression of Users in Massive Open Online Courses. Educational Researcher, 43(9), 421–432. https://doi.org/10.3102/0013189X14562423

Sanchez-Gordon, S., & Luján-Mora, S. (2016). Accessible blended learning for non-native speakers using MOOCs. In Proceedings of 2015 International Conference on Interactive Collaborative and Blended Learning, ICBL 2015 (pp. 19–24). Mexico City. https://doi.org/10.1109/ICBL.2015.7387645

Posted in mooc, social learning | Tagged | 10 Comments

Three degrees of innovation in designing a new distance programme

The FUTURA report that I co-wrote with Brenda Padilla, Lourdes  Guàrdia and Cris Girona was recently published by the Open University of Catalonia’s eLearn Center. It documents current and emerging practices in online teaching in higher education. We analysed over 100 initiatives from a wide range of higher education institutions which were seen as being innovative in their respective contexts. A thematic analysis of descriptions of these initiatives indicated that there were five underlying themes running through them, which could be described under the headings Intelligent, Distributed, Engaging, Agile and Situated (or IDEAS for short). The key aspects of each of these themes are outlined in the image below:

futura-IDEAS.png

Source: Witthaus, Padilla, Guàrdia and Campillo (2016, p.6). Image available at: https://magic.piktochart.com/output/14507466-futura-ideas

One of the purposes of the FUTURA report was to inspire leaders and academics in higher education to consider new possibilities for the ways in which they provide online programmes. Of course, what is a new possibility for one institution or department may be old hat for another – innovation is a relative concept that is meaningless without knowing the context in which it is enacted. With this in mind, I would like to look at how the IDEAS model could be used to inform the design of a new distance learning programme in the specific context of a typical mainstream university that is primarily campus-based, with little or no experience of offering distance learning programmes.

In the table below, I give indicative examples of three degrees of innovation in the design of the new programme in such an institution, in all five categories of the IDEAS model.   The ‘first degree of innovation’ represents existing practices of face-to-face delivery being merely tweaked for online delivery (in this case, the only innovation is the shift in delivery mode from face-to-face to online); the ‘second degree’ represents something genuinely new being created for the purpose of enhancing or improving the learning experience for distance learners; and the ‘third degree’ represents a radical departure from mainstream practice at the institution. The degree of innovation in each case is described in relative terms, with traditional campus-based delivery as the norm for the institution I have in mind.

Intelligent pedagogy

First degree of innovation Second degree of innovation Third degree of innovation
Using the existing institutional VLE, upload PDFs and provide video recordings for students of f2f lectures, offer online multiple-choice quizzes, and use discussion forums for students to ask questions about the course or the assessment. Design new, online activities specifically for distance learners using the full functionality of the VLE, including discussion forum, blogs, wikis and webinars (e.g. Salmon’s e-tivities as developed at the University of Leicester). Abandon the VLE altogether and set up a new online learning ecosystem using open source, Web-based, mobile-friendly educational apps (see Merriman et al’s UOC/MIT report on Next Generation Learning Architecture).

Distributed pedagogy

First degree of innovation Second degree of innovation Third degree of innovation
Build on any existing partnerships with other higher education institutions, as well as with public and community organisations, that can offer a greater international dimension to the curriculum, and can enrich the learning experience for students in some way.

 

 

 

Structure the curriculum in such a way that students are required to choose at least one elective module from elsewhere in your institution, or from another institution altogether. Encourage students to look for suitable, credit-bearing MOOCs as electives to complement the modules provided by your programme team (e.g. Open University Netherlands, as described in Witthaus et al., 2016, p.16). Collaborate with other HE institutions to recognise non-formal, open learning and award students full credentials (such as degrees) for programmes in which many or most of the modules were studied at other institutions (e.g. the OERu). This may involve disaggregating the services your department or institution provides, i.e. separating out the course content provision, teaching, tutorial support, pastoral support, assessment and credentialling activities.

Engaging pedagogy

First degree of innovation Second degree of innovation Third degree of innovation
Offer an online version of the f2f induction week for distance learners, with ‘incentives’ (prizes) for participation. In this induction, include a walk-through of the VLE and library resources, a link to the institution’s study skills support portal, as well as a quiz on key information provided in the programme handbook. Storyboard each module of the programme in the design process to ensure that there is clear alignment between intended learning outcomes, assessment and learning activities. Within this framework, build online activities (e-tivities) that encourage learners to engage positively at a behavioural, emotional and cognitive level (Trowler, 2010).

 

Build in social tasks that require learners to solve complex problems collaboratively or to complete projects in cooperation with others. Include elements of gamification in the design of these tasks if appropriate (e.g. TU Delft – see Iosup & Ipema, 2013).

 

 

Agile pedagogy

First degree of innovation Second degree of innovation Third degree of innovation
Provide flexibility for distance learners by having an open-ended timetable for each module, with only one deadline at the end for assignment submission or examination. Provide a personal tutor to support each student via email or Skype, stipulating a maximum number of hours’ support per student per module.

 

Design collaborative learning activities that can be completed at a flexible pace but with a clear structure and regular milestones/ deadlines.

Expand the options for recognition of prior learning, for example by giving students Challenge Exams, or inviting students to submit e-portfolios (e.g. Athabasca University – see Spencer, n.d.).

Allow learners to create personalised pathways by combining modules in different ways and progressing through the curriculum at their own pace, through a series of competency-based assessments (e.g. Capella University’s FlexPath model).

 

 

Situated pedagogy

First degree of innovation Second degree of innovation Third degree of innovation
Ensure that all module learning outcomes, prescribed readings and assessment tasks include a focus on contextualising the learning in a range of real-world situations.

 

 

Seek opportunities for virtual placements for distance learners, mirroring the kinds of placements that f2f students undertake (e.g. the EU VALS project, in which computer programming students created code for businesses). Set up public, online platforms for students to collaborate with other students outside of the institution, employers or industry bodies, and community groups in practical applications of their learning (e.g. SustainabilityConnect at Arizona State University).

 

While in the above tables, only one example was given for each category of the IDEAS model, a great many more possible examples and scenarios would need to be considered in the case of a real planning exercise. I hope that I have shown how the IDEAS model could be used in a workshop or brainstorming process, to inspire planning for meaningful innovation in the case of a traditional institution embarking on the development of a new distance learning programme.

References

Iosup, A. & Epema, D. (2013). An Experience Report on Using Gamification in Technical Higher Education. Retrieved from http://www.ds.ewi.tudelft.nl/~iosup/gamification-highereducation14sigcse_sub.pdf

Merriman, J., Coppeto, T., Santanach, F., Shaw, C. and Aracil, X. (2016). Next Generation Learning Architecture. Barcelona: Universitat Oberta de Catalunya. Retrieved from http://openaccess.uoc.edu/webapps/o2/handle/10609/47481

Salmon, G. (2013). E-tivities: The key to active online learning (2nd ed.). London and New York: Routledge.

Spencer, B. (n.d.). Defining prior learning assessment and recognition. Athabasca University. Retrieved from: http://priorlearning.athabascau.ca/what-is-plar.php.

Trowler, V. (2010). Student Engagement Literature Review. York: Higher Educational Academy. Retrieved from: https://www.heacademy.ac.uk/sites/default/files/studentengagementliteraturereview_1.pdf

Witthaus, G., Padilla, B.C., Guàrdia, L. and Campillo, C. (2016). Next Generation Pedagogy: IDEAS for Online and Blended Higher Education. Final report of the FUTURA (Future of University Teaching: Update and a Roadmap for Advancement) project. Universitat Oberta de Catalunya. Retrieved from: http://hdl.handle.net/10609/51441

Posted in learning design | Tagged , , | 3 Comments

Lecture capture: what can we learn from the research?

This week I presented at the Chartered Association of Business Schools Learning, Teaching and Student Experience 2016 conference in Birmingham. My talk was on the literature review I had done with Carol Robinson at Loughborough University to find out more about how Lecture Capture is being used by Higher Education institutions around the world, and what impact, if any, it is having on student learning.

My slides are at slideshare.net/witthaus/lecture-capture-what-can-we-learn-from-the-recent-literature, and the original paper is at tinyurl.com/lecture-capture-lboro (opens as a Word doc). Because the original paper was published under a Creative Commons (CC-BY-SA) licence, I will include the Executive Summary in this blog post. But first I’ll give a brief summary of some of the responses to my presentation from participants in the audience.

  • Claire Hoy, from Sunderland University, shared some fascinating findings from her PhD research that had included looking at the emotions of students and lecturers when using lecture capture. I’m looking forward to reading publications from her on this topic.
  • At lunch after the presentation, I was delighted to meet one of the contributors to our literature review, Prof. Caroline Elliott, whose investigation into the use of lecture capture at Lancaster University was referred to in our study.
  • One person said that a very popular lecturer at their institution had begun using lecture capture, and there was a dramatic drop in attendance at his lectures. His lecture style had also become more stilted, as he had to stand in one fixed position throughout the lecture. In this instance, it seemed that lecture capture had not added value to the learning experience for students. Others in the audience commented that they had not experienced any significant drop in attendance, and that it was unnecessary for the lecturer to remain within sight of the camera, as long as the audio was clear and the slides were visible.
  • There was one particularly thought-provoking comment from an academic who said that some of his students, particularly international learners, had developed a study style that involved rote-learning from lecture recordings, transcribing the lecture word-for-word and then regurgitating that text back in exams or assignments. There was a brief discussion about the need to include study skills support for these kinds of students.

The executive summary of the full literature review follows. I will be interested in any comments from others who have looked into the use of lecture capture in their own or others’ teaching in Higher Education.

Executive Summary

This report was written for the Centre for Academic Practice at Loughborough University in order to provide a snapshot of how lecture capture (LC) is currently being used in higher education. It draws from literature published internationally between 2012 and mid-2015. The aim was not to provide an exhaustive review of the relevant literature, but rather to provide indicative findings that could inform day-to-day practice in a higher education institution.

It should be noted that, in many of the studies reviewed, data was gathered by self-reporting of students, and not all students responded to surveys, therefore providing only a partial picture. Also, most of the studies were conducted within specific programmes, mostly within the STEM subjects, and so we should be cautious about making generalisations from the findings.

The first finding was that there appeared to be a wide range in the percentage of students who used LC, from as low as 21% to as high as 100% of cohort members. A few studies found that usage increased when the LC recordings were “enriched” with additional online materials, and some found increased usage when LC was made available in formats that lent themselves to mobile access. There were some examples of different usage patterns across different years of study, with first-year students either watching more LCs than students in later years (possibly due to the novelty effect of the technology) or watching less (possibly because they had not yet settled into a “serious” study routine). Very little information was found in the literature regarding advice given to students about the use of LC by lecturers.

There appears to be a great deal of variety in the manner in which students use LC. LC is most often used for revision and note taking. Almost all respondents claimed to use LC as a supplement to, rather than a replacement for, lectures. Most students use LC selectively, choosing specific sections of videos to watch. A small number of students watch entire LC recordings – these are often (but not always) speakers of English as a second or foreign language or students with learning difficulties such as dyslexia. There is tentative evidence from one study to show that LC is most efficient for learners when the recording contains only the slides and the lecturer’s voiceover (without the video of the lecturer) – this is because the video of the presenter reduces the amount of space available for the slides.

With regard to whether LC has any impact on student learning, the findings here are varied. Some studies found little or no evidence of any impact. Two examples were found of the provision of LC apparently having a negative impact on a minority of students: in these cases, students who used LC as a substitute for attendance at lectures were found to be at a severe disadvantage in terms of their final marks; moreover, those students who attended very few live lectures did not close the gap by watching more LC online. In one study it was found that the quality of student interaction in class dropped when LC was introduced, as students were reluctant to speak up when being recorded. By contrast, in another case it was reported that students’ contributions in class were of a better quality when the class was being recorded.

There were several situations in which a positive relationship was found between the use of LC and student learning outcomes. Students perceive the greatest value for LC in courses that move quickly, rely heavily on lectures, and for which the information provided via lectures is not readily available from any other sources, as well as courses which emphasise the assimilation of information rather than the development of applied skills (an important distinction in medicine and related subjects, where many of these studies took place). LC was also found to have a positive effect when the teacher used it as a tool to “flip” the classroom and asked students to view the LC before coming to class. In several of the studies, students who were non-native speakers of English emphasised the value of LC to them, and this sentiment was echoed by learners with dyslexia or other learning disabilities.

A positive relationship was also identified between learners who used LC and certain approaches to learning. In one paper (Brooks et al, 2014), learners were categorised according to their usage patterns (i.e. how often they viewed the LCs, and at which points in the semester), and it was found that students categorised as “High Activity” outperformed their peers by up to 16.45%, while students in other clusters obtained more or less the same grades as each other. Other studies concluded that there was a positive effect only for those students who use LC as a supplement to regular lecture attendance, and that LC appeared to be correlated with “deep” learning as opposed to “surface” learning.

The overwhelming majority students, when asked, say they do not view recorded lectures as a replacement for attending live lectures. This finding was borne out by several studies which included evidence from analytics on lecture attendance and LC views. In one case, increased attendance at live lectures was reported, on the basis that learners felt more confident about their grasp of the subject matter from having viewed the LCs. However, in several studies, lower attendance at live lectures was found to be a direct result of implementing LC. There is some discussion in the literature about contributing factors here, especially around the notion that learners who skip lectures tend to be “surface learners” (as opposed to “deep learners”, and that these learners do not generally compensate for missing lectures by watching the LC.

A few examples were found of lecturers changing their teaching style as a result of the introduction of LC. These generally revolved around the concept of the so-called flipped classroom (teachers providing lecture content for students to read or view before coming to class, and changing their teaching style towards more active, learner-centred learning in the classroom). Other opportunities for innovating in teaching related to the use of LC by lecturers for reflection on their teaching style, and the creation of additional materials to support learners’ independent learning from LC.

A few further points arose out of the literature that are worth highlighting. It is clear from the comments made by students throughout the literature that the provision of LC is perceived as strongly enhancing their learning experience. There is evidence from one study that if LC is mentioned as being an integral part of the learning and teaching approach in marketing brochures or on programme websites, it may influence students’ choice of programme – or even institution to study at. One study also found that LC was particularly useful for students on work placements.

Certain recommendations arose out of the literature – sometimes implicitly. For example, there is a gap in the literature regarding the nature of the advice given by lecturers to students. This might be especially important in the case of first year students who seem to be less consistent in their viewing patterns. Guidance given by lecturers to learners as to how to make effective use of LC may help here. In addition, at-risk students can be identified through a combination of tracking views on the LC system and tracking attendance in class, and automated alerts could be sent to them with advice on recommended behavioural changes, or information about support mechanisms available.

Another important consideration for institutions is the growth in mobile access to LC by students, which suggests that institutional platforms and tools used to deliver LC to learners need to be mobile-friendly.

The paper concludes with responses from the literature to a number of statements from academics in an earlier survey at Loughborough University, where concerns were expressed about the use of LC. It is clear that lecturers need to be supported in the adoption and implementation of LC – not just from a technological point of view but also in terms of their questions about potential copyright infringement, their worries about the potential drop in attendance if LC is introduced, and any other concerns they have about the possible impact of LC on the learning and teaching experience. For LC to have the greatest possible positive impact on learning for students, lecturers, managers and support staff need to jointly create a learning environment that is conducive to effective use of LC and that limits the risks.

For the full paper, see tinyurl.com/lecture-capture-lboro.

Posted in blended learning | Tagged , | 3 Comments

Validation of Non-formal MOOC-based Learning

The report I co-authored for the EU, “Validation of Non-formal MOOC-based Learning: an Analysis of Assessment and Recognition Practices in Open Education” has at last been published. The study, referred to as OpenCred, began in May 2014, and originally aimed to find examples of recognition of open learning, for example, learning based on open educational resources or massive open online courses (MOOCs). A rework of the original draft was undertaken in late 2015 with substantial input from Anne-Christine Tannhäuser, in response to feedback from reviewers that the word “recognition” needed to be clarified in the report.

Some people use the term “recognition” rather loosely to refer to any form of credentialisation or certification awarded to a learner at any point in their learning journey, while others, including specialists in the field of recognition of prior learning, use the term specifically to refer to the process of validation of credentials by an educational institution or employer. This latter meaning implies a two-stage process – credentialisation, followed by recognition, usually by a different body, at a later stage. To avoid ambiguity in the OpenCred report, we distinguished between credentialisation and recognition as follows:

Credentialisation versus recognition of learning outcomes (Witthaus, Inamorato dos Santos, Childs, Tannhäuser, Conole, Nkuyubwatsi and Punie, 2016, p.6)

Credentialisation versus recognition of learning outcomes (Witthaus, Inamorato dos Santos, Childs, Tannhäuser, Conole, Nkuyubwatsi and Punie, 2016, p.6)

One of the main outcomes of the study was a model which describes elements of non-formal, open learning assessment using a “traffic light” metaphor. In this model, a MOOC can be analysed in terms of the extent to which the following six characteristics are present:

  • Suitable, supervised assessment
  • Identity verification of the learner during assessment
  • Partnership and collaboration with other institutions
  • Award of credit points to learners
  • Quality assurance mechanisms
  • Informative certificates acknowledging specific learning achievements
Open Learning Traffic Light Model (Witthaus, Inamorato dos Santos, Childs, Tannhäuser, Conole, Nkuyubwatsi and Punie, 2016, p.6)

Open Learning Traffic Light Model (Witthaus, Inamorato dos Santos, Childs, Tannhäuser, Conole, Nkuyubwatsi and Punie, 2016, p.6)

The green rim of the hexagon indicates strong presence of each of the elements; the yellow layer indicates some presence, and the red inner core indicates little or no presence. Learners are in a better position to obtain recognition for their MOOC-based learning if the MOOC has all six elements in the green rim. For example, a MOOC learner who travels to a physical location to sit an invigilated exam, where their identity is verified, and who is awarded ECTS credits upon passing the exam, is already in a strong position to have those credits validated at a later stage. If the certificate they receive contains detailed information about the course contents and assessment procedures, all the better. If, in addition, the MOOC provider is known to other organisations in the sector through partnership and collaboration in professional networks, and if the quality assurance procedures used by the MOOC provider are transparent, it would be very difficult for another institution to justify not validating the learning.

It is hoped that the report will be of use to both MOOC providers and institutions/ employers that have to recognise prior learning, and will ultimately enable open learners to receive meaningful, life-changing acknowledgement of their learning achievements.

This post was edited on 4 April 2016 with a fuller description of the writing and editing process of the OpenCred report.

Reference

Witthaus, G., Inamorato dos Santos. A., Childs, M., Tannhäuser, A., Conole, G., Nkuyubwatsi, B., Punie, Y. (2016) Validation of Non-formal MOOC-based Learning: An Analysis of Assessment and Recognition Practices in Europe (OpenCred). EUR 27660 EN; doi:10.2791/809371. Available at: http://publications.jrc.ec.europa.eu/repository/bitstream/JRC96968/lfna27660enn.pdf

Posted in open education | Tagged , , | 1 Comment

My top ten research utilities this week

This is Week 1 of my PhD in Higher Education: Research, Evaluation & Enhancement through Lancaster University, and to get us started, Paul Trowler has asked us to share the top ten research utilities we’ve used in the past week. Paul’s list is great – I didn’t know about Research into Higher Education Abstracts, ABBYY TextGrabber which apparently takes photos, scans and converts to editable text (really looking forward to trying that out!) or Fastscanner, which enables you to take photos on your phone and store them as PDFs.  So those three go at the top of my list of new tools to try.

Things I’ve used this week include:

Trello – I use this app every day for all my to-do lists, and also for lists of articles to read or videos to view. I can update it on my laptop or my mobile phone, so it enables me to do a lot of work planning while on buses and trains. I find it very comforting having lots of lists… (I’m just waiting for the app that carries out all the actions on those lists!)

Google Docs (and Spreadsheets) – brilliant for collaborating with colleagues on research and writing projects

Google Hangouts for meeting with people in other places (This morning I met with Brenda Padilla who was in Copenhagen, and colleagues from the Open University of Catalonia in Barcelona. While we were online together, we collaboratively edited a number of Google Docs.)

Google Forms – a useful survey tool which lives in Google Drive, along with Google Docs and spreadsheets

Twitter – for finding and sharing links to interesting articles, blog posts and thoughts (I’m @twitthaus – when I got my twitter name I totally entered into the spirit of it…)

Paul mentions Academia.edu, which I also find hugely useful. In addition, I am also on ResearchGate. These have become essential literature search tools for me, along with Google Scholar and Mendeley (an alternative to Zotero, which enables me to store bibliographic data about everything I find, as well to upload the articles in one place and to highlight and annotate them online – and to share “libraries” with colleagues).

Finally, one more thing I would have used this week if it only existed… is an A4-sized e-book reader for PDFs. It would need to link to Mendeley in order to be really useful. (There are promising signs that Onyx is developing such a device so I’m watching this space…)

That’s it for now. Thanks, Paul, for getting me thinking, and I’m looking forward to hearing from others about their top ten research utilities.

Posted in Research | Tagged , | 5 Comments