Online Learning Lab Report 2014

1 - First Report Online Learning Lab

Online Learning Lab Report 2014 Summarizing Leiden University’s experience in offering new online learning programmes

2 - First Report Online Learning Lab

Innovating the way we teach and learn

3 - First Report Online Learning Lab

Online Learning Lab Report 2014 Hier wordt ge誰nvesteerd in uw toekomst. Dit project wordt mede mogelijk gemaakt door het Europees Fonds voor Regionale Ontwikkeling.

Table of Contents

Executive summary 5 Foreword by Vice Rector Magnificus prof. dr. Simone Buitendijk 7 Introduction 9 Acknowledgements 11 Background 12 Objectives MOOC development process 12 Exploring Online Education 16 Selection criteria 16 Course development 16 Course design 12 Platforms 20 Democratization of education: Reach and Engagement in Leiden MOOCs 27 Demographics 27 Education and professional background 30 Engagement 32 Aspirations 35 Impact on on-campus education 39 Platform for research 40 Collaborations 42 Reflections and recommendations 43 Online Learning Lab Structure 44 Centre for innovation online learning team 44 Research network 45 Appendix: Course descriptives 46

6 - First Report Online Learning Lab

7 - First Report Online Learning Lab

Executive summary This report presents the result of the Online Learning Lab providing a quantitative and qualitative analysis of the Leiden MOOCs and online initiatives from November 2012 to October 2014. So far: • Over 200.000 participants from 186 different countries enrolled in 5 Leiden Massive Open Online Courses (MOOCs) (approx. 10.000 per month).

• More than 750.000 videos were watched.

• Over 20.000 posts and more than 10.000 comments were made on the Leiden MOOC social media forums.

• More than 11.000 exams were submitted and over 8.000 Statements of Accomplishment were rewarded.

• 97% of the learners had a ‘good’ to ‘excellent’ experience.

• Conducted innovative experiments by launching Small Private Online Course (SPOC) and using the MOOCs as Books, exploring how to use MOOCs in regular education.

• Due to the MOOCs Leiden University reinforces relations with LERU1 partners and other universities.

General findings:

• MOOCs are an accelerator for learning and teaching innovation at scale. They are an important tool bringing new learning technologies into the classroom and the home.

• MOOCs are a new means of education. They are not a substitution of regular education, but can add value.

• MOOCs create global communities of people who share a mutual interest in a scientific topic/theme.

• MOOCs can be highly interactive through forums and social media.

• MOOC technology does boost innovation in blended learning.

• MOOCs are still in an experimental stage. It is not yet clear which (more) objectives they can achieve, and what their long-term role will be.

Recommendations:

• Continue the exploration and innovation of online learning and its possibilities for as well the online as the blended learning offer from Leiden University.

• Explore development and offering of pre-masters, full online masters or blended masters and the market for lifelong learning to tap into an international growing market, and consider offering the first course as a MOOC. • Use MOOCs as a platform for domain research integrated with online learning, easy access to active contributing professionals and to respondents across many different countries.

• Integrate action research into all pilots to ensure continuous learning.

• Keep exploring how MOOCs and SPOCs can contribute to the university strategy and to individual faculties’ strategic goals.

• Keep on developing and professionalizing the production team for online education (MOOCs and SPOCs) which functions for the entire university, with focus on the integration of content, video production and pedagogy to develop engaging learning experiences.

• Further develop services in online learning, didactics and production for campus education that can be used by all faculties in the university. Work together and share experiences with faculty support, towards an integrated offer of services for all faculty.

• Continue to experiment with modular content development, to enhance possibilities for more personalized learning routes and for different learning arrangements with the same content.

• Continue exploration and experimentation with business models for Online Learning and MOOCs.

1 The League of European Research Universities (LERU) is an association of 21 leading research-intensive universities that share the values of high-quality teaching within an environment of internationally competitive research. See http://www.leru.org

8 - First Report Online Learning Lab

9 - First Report Online Learning Lab

Foreword

by Vice Rector Magnificus prof. dr. Simone Buitendijk This first report of the Online Learning Lab presents the results of the online educational explorations at Leiden University. After one year of work the lab has offered our university many valuable insights into how educational technologies can impact our institution. Our Massive Open Online Courses (MOOCs), for example, have shown to spark creativity and passion among our faculty for enhancing the learning experience, they offer our courses to a global audience and they provide new possibilities in research and teaching for our on-campus students and our researchers.

This is just the beginning. Digitization of higher education is changing how we learn and how we teach.

It is the ambition of Leiden University to be amongst those institutions that shape the future of higher education by actively experimenting and evaluating what works and what doesn’t, in close collaboration with our national and international partners. IN the coming years, Leiden University will invest 1,4 Mln€ in further development of online learning activities through MOOCs, SPOCs, flipped classroom, online masters and online modules for lifelong learners.

I very much look forward to the outcomes of this next phase of online learning innovation at Leiden University.

10 - First Report Online Learning Lab

11 - First Report Online Learning Lab

Introduction This is the first generation of people that work, play, think and learn differently than their parents.

They are the first generation to not be afraid of technology. It’s like the air to them.”

– Don Tapscott – The new generation of students hold a natural and positive attitude towards new technologies, taking them for granted as part of their lives. Communication technologies have become ‘extensions’ of our mental and physical being. As new generations grow up in a time where technology is interwoven with their daily activities, it is to be expected that students will demand for a more integrated part of technology within their education. To cope with these changes and to be prepared for the future it is important to keep innovating, also within the academic field.

The worldwide demand for teachers in higher education will grow from 97 million in 2000, to 262 million by 2025. In India alone, there are 300.000 more lecturers needed to cope with the amount of students2.

To deal with such global challenges, new technological developments might offer solutions. Open educational resources and open online courses, such as MOOCs, provide revolutionary access to education worldwide and aim to improve the learning and teaching experiences of both teachers and students.

Moreover, online courses can be used in research as well, to learn how we can enhance the impact of scientific research through education.

As the information society comes of age, learning is no longer confined to a specific period of one’s life.

Today people are required to adapt to new environments by adapting and adopting new skills and gaining knowledge on the go.

The Online Learning Lab explores how online learning developments can contribute to strengthen both Leiden University’s education and research, its potential to democratize education globally and the ways in which online learning environments can be used for the creation of new knowledge. Over the past two years more than 200.000 people registered in a Leiden University MOOC from over 180 countries. It is important to note that the future of online learning is just beginning and by exploring and harnessing the possibilities presented by rapidly developing information technologies, we want to be at the forefront of innovating education. We hope this report contributes to the debate about the way in which education can be enhanced through technology and we invite you to join us in our quest to keep on innovating in academia by disseminating information and knowledge to all corners of the globe.

Marja Verstelle & Gideon Shimshon

Co-founders of Leiden University’s Online Learning Lab @ Centre for Innovation The Hague

2 Link: http://www.universityworldnews.com/article.php?story=20110812092917805 (last accessed 22/11/14)

12 - First Report Online Learning Lab

13 - First Report Online Learning Lab

Acknowledgements This report is a collective achievement of a large number of colleagues who spent massive amounts of time developing methods, tools and courses.

Without their enthusiasm, support and ideas all this would not have been possible.

Online course teams The Law of the European Union: An Introduction (MOOC) Prof. dr. Stefaan Van den Bogaert, Dr. Darinka Piqani, Sjef Janssen, Stephan van der Raad and colleagues from the EU Law department

Terrorism and Counterterrorism: Comparing Theory and Practice (MOOC) Prof. dr. Edwin Bakker, Jeanine de Roy van Zuijdewijn The Changing Global Order (MOOC) Prof. dr. Madeleine Hosli, Prof. dr. André Gerrits, Prof. mr. Larissa Van den Herik, Prof dr. Giles ScottSmith, Dr. Siniŝa Vuković, Prof. dr. Rob de Wijk, Jeanine de Roy van Zuijdewijn Configuring the World: A Critical Political Economy Approach (MOOC) Prof. dr. Richard Griffiths, Einat Shitrit Wheels of Metals: Urban Mining for a Circular Economy (MOOC) Dr. Ester van der Voet, drs. Ruben Heule, Miranda Verboon, Wesley Crock Sharia in the West (SPOC) Prof. dr. M. Berger, Rianne Bouwman IP knowledge clips Prof. dr. Dirk Visser and his team of students

ICTO Program steering committee Prof. dr. Simone Buitendijk, prof. dr. Jouke de Vries, prof. dr. Rick Lawson, Jolanda Riel, prof. dr. Han de Winde, prof. dr. Joost Kok, prof. dr. Henk Dekker, prof. dr. Paul Nieuwenburg, Hedwig Darley, Jan-Willem Brock, Kurt de Belder, Jeanette Vonk, Hans van Dommele, Céril van Leeuwen, Kimberley Innemee Centre for Innovation – Online Learning Lab Thomas Hurkxkens, Jeanine de Roy van Zuijdewijn, Tanja de Bie, Maarten van der Ven, Jesse Bruins, Jan Porth, Markolf von Ketelhodt, Janne Polman, Joeri Schasfoort, Arlo Laibowitz, Jacqueline van Zoggel, Nadia Kreeft (UBL), Alexander Mouret (LURIS), Damiaan van Eeten (UBL), Janneke van Hemmen, Robin de Lange, Rosen Bogdanov, Kasper Gossink, Roy van Rooijen, Ivo de Nooijer (LURIS), Willem de Wit (Legal Affairs), Danny Damen, Bernard Nauta Our community TA’s A special thanks to all the volunteers who help us with the community moderation. It’s a pleasure working with such a great group of dedicated and knowledgeable people.

Coursera support team Daphne Koller, Pang Wei Koh, Emma Webb, Clara Ng, Mark Pan, Melanie Lei, Vivek Goel

14 - First Report Online Learning Lab

Background We started in 2012 with a scenario building exercise of the future of education, focusing on the impact of technology on the way we learn and teach. Based on this analysis and the enthusiasm generated, concrete steps could be taken to develop a pilot project MOOC with Coursera as a partner. Since then, four more MOOCs have been created and a number of projects have been started around the use of MOOCs in on campus education e.g. using MOOCs as BOOKs, Flipping the Classroom and the setting up of Small Private Online Courses (SPOCS). The development of new online learning activities at the Centre for Innovation The Hague has resulted in the establishment of the Online Learning Lab at The Campus The Hague.

Its aims are to explore possibilities of innovating education, contribute to the opening up of education to a global audience and find ways to improve on-campus education through new educational technologies. So far the lab has developed a range of services using new and innovative education methods, which are accessible for lecturers and professors from Leiden University as a whole, creating a place where new ideas can be brought to reality. For MOOCs, all video lectures are open for use by third parties under creative commons license (open educational resources) through www.coursera.org/leiden and http://leidenuniv.onlinelearninglab.org.

Objectives MOOC development process During the starting phase of MOOC production a number of objectives were identified that have guided the way MOOCs and additional projects were developed, listed below: Exploring Online education: Explore the possibilities of new learning and teaching methods using innovative educational technologies. Build expertise

in technology driven didactic models, online course development, engagement, online community building and online testing.

Democratization of education: Provide the best content for free to everyone with a good internet connection across the globe. This has tremendous potential positive impact to citizens around the world with a need for quality higher education.

On-campus education: Start new conversations around teaching and learning and rethinking course design through new technologies. For example by developing Small Private Online Courses (SPOCs) and blending the online content for our on campus students. And also fuelling the debate on open, online and blended learning through presentations and meetings.

Platform for research: MOOCs provide opportunities for new research. On the one hand within MOOCs one can look at the data generated by thousands of students and using learning analytics find ways to understand learning better. One can now use randomized testing of control groups to investigate impact of teaching approaches in a ‘virtual lab’. On the other hand platforms such as Coursera offer the possibility to do research through MOOCs using citizen science to provide, classify and analyse large amounts of information crowd sourced through the MOOC. Sustainability of development: MOOCs are expensive to produce and will need to find ways to become sustainable models. The goal here is to explore what possible business models could be put in place that would enable funding MOOC production and delivery in the future.

15 - First Report Online Learning Lab

Below an overview of the courses currently offered through the Coursera platform: Table 1 - To date the following courses have been or soon will be running on the Coursera platform MOOC

Discipline

Lead lecturer

The Law of the European Union

Law

Prof. dr. Stefaan van den Bogaert

Terrorism and Counterterrorism

Social sciences

Prof. dr. Edwin Bakker

The Changing Global Order

Social sciences

Prof. dr. Madeleine Hosli

Configuring the World, a critical political economy approach

Humanities, Social sciences

Prof. dr. Richard Griffiths

Wheels of Metals: Urban Mining for a Circular Economy

Physical, Energy & Earth Sciences

Dr. Ester van der Voet

Specialisation and capstone projects linking multiple MOOCs (start in Q1 2015)

Humanities, Social sciences

Prof. dr. Madeleine Hosli , Prof. dr. Richard Griffiths & Prof. dr.

Gilbert Probst (Geneva University)

Miracles of Human Language: An Introduction to Linguistics (Start in Q 1 2015)

Humanities

Prof. dr. Marc van Oostendorp

There are a number of new courses in development, which will be announced in the coming months.

Based on the MOOCs developed to date a number of new concepts were developed that impact on campus education see table 2 below.

Table 2 – Other initiatives Initiative

Discipline

Lecturer

Small Private Online Courses (SPOC)

Humanities

Prof. dr. Maurits Berger

Use of a MOOC as a Book in class

Various

Various

Short online knowledge clips format

Law

Prof. dr. Dirk Visser

In this report we will mostly focus on the MOOCs that have run thus far and provide some data on the initiatives that came out of the MOOCs.

16 - First Report Online Learning Lab

Faculty/Instructor: Leiden Law School.

Prof. dr. Stefaan van den Bogaert Course: MOOC European Law (Coursera) “The Law of the European Union: An Introduction” was the first MOOC launched in the Netherlands with a global platform. In this five-week course, students explored the functioning of the European Union, the impact of its laws on states, citizens and companies and the current challenges it faces. The ambition of Prof. dr. Stefaan van den Bogaert was to attract a very diverse group of participants. It was designed for students, lawyers, entrepreneurs or anyone with an interest in European Law. The course very successful in attracting this diverse audience. Participants ranged from 16-years old high school students to judges of 85 years old. The discussion forum was a central aspect of this course. Prof. van den Bogaert said “he could never have imagined before that I would actually witness a student from Nepal discussing an EU law issue with a student from Madagascar and a student from Bolivia online. That is actually what happened and for a teacher this is a priceless experience.” This MOOC attracted more than 40,000 participants in its first offering.

Part of that work involved understanding how law is formed in the European Union, and how it relates to the laws adopted in member states. The knowledge I gained in the Leiden MOOC was essential to successful performance of my work at the UN.” - Course participant

17 - First Report Online Learning Lab

18 - First Report Online Learning Lab

Exploring Online Education Exploring the possibilities of new learning and teaching methods using innovative educational technologies required a learning-by-doing approach. This resulted in 1) criteria for MOOC proposal selection 2) a course development process which was agile and lean and 3) a different course design for every project to create maximum learning on elements such as online community building, online testing and grading, and developing (interactive) video lecture formats.

Selection criteria The first MOOC was selected as part of a set of informal discussions because of two key factors: the lecturer was enthusiastic about the idea, and the topic seemed to appeal to a large and broad audience.

After the first MOOC it was important to put in place for a more formal structure for selection. A call was written out to all faculties inviting all professors to come up with ideas for a MOOC based on the criteria in table 3.

Course development In 2012 when development started there was a lack of MOOC development ‘best practices’. There were initial practices from partner Universities but everyone was figuring out how a MOOC production should work and what works best for the learners. It was therefore important to develop a process that would allow continuously improving the way participants learn. At the same time the process had to deliver high quality courses (see figure 1 below for a high level process description). Development of MOOCs also required a process that would fuel innovation and allow flexible use of resources. To achieve this we implemented a methodology called agile project management which is an iterative and incremental method of managing the design and build activities in a highly flexible and interactive manner. This allowed for a continuous evaluation of intermediate results in order to implement changes to the approach as the project progressed based on new insights.

Table 3 – criteria for MOOC proposal selection

a. Improves reputation of the University b. Links to the strengths of the university in terms of research and education c. Speaks to a potentially large audience and is not competing with other MOOCs on Coursera d. Lecturer is able to bring over content with passion e. Innovation: explores new ways of learning and teaching and aspects relevant for on campus education f. Can be used in courses of Leiden University and also for example by other LERU partners g. MOOC content needs to be independent from 3d party influences when working with 3d parties h. There is a balance in the portfolio of MOOCs with regards to the disciplines and gender that are represented

19 - First Report Online Learning Lab

Innovate

Produce

• Develop pedagogy • Design student experience • Design community • Create video approach • Publish course

• Develop Scripts • Create Quizes & Assignments • Produce video’s • Setup course environment • Test course

Initially the focus of MOOC production was on video production. We were interested in bringing out the personal best out of a lecturer in the video lectures.

This meant implementing a tailored, personalized approach, testing what works best with a specific lecturer before moving into standardized production.

For example some lecturers come across better on camera when standing, others can record themselves and there are those who prefer to film on location rather than the studio. The result was that the studio and equipment needed to be fully flexible and tailored to the needs of the teachers.

We also found that because the video aspect is so visible, novel and time consuming, it is easy to fall

Deliver • Run course • Manage interaction • Manage testing Generate grades • Send out certificates

Evaluate • Evaluate course • Define improvements • Provide research data

into the ‘video trap’ where video production becomes leading in the development process. This creates the risk that the learning experience and pedagogy become less explicit and thought through during the production phase. We wanted the pedagogy and learning experience to be leading. This meant taking more time upfront to think through the pedagogy and learning experience flow and structure of the course before any discussion about video takes place.

This approach placed instructional design at the heart of the development process and the video format improves by making sure that the courses become an integrated body (rich experience) of knowledge/ learning for participants.

20 - First Report Online Learning Lab

A third aspect that became important for the development process is solving the constant dilemma in making MOOCs to create one-on-one copies of the courses that we offer at university using new formats which are shorter and modular and perhaps more suited for online learning. This was explored throughout the MOOC production process. Examples where we create a more modular approach to online content development are 1) a set of short knowledge clips and 2) a joint specialization with the University of Geneva. The knowledge clips are short focused information video’s that each deal with very specific topic and can be viewed in different order depending on the need. The specialization track is a way to combine MOOCs and create more depth for the learners by developing integrated capstone projects with the aim to synthesize the content of a number of MOOCs. This is something we need to keep thinking about and develop further in the next iteration of online programmes.

Interaction in an online course is key and the technology offers students various forms of interaction with each other and the lecturer(s). This is facilitated through discussion forums, course wiki, social media and (virtual) meetings. We were interested to create meaningful interaction at scale: How can we engage large groups of students in a digital environment?

Over the past two years a community of volunteers from all around the world has been created that actively help moderate the interaction in the various MOOCs as Community Teaching Assistants (CTA’s).

The Leiden style of community management received international recognition in the MOOC field. The Academic Toolkit & First Timers support area is listed as a best practice at Coursera’s portal for partners.

One aim of community management is to ensure positive engagement between participants and to create a safe learning environment where everybody is free to express him or herself. This is done by finding ways to lower the threshold of posting and participating by creating a welcoming environment, by encouraging learners to participate, and so creating an academic community. When negative behaviour is displayed, the community management team troubleshoots by shutting down, preferably preventively, of deliberate damage and bend a negative curve towards more positive behaviour. Moderation also focuses on increasing the level of interaction and academic approach in the online programmes. For example by pointing out the academic standards of Leiden University e.g. questioning references, pointing out critique to sources, discussing logical fallacies etc.

Characterizing for the community management impact was the initiative of community members to translate the video subtitles. Native volunteers have now translated video subtitles of several Leiden MOOCs to Dutch, Spanish, Portuguese and Serbian.

We want to keep on stimulating these initiatives and target specific languages such as Chinese and Russian.

21 - First Report Online Learning Lab

Data about community activity within MOOCs The Law of the European Union

Terrorism and Counter­ terrorism #1

Terrorism and Counter­ terrorism #2

The Changing Global Order

Configuring the World, a critical political economy approach

1

4

203

8

6

30,2%

51,2%

51,4%

37,7%

20.4%

6.073

13.028

8.025

6.935

3959

Plagiarism

0

51

7

5

5

Rules violation

4

27

5

7

1

Forum bans

0

5

3

3

2

# Wiki pages

9

21

21

7

1

# CTA’s Participation # Posts

Participation: number of learners who accessed the forums as a percentage of learners who accessed the MOOC. Platform doesn’t register the number of total participants in any other way.

Posts: Total posts + comments.

Plagiarism: the number of official reports in the flagged content section. All reports were checked. Some were multiple reports on the same case. Not all cases were judged to be plagiarism. It is thought that plagiarism goes underreported and attention from staff & Coursera to the issue is an influence on the number of cases reported.

Rules violation definition: number of official reports in the flagged content section. Not included are cases dealt with by CTA’s proactively, or complaints made to staff in email, nor have the communities been checked upon missed cases.

Forum ban: a student is no longer allowed to post due to multiple violations, but can still finish the course with a certificate.

Wikipages definition: the total number of pages created in the learner maintained wiki resource project, combined over multiple runs of the course. Thus there is a certain logic to TCT scoring higher than the rest, as there was more than one run.

3 The number of CTA’s at Terrorism2 was larger to prepare for the upcoming MOOCs in that year, so it was used as a training ground. The average of CTA’s is now 8-10 including coordinator.

22 - First Report Online Learning Lab

Course design The learning experience in an online course (as in any course) is the result of a design process with underlying (explicit and implicit) design principles. The

aim was to develop every course with its own specific learning and teaching design. We have listed the various elements in the table 4 below showing some key differences in approach per project.

Table 4 – Course learning and teaching design elements Experience

Design elements

Evidence

The Law of the European Union (EU)

Key elements

This was the first MOOC we developed on Coursera. The focus was on transforming an existing course into an online one. The difficulty of the course ranged from beginner level at the start to fairly high complexity in the last couple of weeks.

Platform

Coursera

Open

High - under creative commons license cc-by-nc-sa

Massive

High - open to all

Multimedia

Low – There was the option to access an online book from oxford university press through Chegg and students used the forum functions of the Coursera platform.

Degree of communication

Low to medium - Participants were encouraged to contribute to a number of key debates on the discussion forum but moderation was limited. In addition the professor provided weekly updates.

The degree of collaboration

Low - focus was on individual learning but the option of knowledge exchange was facilitated through the forum and wiki.

Learning Pathways

Medium - There were three learning pathways pre-defined for course completion, the first was the basic track based on multiple-choice questions and the second a more in depth track adding a peer review assignment. Thirdly, students could also audit the course.

Quality Assurance

Medium - The quality assurance was done through review of materials by production and didactics team and content wise by colleagues from the department.

Reflection

Low - Although the forum provided some elements for reflection this was not a core part of the course design.

Certification

Medium - Certification participants could obtain a regular statement of accomplishment or one with distinction on completion of either the basic or the advanced track. The multiple-choice questions were randomized.

23 - First Report Online Learning Lab

Experience

Terrorism and Counterterrorism (Terr)

Design elements

Evidence

Formal Learning

Low - The course was informal and optional.

Autonomy

Medium - Participants were expected to work individually and take control of their learning, there was little support from teaching assistants or the professor (the course did provide some forum support and weekly reflections from the professor on the issues raised by students).

Diversity

High - The course was open to everyone who was interested and was directed to as well EU as non EU residents.

Key elements

The focus of this MOOC was on community management and the use of a MOOC as a research platform. The course was aimed at interested people in the topic.

Platform

Coursera

Open

High - under creative commons license cc-by-nc-sa

Massive

High - open to all

Multimedia

Medium - The course used a variety of Multimedia applications ranging from the functionalities of the Coursera platform, the use of LinkedIn and Google hangout for student - professor interaction.

Degree of communication

Medium - Participants were encouraged to contribute to a number of key debates on the discussion forum with active moderators, in addition the professor provided weekly updates, organised a live hangout and a physical meet up with participants.

The degree of collaboration

Medium - Community learning and collaboration was encouraged through the forum and the peer reviews. In addition students were actively asked to contribute to the debate and provide their opinions on issues the professor raised.

Learning Pathways

Medium - There were three learning pathways pre-defined for course completion, the first was the basic track based on multiple choice questions and the second a more in depth track adding a peer review assignment to the course completion requirements. Students could also audit the course.

Quality Assurance

Medium - The quality assurance was done through review of materials by production and didactics team and content wise by colleagues from the department.

Reflection

Medium - The forum and in video quizzes provided elements for reflection on the content and discuss students opinions and link them to evidence. The research element was a key part of this effort.

Certification

Medium - Certification participants could obtain a certificate on completion of different aspects of the course.

24 - First Report Online Learning Lab

Experience

The Changing Global Order (CGO)

Design elements

Evidence

Formal Learning

Medium - The course was initially informal and optional and now will be piloted to run parallel with an on campus course using the material of the MOOC as a BOOK and resource for our on campus students.

Autonomy

Medium - Participants were expected to work individually and take control of their learning, there was some support from teaching assistants or the professor.

Diversity

High - The course was open to everyone who was interested and was set up to include as much as possible the perspective from different parts of the world.

Key elements

A differentiator in this MOOC was the number of guest speakers and lecturers that were brought in to provide different perspectives and bring in specific expertise. The difficulty of the course was university entry level.

The course had a strong focus on community building encouraging collaboration and interaction. This MOOC also is part of a specialisation track looking at combining MOOCs and develop a capstone project for more depth.

Platform

Coursera

Open

High - under creative commons license cc-by-nc-sa

Massive

High - open to all

Multimedia

Medium - The course used a variety of Multimedia applications ranging from the functionalities of the Coursera platform and Google hangout for student - professor interaction.

Degree of communication

Medium - Participants were encouraged to contribute to a number of key debates on the discussion forum with active moderators, in addition the professor provided weekly updates, organised a live hangout and a physical meet up with participants.

The degree of collaboration

Medium - Community learning and collaboration was encouraged through the forum and the peer reviews. In addition students were actively asked to contribute to the debates. The professor also organised a contest for students to develop solutions to a problem in groups.

Learning Pathways

High - There were four learning pathways pre-defined for course completion, the first was the basic track based on multiple-choice questions and the second a more in depth track adding a peer review assignment to the course completion requirements. Students who completed this course could also follow two other courses and complete a capstone project for a specialisation certificate. Students could also audit the course.

Quality Assurance

Medium - The quality assurance was done through review of materials by production and didactics team and content wise by colleagues from the department.

25 - First Report Online Learning Lab

Experience

Sharia in the West

Design elements

Evidence

Reflection

Low - Although the forum provided some elements for reflection this was not a core part of the course design.

Certification

Medium - Certification participants could obtain a certificate on completion of different aspects of the course.

Formal Learning

Medium - The course was initially informal and optional and now will be piloted to run parallel with an on campus course using the material of the MOOC as a BOOK and resource based for our on campus students.

Autonomy

Medium - Participants were expected to work individually and take control of their learning, there was some support from teaching assistants or the professor.

Diversity

High - The course was open to everyone who was interested and was set up to include as much as possible the perspective from different continents

Key elements

This was a Small Private Online Course (SPOC) which focused on bringing together an international set of students/experts in this topic together with our own on-campus students and do research. The difficulty of the course was master level. The course had a strong focus on research and community building encouraging collaboration and interaction.

Platform

First run used Coursera, second run used CourseSites (switching platform allowed to compare tools)

Open

Low - closed to participants

Massive

Low - open to selected participants

Multimedia

Medium - The course used a variety of Multimedia applications ranging from the functionalities of the Coursera and Coursesites platform and Google hangout for student - professor interaction.

Degree of communication

High - Participants are in continuous dialogue with each other and the professor.

The degree of collaboration

High - Community learning and collaboration are key in this course.

Learning Pathways

Low - the course has one structured route for completion.

Quality Assurance

Medium - The quality assurance was done through review of materials by production and didactics team

Reflection

High - Participants are asked to reflect continually during the course.

26 - First Report Online Learning Lab

Experience

Intellectual Property knowledge clips (IP)

Design elements

Evidence

Certification

High - Certification participants could obtain a certificate on completion and when registered in Leiden University the course provides students with credits

Formal Learning

Medium - For the students from outside the university the course was optional. Students from the university who followed the course received credits in the formal structure.

Autonomy

Medium - Students responsible for their own learning in the course but are guided by the professor through every step.

Diversity

Medium - The course consisted of students from the university and students from outside the university living in countries relevant for the course

Key elements

Stand-alone clips on specific content

Platform

YouTube

Open

High - under open license through YouTube

Massive

High - open to all

Multimedia

Low - YouTube is used as dissemination platform

Degree of communication

Low - all material is video content

The degree of collaboration

Low - No collaboration tools available

Learning Pathways

High - Students and lecturers can freely use the material to suit their own needs. Not defined paths were set.

Quality Assurance

Medium - The quality assurance was done through review of materials by production and didactics team.

Reflection

Low - fully self-paced without formal structures

Certification

Low - no certifications are provided

Formal Learning

Low - not part of a formal learning structure

Autonomy

High - Participants were expected to work individually and take control of their learning, material can be flexibly used

Diversity

Medium – the course is open to students and the general public, in Dutch and about Dutch law.

27 - First Report Online Learning Lab

Experience

Design elements

Evidence

Configuring the World

Key elements

A differentiator in this course was the active participation of prof. Griffiths in the online forums. In addition the course asked teachers from all over the world to use the course in their class.

Platform

Coursera

Open

High - under creative commons license cc-by

Massive

High - open to all

Multimedia

Medium - The course used a variety of Multimedia applications ranging from the functionalities of the Coursera platform and google hangout, and Facebook for student - professor interaction.

Degree of communication

Medium - Participants were encouraged to contribute to a number of key debates on the discussion forum with active moderators, in addition the professor provide weekly updates, organised a live hangout and a physical meet up with participants.

The degree of collaboration

Medium - Community learning and collaboration was encouraged through the forum and the peer reviews. In addition students were activly asked to contribute to the debates and provide feedback on eahother's work on the forums.

Learning Pathways

Medium - There were three learning pathways pre-defined for course completion, the first was the basic track based on multiple choice questions and the second a more in depth track adding a peer review assignment to the course completion requirements. Students could also audit the course.

Quality Assurance

Medium - The quality assurance was done through review of materials by production and didactics team.

Reflection

Medium - The forum and in video quizzes provided elements for reflection on the content and discuss students opinions and link them to evidence. The research element was a key part of this effort.

Certification

Medium - Certification participants could obtain a certificate on completion of different aspects of the course.

Formal Learning

Low - The course was setup to be informal and optional.

Autonomy

Medium - Participants were expected to work individually and take control of their learning, there was some support from teaching assistants or the professor.

Diversity

High - The course was open to everyone who was interested.

Note: The classification used in this table is based on article in the Special Issue on Quality in Massive Open Online Courses: http://papers.efquel.org/index.php/innoqual/article/view/164/44

28 - First Report Online Learning Lab

Faculty/Instructor: Faculty of Social Sciences. Instructor: Prof. dr. Madeleine Hosli Course: MOOC The Changing Global Order (Coursera) For Prof. dr. Madeleine Hosli, the MOOC “The Changing Global Order” was much more than just an online course. It truly was a collective effort with many guest speakers, both from within the university as well as from partners in the vicinity of The Hague. This showed the learners the broad range of expertise offered by Leiden University and the city of The Hague on the topic of researching and teaching on themes such as peace, stability, security and justice. Part of this collective effort was a NATO theme week when Campus The Hague could welcome NATO’s Deputy Assistant Secretary General for Emerging Security Challenges, Dr. Jamie Shea to give a public lecture on this theme.

Together with the “Configuring the World” MOOC and the “International Organizations Management” from University of Geneva it is part of one of the first Specialization Programmes offered by Coursera. In this Specialization, “Challenges in Global Affairs”, students get familiar with the three core concepts of global affairs: Political Economy, Global Security and International Management. Upon completion of the capstone project in which the students get the chance to apply their knowledge on a real case study, they will receive a verified certificate that proves their expertise. The best students of this Specialization Programme also get the opportunity to participate in a variety of activities: such as a six-month internship at NATO Headquarters in Brussels “To all the professors and contributors presenting the vital course ‘The Changing Global Order’: I am overwhelmed and humbled by the enormous amount of information, as well as the vastly diverse directions presented in this course. I feel that a study of a lifetime is necessary in order to even begin solving some of the problems that mankind is faced with. In the meantime, participants could play a part of the upcoming solutions. We can write the future history of humanity together.” - Course participant Platforms The platforms that are used in the development of online courses and materials are: Massive Online Open Courses (MOOCs) of Leiden University are uploaded on the Coursera website. In total, the Coursera platform has over 10.000.000 users. Participants come from 186 countries and various backgrounds, ranging from 14 to 84 years old.

For private courses (such as SPOCs or complete online masters) we are currently working with and exploring different platforms to create meaningful online learning experiences, such as the Leiden University website for Open Courseware, iTunesU, Canvas, CourseSites and Youtube.edu channel.

For video conferencing and real-time collaboration we use Google hangout or WebEx.

29 - First Report Online Learning Lab

Democratization of education: Reach and Engagement in Leiden MOOCs4 More than 200.000 people have registered for one, or more, Massive Open Online Courses taught by Leiden faculty and offered through Coursera and approximately 40.000 waiting for reruns on waiting lists.

Table 5 – MOOCs by the numbers Course

Enrolments

Signature Track Enrolments**

With Financial Aid***

The Law of the European Union

52.383

Not applicable

Not applicable

Terrorism and Counterterrorism*

74.144

1.218

215

The Changing Global Order*

49.167

1.725

409

Configuring the World*

32.767

1.086

300

208.461

4.029

924

TOTAL

Figures are from 22-11-2015 (this is the current situation) * This number includes the reruns ** Paying participants for verified certificate *** Participants requesting financial aid for verified certificate

Demographics The enrolments reflect participation from almost every country across the globe. The countries with the highest rates of enrolment (in order of highest to lowest) are: United States, Netherlands, Spain, India, Brazil, German, Greece, Australia, United Kingdom and Italy. These 10 countries represent 48,5% of all

global joiners. From the total number of joiners 7,7% come from the Netherlands.

As far as the percentage of participants coming from emerging economies, Leiden MOOCs are around the Coursera average of 36,7%: Intro EU law: 30% Terrorism 30%, 32% Changing Global Order: 37% and Configuring the World 36%

4 Unless otherwise indicated all graphs in this section are based on data from the MOOCs Law of the European Union, Terrorism and Counterterrorism run 1 and 2, Changing Global Order and Configuring the world.

30 - First Report Online Learning Lab

Figure 2 - Demography

Using sample data from the pre-questionnaire5 on the MOOCs that have run we see that on average, in Leiden MOOCs women represent 45% of the total respondents.

5 For each MOOC we send a pre- and a post questionnaire. The questions are based on the questionnaire Edinburgh University developed for their MOOCs for easy comparison. In total over all MOOCs 19613 joiners responded to the pre-survey, the post-survey had 5312 respondents

31 - First Report Online Learning Lab

Table 6 - gender Gender

Eu

Terrorism 1+2

Changing Global Order

Configuring the World

Coursera average

Female

57%

43%

46%

42%

40%

Male

43%

57%

54%

57%

60%

Based on the sample data from the MOOC courses we see that most participants are between 18 and 34 years old.

The Coursera average is 30.

Graph 1 - age distribution

40% 35%

35% 30% 26%

25% 20%

17% 15% 10%

10%

6%

5%

3%

2% 0% Under 18

18 to 24

25 to 34

35 to 44

45 to 54

55 to 64

65 or over

32 - First Report Online Learning Lab

Education and professional background Most of the respondents have an undergraduate or postgraduate background. About 15% of the respondents indicated that their highest level of education is high school, secondary school or additional training.

Graph 2 - Highest level of education EU Law, Terrorism 1&2, Changing Global Order

Postgraduate university

43%

Undergraduate university

31%

College

10%

Completed high school

10%

Some additional training (apprenticeship, CPD courses etc.)

4%

Some secondary school

2%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

33 - First Report Online Learning Lab

Graph 3 - Areas of employment Student (college/university)

This figure reflects the participants’ areas of employment; 20% of respondents is student, 4,4% is seeking work and 3,9% is retired. Of those who work, the best represented sectors are Law, Public Sector and Education, in all three MOOCs.

Law

Public sector

Teaching and education

Business, consulting and

IT and informawon services

Seeking work

Accountancy, banking and

Student (school)

Retired Configuring the World, a criwcal poliwcal economy approach

Media and publishing

Engineering and

The Changing Global Order Terrorism and Counterterrorism: Comparing Theory and Pracwce (Round 2)

Health and social care

Chariwes and voluntary work

Terrorism and Counterterrorism: Comparing Theory and Pracwce (Round 1)

Armed forces and emergency

The Law of the European Union: An Introducwon 0%

5%

10%

15%

20%

25%

30%

34 - First Report Online Learning Lab

Summing up some of the aspects highlighted around the reach and demography are that a large part of the learners participating in the MOOC (and have been active) are professionals with a higher education degree. This confirms the idea that there is a growing need for programmes for lifelong learners. At the same time there is a large group of learners (15%) who did not have a university degree or are currently enrolled in a university programme.

Engagement Of all people who enrol at some moment before the start, typically 50-60% turns up in the MOOC on the first day. Those who at least once access the course site are called ‘learners’. As the community develops we see different learning strategies emerge ranging from learners who audit the course to students who want to complete with honours doing extra work and learners that are interested in the social aspects of the MOOC. Below a snapshot is represented of Leiden University’s community that was built during the running of the MOOCs so far. Even the enrollers, those who only registered for the course but never entered it have had an initial (touch) relation with Leiden University.

Table 7 – Definitions Term

Definition

Enrollers

Total number of students who enrolled for this session of the course

Learners

Total number of learners who have streamed or downloaded a video

Exam takers

Total number of exams submitted by unique users

Certificates

Total number of basic track certificates that have been given out

Honours

Total number of advanced track certificates that have been given out (for example to students doing extra work such as the peer review assignment)

Signature track

Total number of participants participating in a signature track

Financial aid

Total number of participants that have requested and received financial aid to be part of the signature track

35 - First Report Online Learning Lab

Table 8 - engagement breakdown (excluding currently running MOOCs and other reruns) MOOC

Enrollers

Submitted exam

Certificates (completed basic track)

Honours (completed peer review assignment)

Sig Track

Watched a video (Learners)

With Financial Aid

The Law of the European Union (EU)

52.383

21.655

3.163

1.292

166

na

na

Terrorism and Counterterrorism #1

26.721

12.280

2.977

1.877

411

na

na

Terrorism and Counterterrorism #2

18.169

10.191

2.271

1.422

352

410

69

Changing Global Order

36.110

18.646

2.873

2.088

489

1.165

280

Configuring the World:

32.800

13.888

2.385

1.351

308

1.086

300

Total

166.183

76.660

13.669

8.030

1.726

2.661

649

166.183 Enrollers

76.660 Learners

13.669 Exam takers 8.030 Certificates 1.726 Honors 2.661 Signature track 649 Financial Aid

Based on the post – questionnaire, the average time spend studying by a participant on the MOOC per week was 4,4 Hours. These can be broken down to the time spent on each certain course component. As the graph below reflects, respondents most frequently

watched video lectures, participated in quizzes and in-video quizzes. This indicates that the MOOCs developed so far focused on individual learning, and aimed at course completion.

36 - First Report Online Learning Lab

Graph 4- Frequency of participating in course components

100% 90% 80% 70%

Never

60%

Seldom Some weeks

50%

Most weeks 40%

Every week

30% 20% 10% 0% Watching video lectures

Forum discussions

Readings

In-video quizzes

Quizzes, exams and assignments

37 - First Report Online Learning Lab

Aspirations The underlying reasons for learners to participate in a MOOC based on the survey results are presented in the graph 5 below. 14% of respondents stated (multi-

ple answers were possible) that one of their motives to take the MOOC was “to get a feeling for education at the correspondent faculty or at Leiden University”.

Graph 5 - Aspirations for MOOC participation (What do you hope to get out of the MOOCs you are enrolled in?)

Learn new things

Improve my career prospects

Try online education

Get a feeling for education at É

See what MOOCs are

Meet new people

0%

5%

Satisfaction Participants responding to the post-survey appreciate the MOOCs of Leiden University to a very high de-

10%

15%

20%

25%

30%

35%

40%

45%

gree. Both in terms of the overall experience and with regards to the expectations participants had from the course.

38 - First Report Online Learning Lab

Graph 6 - Rating of overall experience with this course

Very good 44%

Excellent 41%

Good 11% Satisfactory 3%

Poor 1%

Graph 7 - Overall, did you get what you wanted from the course?

Yes, completely 49%

Yes, the course exceeded my expectations 34%

To some extent 16% No 1%

39 - First Report Online Learning Lab

Faculty/Instructor: Faculty of Humanities. Instructor: Prof. dr. Richard Griffiths Course: MOOC Configuring the World: A Critical Political Economy Approach (Coursera) This course uses data from the World Bank and teaches students how to critically assess it. To Prof. dr. Richard Griffiths, critical thinking is one of the most important assets students take home from this course. Sharing insights between the course participants is key to a course that is about understanding the world we live in today. This MOOC organised “preunions” – meet ups before the course had even started – in different cities around the world. Prof. dr. Griffiths explained that “the meeting brought home what an impact the MOOCs were already having in transforming the approach to learning of those who were embracing it and it was a humbling experience to reflect that I was going to be an integral part of this experience. I was so impressed by the commitment that was shown towards global issues and the desire to help create a better world. This is where academic knowledge meets enhanced aspirations and practical solutions.” The evidence-based approach used in this MOOC gained the attention of other universities.

A lecturer of Development Studies in Kenya is now using this MOOC to “discover” Development Economics and Political Economy.

I recently started the Configuring the World MOOC on Coursera, and it was introduced to me by Mr. Griffiths on the BAIS Facebook group. Starting the MOOC allowed me to discover what CtW was before the actual start of my University courses, which was a great advantage. It also greatly helps to have condensed videos online where I can easily review and replay any concept I am not confident about. The MOOC has great visualisation videos which really help to put things into perspective. Additionally, it contains extra readings which allow to read about different views on our world. The greatest thing is that you can learn at your own pace, in your own home. Course participant The content of the MOOC configuring the world has been used by Giovanna, a lecturer in Kenya, in a module called “International Development”. The course is followed by third year students in the department of Development Studies. According to her the response by the students using the MOOC is overwhelmingly positive: “The videos are powerful learning tools, I can use them not only to provide students with high-quality knowledge and to engage them, but also to enable them to build a set of skills which they lack.”

“While learning about the topics discussed, the students were also at the same time building listening skills, the ability to pay attention to details, the skill to select information, to summarise it and take note effectively.“

40 - First Report Online Learning Lab

41 - First Report Online Learning Lab

Impact on on-campus education A key reason for doing MOOCs is the exploration of their potential for on-campus education. These can range from the provision of the course content to students before they start the course to an additional resource in their study or even to the use of the MOOC as an integrated part of a course. Apart from the direct use of MOOCs in an on-campus setting a number of ideas came out of the MOOC process which led to new educational formats we are explor-

ing and developing for on campus use listed below: Small Private Online Courses (SPOCs): “Sharia in the West” was the first SPOC – Small Private Online Course – offered by Leiden University on Coursera and CourseSites. In this course twenty-five students from Leiden University and twenty-five participants from various Western countries collaborate remotely on a research project.

Faculty/Instructor: Faculty of Humanities.

Instructor: Prof. dr. Maurits Berger Course: SPOC Sharia in the West (CourseSites -BlackBoard) Course instructor Prof. dr. Maurits Berger has been very enthusiastic about the use of this form of education. In the first run of this course, Leiden University’s Master students joined forces with a number of highly motivated online students from all over the world. This allowed Prof.

Berger to confront his own on-campus students with a variety of perspectives offered by its global participants.

Upon completion of this course, his Master students received academic credits (10 EC) while the online students received a certificate. The use of small groups and the positive experience with the climate of international academic learning led him to offer another run of this course in September 2014 with new features like a documentary as research project.

“The intensive course has received clamouring reviews by its participants (“Incredible experience”; “Best course ever”; “Different and so interesting”).” - Course participant Knowledge clips: Prof. dr. Dirk Visser, inspired by Leiden University’s MOOCs, decided to record a series of ‘Knowledge Clips’ on Intellectual Property.

These clips are short video lectures from approximately 10 minutes. Topics include for example copyright work, trademark rights and patent registration.

Prof. Visser will use these ten clips for his regular students and for a commercial post-academic course.

The videos are now published on the open channel YouTube for anyone to watch and serves as a great example of developing open resources designed for a worldwide audience.

42 - First Report Online Learning Lab

Platform for research MOOC as a book: During the fall of 2013 the instructors of a Leiden University honours class used a Vanderbilt University MOOC instead of a book. The instructors argued that MOOCs facilitate some of the most effective learning techniques, a claim that can be supported by research (e.g., Dunlosky, Rawson, Marsh, Nathan, & Willingham, 2013). The evaluation concluded that it is not self-evident that a student possesses skills for effective online collaboration.

Instructional design should recognize online collaboration skills in order to include scaffolds for students.

Without the right skills, even the most well designed courses might result in students’ course dissatisfaction, course failure or drop-out.

New Initiatives: apart from creating new forms of education for new target groups, MOOCs and SPOCs inspire faculty to add online components to on campus courses. Apart from the mentioned knowledge clips and MOOC as a book pilot, examples are peer review and community management techniques. In the coming years, we will closely work together with faculty support to leverage the lessons learned from online learning to blended learning.

Online platforms and courses offer the possibility to do research through MOOCs. Participants (the crowd) interested in a topic can provide, classify, and analyse large amounts of information. In online modules Terrorism and Counter- Terrorism and Sharia in the West, the respective professors took an active research angle, which lead to larger and better data sets for a much lower cost.

Based on the initial findings in the research there are a number of ways to involve MOOC participants and the general public in science through online platforms. Below a number of ideas that could be explored in the future when rolling out new online programmes: Wisdom of the crowd: A number of online platforms have invited citizens to classify and work on data that can be used by scientists. Topics range from classifying galaxies, to mapping animals on pictures in the Serengeti and to the folding of proteins. In these platforms gamification is often used to make it ‘fun’ to perform an often repetitive task. The participants need a clear and simple task. To reach a large audience the research topic needs to be inspiring for people to actively participate and donate their time.

Examples of such platforms are: Galaxy Zoo, Foldit, Amazon’s Mechanical Turk.

Wisdom of the MOOC: The MOOC platform offers the ability to engage with a highly interested group of leaners in research projects. As the MOOC is a course it is possible to provide training to the participants before the research starts. Also more analytical tasks can be given to participants. Using the population of the MOOC to source more open questions, develop

43 - First Report Online Learning Lab

Faculty/Instructor: Campus The Hague – Centre for Terrorism & Counterterrorism: Prof. dr. Edwin Bakker Course: MOOC Terrorism and Counterterrorism: Comparing Theory & Practice (Coursera) The MOOC “Terrorism and Counterterrorism: Comparing Theory & Practice” was the first course of Leiden University that offered multiple re-runs. The first offering, in September 2013, was very well-received with high numbers of student participation and completion. For Prof. dr. Edwin Bakker, teaching a MOOC had clear benefits. It offered a unique opportunity to teach a worldwide audience and compare different perspectives offered by all the participants. To this end, he implemented in-video questionnaires, asking learners about their perceptions on terrorism and counterterrorism. In January and September 2014, the MOOC was offered again. This MOOC has now been followed by more than 60,000 learners. Prof.

Bakker has a large dataset that he can use for research purposes. He is now working on translating this large database into a number of publications. Besides this research aspect to the MOOC, he also explained how this course helped him to improve his on-campus education. Later this year, he will use his MOOC in a Flip The Classroom-pilot, combining on- and offline teaching methods.

“This has been a wonderfully enriching experience. I have learned a lot, and have especially appreciated the worldwide perspectives of the class found in the discussion forum. Being in this class has helped me open up the subject to discussion with colleagues and friends, which has led to interesting and insightful conversations. The class materials were effective and polished. The peer reviewed assignments were challenging and fun. This was my first MOOC, but certainly not my last!” - Course participant databases with primary sources e.g. pictures, interviews, polls and opinions etc. Examples are the Terrorism MOOC research on opinions on terrorism and definition analysis and the TU Delft MOOC on water treatment, which developed a large database of pictures of the water sanitation situation around the world.

Focused joint research through SPOCs: In this type of course a selected group of students from different countries who are highly motivated can be asked to do in depth research and provide specific data on a topic. Example: the SPOC Sharia in the West provided detailed data about specific laws and procedures in a number of western countries.

Learning Analytics and educational research: Apart from being a research tool for the teacher of an online course for his or her own domain, the data generated by online programmes can be used to improve teaching and learning. We can look at the data generated by thousands of students and using learning analytics find ways to understand learning better. We can now use randomized testing of control groups to investigate impact of teaching approaches in a ‘virtual lab’.

At Leiden University a research group has been setup to analyse the data of the MOOCs and come up with ideas to use the MOOCs. The research group is linked to the TU-Delft and Rotterdam within the LDE framework under the Centre for Education and Learning (CEL).

44 - First Report Online Learning Lab

Collaborations These collaborations have been of major importance in the success of the online learning initiatives.

Coursera partners Via the Coursera partner channels and conferences there is a very fruitful and stimulating exchange of expertise and best practices. Furthermore, collaboration is facilitated in the reuse of each other’s course materials (as the MOOC we licensed from Vanderbilt University for use in an honours class) and co-offering joined tracks (together with University of Geneva we offer the online specialization on global affairs).

LERU Exchange with many LERU partners on the field of online learning has been valuable, especially with the University of Edinburgh. The LERU seems a natural partner for collaboration in online learning. Last July, the vice-provosts of Oxford, Leiden and Lund universities published a position paper on online learning that has been very well received inside and outside LERU universities, and serves as a starting point to explore further collaborations.

Leiden University, Technical University Delft and Rotterdam Erasmus University (LDE) Partnership In the Centre for Education and Learning, Leiden University, Delft University of Technology and Erasmus University Rotterdam have joined forces in multidisciplinary research into university teaching and learning.

45 - First Report Online Learning Lab

Reflections and recommendations Technology has been part and parcel of education for many years. MOOCs are a mere addition to a large arsenal of technical tools available to teachers and learners. What MOOCs do offer is the ability to scale up, tap into new learner communities, and provides inspiration to a number of new on-campus initiatives.

In the coming years we will expand the portfolio of MOOCs and SPOCs, invest in new online channels, increase impact on on-campus education, and create a set of services to meet the increasing demand for open and online education.

Based on the insights gained from the initial online programme we would like to reflect on the goals set out at the beginning and draw some initial ideas for the future.

Recommendations

• Continue the exploration and innovation of online learning and its possibilities for as well the online as the blended learning offer from Leiden University.

• Explore development and offering of pre-masters, full online masters or blended masters and the market for lifelong learning to tap into an international growing market, and consider offering the first course as a MOOC. • Use MOOCs as a platform for domain research integrated with online learning, easy access to active contributing professionals and to respondents across many different countries.

• Integrate action research into all pilots to ensure continuous learning.

• Keep exploring how MOOCs and SPOCs can contribute to the university strategy and to individual faculties’ strategic goals.

• Keep on developing and professionalizing the production team for online education (MOOCs and SPOCs) which functions for the entire university, with focus on the integration of content, video production and pedagogy to develop engaging learning experiences.

• Further develop services in online learning, didactics and production for campus education that can be used by all faculties in the university. Work together and share experiences with faculty support, towards an integrated offer of services for all faculty.

• Continue to explore modular content development, to enhance possibilities for more personalized learning routes and for different learning arrangements with the same content.

• Continue exploration and experimentation with business models for Online Learning and MOOCs.

46 - First Report Online Learning Lab

Online Learning Lab Structure The core support team is responsible for learning and teaching innovation, and project management.

This includes instructional design; video production; coaching; technical support; development support; online community management; communication and participant recruitment; evaluation and data analysis.

The organisation is based on a network model and deploys agile methods to ensure that innovation and production quality are aligned at all times.

A broader network is involved to help with issues around copyrights clearance (Library (UBL)) and for contract and IP issues (LURIS). Also the lab works in collaboration with various researchers at the Centre for Education and Learning and the Leiden Centre for Learning (LICLON and CROHO). On press and recruitment approaches the Lab works with the Leiden Strategic Communication and Marketing Department (SCM). During the production process MOOC teachers are supported by a junior teacher or student assistant who takes care of daily coordination together with the project management team of Online Learning Lab.

Centre for innovation online learning team

Gideon Shimshon Co-founder Online Learning Lab & Director of Centre for Innovation

Marja Verstelle Co-founder Online Learning Lab & Learning Innovation at Centre for Innovation

Tanja de Bie Community Manager & technical support

Jeanine de Roy van Zuijderwijn Project Coordinator & Technical Support

Thomas Hurkxkens Video Didactics, Innovation & Production

47 - First Report Online Learning Lab

Research network

Janne Polman Knowledge Valorisation

Henk Dekker Research Group CEL / CROHO

Maarten van de Ven Instructional Design ICLON

Joost Kok Professor & Scientific Director LIACS

Jan Porth Data Analyst

Wilfried Admiraal Professor Educational Sciences CROHO / ICLON

Markolf von Ketelhodt Website Development & Maintenance

Jesse Bruins Researcher FSW Leiden University

Data The data presented in this report is based on the Coursera Demographic Survey and Course Database.

For the pre-course and post-course survey we use the survey as developed by Edinburgh so as to be able to compare data in a later stage.

Bart Huisman PhD Researcher ICLON

48 - First Report Online Learning Lab

Appendix: Course descriptives The Changing Global Order (CGO)

Configuring the World (CtW)

Sharia in the West SPOC (SiW)

Terrorism and Counterterrorism (Terr)

Intellectual Property knowledge clips (IP)

# academics

2

1

6

1

1

1

# teaching assistants

2

1

1

1

1

5?

community TA’s

1

12

8

6

1

0

# video’s

53

47

52

56

35

10

Course structure

EU law (EU)

(48 lectures)

(32 lectures)

(36 lectures)

(40 lectures)

total length of video’s (minutes)

479

370

252

296

276:05 (for 29 videos, 6 videos still need to be edited)

94,2

average length of video (minutes)

9:38

11:34

7:00

7:30

9:31

9,42

Item

EU law

Terrorism and Counterterrorism

Structure of the course

5 weeks, weekly video’s, exercises, forum and assessment

5 weeks, weekly video’s, exercises, forum and assessment

Types of assessment

Weekly quizzes, in video quizzes, optional peer-assessment assignment and a final multiple choice based test. Randomized multiple choice multiple choice questions were developed.

Weekly quizzes, in video quizzes, optional peer-assessment assignment and a final multiple choice based test. Randomized multiple choice questions were developed.

Runs

1 (and opened up for own students via YouTube)

4 (moving to continuous running course end of the 2014)

Google Hangout

No

Yes

Video responses to forum topics

Yes

Yes

Wiki used

Yes

Yes

Instructor presence on forum

No

No

Social media used

No

Yes

Item

The Changing Global Order

Configuring the World

Structure of the course

6 weeks, weekly video’s, exercises, forum and assessment

8 weeks, weekly video’s, exercises, forum and assessment

Runs

2

1

49 - First Report Online Learning Lab

Item

The Changing Global Order

Configuring the World

Types of assessment

Weekly quizzes, in video quizzes, optional peerassessment assignment, additional exercise: group assignment on the UNSC reform and a final multiple choice based test. Randomized multiple choice questions were developed.

Weekly quizzes, in video quizzes, optional peerassessment assignment and a final multiple choice based test. Randomized multiple choice questions were developed.

Google Hangout

Yes

Yes

Video responses to forum topics

Yes

No

Wiki used

Yes

Yes

Instructor presence on forum

No

Yes

Social media used

Yes

Yes

Item

Wheels of Metals

Sharia in the West

Structure of the course

5 weeks, weekly video’s, exercises, forum and assessment

10 weeks, videos, tests, research project

Runs

1

2

Types of assessment

Weekly quizzes, in video quizzes, optional peerassessment assignment, forum assignments and a final multiple choice based test. Randomized multiple choice questions were developed.

Weekly quizzes, in video quizzes, optional peerassessment assignment and a final multiple choice based test. Randomized multiple choice questions were developed.

Google Hangout

Yes

Yes

Video responses to forum topics

Yes

Yes

Wiki used

No

Yes

Instructor presence on forum

No

Yes

Social media used

Yes

No

Item

Intellectual Property Knowledge clips

Structure of the course

10 clips explaining the basics of intellectual property laws

Runs

Continue via YouTube

Types of assessment

No

Google Hangout

No

Video responses to forum topics

No

Wiki used

No

Instructor presence on forum

No

Social media used

No

50 - First Report Online Learning Lab

Ontwerp: Studio Piraat

51 - First Report Online Learning Lab

52 - First Report Online Learning Lab