Moocs expectations and reality by Jordi Martí

MOOCs: Expectations and Reality Full Report May 2014

Fiona M. Hollands, Ph.D. Devayani Tirthali, Ed.D.

Center for Benefit-Cost Studies of Education Teachers College, Columbia University

Table of Contents

Acknowledgements....................................................................................................................................... 6 Purpose of the Study and Intended Audience .............................................................................................. 6 Executive Summary....................................................................................................................................... 7 Introduction ................................................................................................................................................ 18 Methods ...................................................................................................................................................... 22 What is a MOOC? Definitions and Characteristics...................................................................................... 25 Definition of “MOOC” ............................................................................................................................. 25 Characteristics of MOOCs ....................................................................................................................... 30 Origins of the cMOOC: George Siemens ................................................................................................. 31 Origins of the cMOOC: Stephen Downes ................................................................................................ 33 Origins of the Modern MOOC (xMOOC): Article contributed by Andrew Ng and Jennifer Widom ...... 34 MOOC Derivatives ................................................................................................................................... 48 How and Why are Institutions Engaging with MOOCs?.............................................................................. 49 Goal 1: Extending Reach and Access........................................................................................................... 53 Case 1: Virology for the Masses: Vincent Racaniello, Columbia University............................................ 63 Case 2: Developmental Education MOOCs: Cuyahoga Community College’s Pre-Algebra CCC ............. 65 Goal 2: Building and Maintaining Brand ..................................................................................................... 67 Case 3: MOOCs to Increase Visibility: American Museum of Natural History MOOC Initiative ............. 71 Case 4: MOOCs for Recruitment: Academic Partnerships MOOC2Degree Initiative ............................. 72 Goal 3: Improving Economics: Reducing Costs or Increasing Revenues..................................................... 74 Potential Cost Savings from MOOCs ....................................................................................................... 75 MOOCs as a Source of Revenue .............................................................................................................. 81 Case 5: Georgia Tech’s Online Master of Science in Computer Science Program .................................. 87 Goal 4: Improving Educational Outcomes .................................................................................................. 90 Case 6: Course Re-design to a Mastery-Based Model: Michael Cima, MIT .......................................... 100 Case 7: Re-designing Introduction to Circuit Analysis: Khosrow Ghadiri, San José State University .... 103 Goal 5: Innovation in Teaching and Learning ........................................................................................... 106 Case 8: Maximizing the Use of Technology in Learning: University of Oklahoma ................................ 110 Goal 6: Research on Teaching and Learning ............................................................................................. 112 Case 9: RELATE at MIT ........................................................................................................................... 119 Examples of Research being Conducted with MOOCs.............................................................................. 121 Types of Data Available from MOOCs....................................................................................................... 127 Where are we with Data Mining, Learning Analytics, and MOOCs? ........................................................ 130 Resource Requirements and Costs of MOOCs .......................................................................................... 134 Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Case 10: Cost Analysis for Development and Delivery of Connectivism and Connected Knowledge (a cMOOC) ................................................................................................................................................. 141 Case 11: Costs of MOOC Production at a Large Midwestern University ............................................. 144 Case 12: American Museum of Natural History MOOC Initiative: Resource Requirements and Expected Benefits.................................................................................................................................. 146 Case 13: Time-by-Task and Cost Analysis for Big Data in Education Development and Delivery ........ 148 Where are MOOCs Going over the Next Five Years?................................................................................ 152 I. How MOOCs will Evolve as Courses ................................................................................................... 155 II. How MOOCs could Affect the Economic Model of Higher Education .............................................. 160 III. How MOOCs can Facilitate Relationships among Institutions ......................................................... 164 Conclusions and Recommendations ......................................................................................................... 167 References ................................................................................................................................................ 171 Appendices................................................................................................................................................ 180 Appendix I: Institutional Affiliations and Roles of Interviewees ........................................................... 180 Appendix II: MOOCs: Expectations and Reality Interview Protocol ...................................................... 181 Appendix III: What Institutions are Doing in the MOOCspace.............................................................. 182 Appendix IV: Costs of Online Video Production.................................................................................... 197 Appendix V: Sample Pre- and Post-MOOC Participant Surveys ............................................................ 200 Appendix VI: List of Interviewees.......................................................................................................... 208

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

List of Tables Table ES 1: Institutional Goals for Developing and Delivering or Using MOOCs .......................................... 8 Table ES 2: Estimated Costs of MOOC Production and Delivery at Four Institutions................................. 12 Table ES 3: Ideas Raised by Interviewees about MOOCs in the Future of Education................................. 14

Table 1: Institutional Affiliations of Interviewees ....................................................................................... 24 Table 2: Institutional Goals for Developing and Delivering or Using MOOCs ............................................. 50 Table 3: Preliminary Instructional and Support Staffing Plans for Georgia Tech’s OMSCS Program ......... 88 Table 4: Summary Projected Costs and Revenues for Georgia Tech’s OMSCS Program ............................ 89 Table 5: Estimated Costs of MOOC Production and Delivery at Four Institutions ................................... 139 Table 6: Estimated Replication Costs for the First Run of CCK08 and Re-run .......................................... 142 Table 7: Range of Hours Spent per MOOC on Design, Production, and Delivery at a Large Midwestern University .................................................................................................................................................. 144 Table 8: Range of Estimated Personnel Costs per MOOC for Design, Production, and Delivery at a Large Midwestern University.............................................................................................................................. 144 Table 9: Personnel Hours Spent by AMNH to Develop Three MOOCs ..................................................... 147 Table 10: Personnel Costs for Big Data in Education Development and Delivery .................................... 150 Table 11: Estimated Replication Costs of Big Data in Education under Various Assumptions................. 151 Table 12: Professional Productivity Changes as a Result of Time Devoted to Big Data in Education ...... 151 Table 13: Ideas Raised by Interviewees about MOOCs in the Future of Education ................................. 154

Table A.1 Institutional Affiliations and Roles of Interviewees .................................................................. 180 Table A.2 Production Costs for an Online Course with 10 hours of Finished Video ................................. 198

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

List of Boxes Box 1: What Does it Mean for a Course to be “Open?” ............................................................................. 26 Box 2: xMOOC vs. cMOOC Characteristics.................................................................................................. 30 Box 3: Accepting MOOCs for Credit: Georgia State University ................................................................... 46 Box 4: Rethinking the Design and Delivery of Engineering Software as a Service at UC Berkeley ............. 93 Box 5: A/B Testing Online ......................................................................................................................... 113 Box 6: San José State University SJSU+ Augmented Online Learning Environment Pilot Project ............ 115 Box 7: Challenges to Conducting Research on Teaching and Learning Using MOOCs ............................. 116 Box 8: IRB Treatment of MOOC Research: What Does and Does Not Require IRB Review?.................... 118 Box 9: What are the Platform Data? ......................................................................................................... 128 Box 10: One Interviewee’s Estimates of MOOC Production and Delivery Costs ...................................... 140 Box 11: What is gRSShopper Anyway? ..................................................................................................... 143 Box 12: High Production vs. Low Production Values for Video ................................................................ 199 List of Figures Figure 1: Types of Institutions Represented ............................................................................................... 22 Figure 2: Roles of Interviewees ................................................................................................................... 23 Figure 3: Estimated Costs of MOOC Production and Delivery at Four Institutions (low end estimates) . 138 Figure 4: Faculty Member's Hours by Month for Development and Delivery of Big Data in Education .. 148 Figure 5: Faculty Member's Hours by Task for Development and Delivery of Big Data in Education ..... 149

Suggested citation: Hollands, F. M., & Tirthali, D. (2014). MOOCs: expectations and reality. Full report. Center for BenefitCost Studies of Education, Teachers College, Columbia University, NY. Retrieved from: http://cbcse.org/wordpress/wpcontent/uploads/2014/05/MOOCs_Expectations_and_Reality.pdf

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Acknowledgements We are very grateful to all our interviewees and other contributors who generously shared with us their many ideas, experiences, and enthusiasm for education, and tolerated sometimes excruciatingly detailed questions. We also greatly appreciate the input and advice of many colleagues. Tom Bailey at Teachers College provided us with initial perspectives on the current “researchability” of MOOCs and what would or would not be realistic to achieve in this study. Henry Levin and Clive Belfield at the Center for Benefit-Cost Studies of Education (CBCSE) advised on strategies for estimating costs and costeffectiveness. Susan Lowes at the Institute for Learning Technologies provided perspectives on online learning and study design, in addition to a constant and up-to-date flow of MOOC-related information. Robert Shand at CBCSE helped think through the initial study design and interview protocol. Robert, and Yilin Pan, also at CBCSE, were instrumental in resolving questions over cost assumptions. Yilin served as our Chinese translator. Henan Cheng identified relevant sources of national prices and Barbara HanischCerda resolved technical issues with our cost tool. Chuck Kinzer in the Communication, Computing, and Technology in Education Department at Teachers College, and Ellen Meier at the Center for Technology and School Change provided useful advice and ideas in the planning stage of the study. Russell Rumberger at University of California, Santa Barbara, provided us with several excellent suggestions for people to interview. Jasmine Ortiz in media services patiently and cheerfully supported our digital audio requirements. Many friends, family members, acquaintances and professional colleagues sent us everything MOOC-related they came across. Last, but certainly not least, we appreciate the input of Fiona’s husband, Ethan Berman, simultaneously our harshest critic and strongest supporter.

Purpose of the Study and Intended Audience When Stanford’s Artificial Intelligence MOOC made headline news in 2011, one of the early predictions was that quality education at mass scale and at low cost was around the corner. Given our research center’s interest in the productivity of educational interventions, we have been watching for evidence that MOOCs are cost-effective in producing desirable educational outcomes compared to face-to-face experiences or other online interventions. While the MOOC phenomenon is not mature enough to afford conclusions on the question of long-term cost-effectiveness, this study serves as an exploration of the goals of institutions creating or adopting MOOCs and how these institutions define effectiveness of their MOOC initiatives. We assess the current evidence regarding whether and how these goals are being achieved and at what cost, and we review expectations regarding the role of MOOCs in education over the next five years. We expect this report to be of interest to administrators, faculty members, and other instructors at colleges, universities, and other educating institutions who are wondering whether MOOCs can advance their missions and serve learners well; to researchers in the fields of educational technology and higher education who are contemplating what kinds of research questions might be useful to address; and to policymakers, investors, and funders considering whether MOOCs are a worthwhile investment of often limited resources for education.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Executive Summary Over the past few years, observers of higher education have speculated about dramatic changes that must occur to accommodate more learners at lower costs and to facilitate a shift away from the accumulation of knowledge to the acquisition of a variety of cognitive and non-cognitive skills. All scenarios feature a major role for technology and online learning. Massive open online courses (MOOCs) are the most recent candidates being pushed forward to fulfill these ambitious goals. To date, there has been little evidence collected that would allow an assessment of whether MOOCs do indeed provide a cost-effective mechanism for producing desirable educational outcomes at scale. It is not even clear that these are the goals of those institutions offering MOOCs. This report investigates the actual goals of institutions creating MOOCs or integrating them into their programs, and reviews the current evidence regarding whether and how these goals are being achieved, and at what cost. How and Why are Institutions Engaging with MOOCs? Through interviews with 83 administrators, faculty members, researchers, and other actors from 62 different institutions (see Appendices I, III and VI for details) active in the MOOCspace or more generally in online learning, we observed that colleges and universities have adopted several different stances towards engaging with MOOCs and are using them as vehicles to pursue multiple goals. Some institutions are actively developing MOOCs and may be termed “producers,” some are using MOOCs developed by other institutions in their programs and could be termed “consumers,” and a few are doing both. Others are adopting a “wait-and-see” approach, or have considered MOOCs and have decided against any form of official engagement. There is no doubt, however, that the advent of MOOCs has precipitated many institutions to consider or revisit their strategy with respect to online learning, whether at large scale or small. Among the 29 institutions in our sample that were already offering or using MOOCs in some way, we identified six major goals for MOOC initiatives: • • • • • •

Extending the reach of the institution and access to education Building and maintaining brand Improving economics by lowering costs or increasing revenues Improving educational outcomes for both MOOC participants and on-campus students Innovation in teaching and learning Conducting research on teaching and learning.

The frequency with which each goal was mentioned by the 29 institutions offering or using MOOCs is shown in table ES1. Goal 1: Extending Reach and Access Extending the reach of the institution to a wider audience and improving access to education was the most commonly identified goal for offering a MOOC, mentioned by 65% of the institutions in our study that were offering or using MOOCs and 42% of our interviewees overall. Ways in which MOOCs are expected to increase access to education include: • “Broadcasting” to global audiences (see Case 1) • Alleviating infrastructure constraints domestically and in rapidly developing countries where the existing physical plant cannot accommodate the growing demand for postsecondary education Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

• • • • • •

Easing the pressure on oversubscribed programs or “bottleneck” courses Increasing access to instructors skilled in specialized domains and niche subjects Providing flexibility in time and place of study Providing a no-risk, low-cost option for at-risk students in developmental education, setting them on an accelerated path to credit-bearing courses and more timely completion of a degree (see Case 2) Flexibility for students to create their own programs with courses from various institutions Continuing education or “professional development” for alumni and other working adults.

Data from MOOC platforms indicate that MOOCs are providing educational opportunities to millions of individuals across the world. However, most MOOC participants are already well-educated and employed, and only a small fraction of them fully engages with the courses. Overall, the evidence suggests that MOOCs are currently falling far short of “democratizing” education and may, for now, be doing more to increase gaps in access to education than to diminish them. Table ES 1: Institutional Goals for Developing and Delivering or Using MOOCs Institutional Goal

% of institutions offering /using MOOCs stating this as a goal (n=29)*

% of all interviewees who raised this as a goal (n=83)

Extending Reach and Access

65%

42%

Building and Maintaining Brand

41%

25%

Improving Economics

38%

29%

Improving Educational Outcomes

38%

20%

Innovation

38%

19%

Research on Teaching and Learning

28%

18%

* Includes one museum in addition to the universities and colleges.

Goal 2: Building and Maintaining Brand For institutions of higher education, building and maintaining brand serves to attract and retain students, faculty members, and partnership opportunities with other institutions, funders, and alumni networks. Interviewees from 41% of the institutions in our study sample that were offering or using MOOCs stated that branding, positioning, or attracting students was a strategic goal for the initiative. None of the community colleges mentioned branding as a goal for engaging in MOOCs. Institutions that were employing MOOCs as a vehicle to expand their brand made strategic decisions to showcase programs, specialties, and research capabilities in which they believed they were leaders or wanted to be among the first to stake a claim to world-class expertise. Case 3 and Case 4 provide examples of how institutions are using MOOCs to increase visibility and recruitment. While many institutions have received significant media attention as a result of their MOOC activities, isolating and measuring impact of any new initiative on brand is a difficult exercise. Most institutions are only just beginning to think about how to capture and quantify branding-related benefits, for example, by comparing historical data on applications and admissions with post-MOOC statistics. Selective Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

institutions must balance the apparently contradictory goals of increasing access to their offerings with building and maintaining brand. Goal 3: Improving Economics: Reducing Costs or Increasing Revenues Thirty-eight percent of the institutions in our sample that were offering or using MOOCs and twentynine percent of our interviewees overall claimed that a goal for MOOC initiatives was to lower costs or increase revenues, or both. These objectives appeared to be more important to public universities than to private ones. There was widespread acknowledgement that the current expenditures on MOOC development could not continue indefinitely without financial justification. Potential Cost Savings from MOOCs In spite of recognition that MOOCs have, so far, proved to be a significant drain on time and money for institutions, interviewees offered several possibilities for eventual cost savings: • • • • • • • • •

Re-using MOOC materials multiple times Sharing MOOC materials across instructors and campuses Developing common courses to offer across institutions Replacing on-campus courses with MOOCs Faculty time savings Reducing the need for facilities Recruitment efficiencies Less costly student support services provided by non-faculty members (see Case 5) Increasing student throughput. MOOCs as a Source of Revenue

While a small number of interviewees (less than 5%) speculated that some institutions of higher education are pursuing MOOCs as a potential source of revenue, only two representatives from state universities explicitly stated revenue generation as a goal. Most institutions, at least for now, appear to be treating MOOCs as an investment. Potential current and future sources of revenue included: • • • • • • • •

Offering credit and charging tuition Creating new for-fee courses and programs Drawing MOOC participants into existing, full-tuition degree programs Increasing class sizes Licensing fees for use of MOOC materials or data by other institutions Fees for additional services, for example, online tutoring, face-to-face instruction with a local instructor Grant revenues Matchmaking or training for employers

It appears that revenue streams for MOOCs are slowly materializing but we do not expect the costs of MOOC production to fall significantly given the highly labor-intensive nature of the process. While these costs may be amortized across multiple uses and multiple years, they will still be additive costs to the institutions creating MOOCs. Free, non-credit bearing MOOCs are likely to remain available only from the wealthiest institutions that can subsidize the costs from other sources of funds. For most Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

institutions, ongoing participation in the current MOOC experimentation will be unaffordable unless they can offer credentials of economic value to attract fee-paying participants, or can use MOOCs to replace traditional offerings more efficiently, most likely by reducing expensive personnel. Goal 4: Improving Educational Outcomes Thirty-eight percent of the institutions participating in our study that were offering or using MOOCs, and twenty percent of our interviewees overall, expected MOOCs to lead to an improvement in educational outcomes, some directly within the MOOC format and some indirectly through the transfer of new strategies and techniques to on-campus teaching. Strategies for improving educational outcomes included: • Motivating instructors to rethink pedagogy • Course re-design • “Chunking” lectures and interspersing questions • Fine-tuning instructional materials • Providing instant feedback to course participants • Gamification and badging to increase motivation • Outreach to course participants to encourage persistence • Adaptive learning/personalization/mastery-based learning • Using MOOCs in K-12 to prepare students for college. While interviewees provided many examples of how MOOCs have been used to change instruction, for the most part, actual impact on educational outcomes has not been documented in any rigorous fashion. Consequently, in most cases, it is unclear whether the goal of improving educational outcomes has been achieved. Case 6 and Case 7 highlight two exceptions where instructional changes and their effects on student performance have been carefully recorded, providing evidence of improvement in student performance as a result of adopting MOOC strategies in on-campus courses, such as frequent assessment and automatic feedback, or of integrating MOOCs into flipped on-campus courses. Goal 5: Innovation in Teaching and Learning Thirty-eight percent of the institutions in our study that were engaging with MOOCs, and nineteen percent of our interviewees overall, presented MOOCs as vehicles for experimenting and innovating with pedagogy (e.g., Case 8) and new models of higher education. While these institutions specified innovation as an end goal in itself, we note that the other five goals we report involve innovating as a means to other ends. A number of interviewees portrayed experimentation with MOOCs and online learning as preparation for an uncertain future. The large number of courses that have been developed or substantively re-designed since the appearance of MOOCs stands as evidence of innovation, for example, MOOC.list currently catalogs around 1,700 courses. It is abundantly clear that MOOCs have prompted many institutions and faculty members to engage in new educational activities. The strategies employed online such as frequent assessments and short lectures interspersed with questions are being taken back on-campus. It is less clear what has been gained by these new initiatives because the value of innovation is hard to measure unless it can be tied to a further, more tangible objective. We address the evidence of success in using MOOCs to achieve other goals in relevant sections of this report but conclude that most institutions are not yet making any rigorous attempt to assess whether MOOCs are more or less effective than other Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

strategies to achieve these goals. A few are starting to develop metrics to assess the impact of MOOCrelated innovations on various objectives. Goal 6: Research on Teaching and Learning Research on teaching and learning was stated as a goal for MOOC initiatives by 28% of the 29 institutions offering or using MOOCs and by 18% of our 83 interviewees. Areas in which interviewees described research explorations and advances included: • • • • • • • • •

The role of social media networks in teaching and learning Testing pedagogical strategies Student engagement and motivation Machine learning/modeling research Natural Language processing Human-computer interaction Personalized/adaptive learning Comparing hybrid with traditional courses Developing data standards and a common platform for data mining.

A great deal of effort is being expended on trying to improve participant engagement and completion of MOOCs and less effort on determining whether participants actually gain skills or knowledge from the courses. Case 9 describes the work of one of the few research groups focusing on the latter. It was, however, noted that this question of efficacy is rarely asked with respect to regular college courses. Most instructors and researchers appear content with equating the completion of a MOOC with having learned something. It was apparent that progress in using MOOCs to improve teaching and learning is being impeded by difficulty using the platform data and lack of clarity regarding regulations applicable to the participants and their data. While the potential for MOOCs to contribute significantly to the development of personalized and adaptive learning is high, the reality is far from being achieved. A great deal of coordination and collaboration among content experts, instructors, researchers, instructional designers, and programmers will be necessary to result in meaningful improvements to teaching and learning. Resource Requirements and Costs of MOOCs While the evidence is mixed as to whether MOOCs have helped institutions achieve their stated goals, it is important to monitor the costs of developing and delivering MOOCs to allow a comparison with alternative means to achieving these goals. While interviewees provided rough estimates of such costs ranging from as low as $5,000 per MOOC to as high as $1.2 million, we conducted cost analyses of MOOC production and delivery at four different institutions to obtain reliable estimates. The major cost drivers in MOOC production and delivery are: •

•

Number of faculty members, administrators, and instructional support personnel involved MOOC production teams seldom included fewer than five professionals and, in at least one instance described to us, over 30 people were involved. Faculty members typically reported spending several hundred hours in the production and delivery of a single MOOC. Quality of videography; we estimated costs for high quality video production at $4,300 per hour of finished video, using national average prices. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

• • • •

The nature of the delivery platform Technical support for participants Programming for special features such as computer code auto-graders, virtual labs, simulations, or gamification Analysis of platform data

Using the ingredients method (Levin & McEwan, 2001), we estimated the actual costs of MOOC production and delivery at three universities and one museum (see Case 10, Case 11, Case 12, and Case 13). Personnel costs alone ranged from $29,000 to $244,000 per MOOC (using national average salaries and benefits), depending on the number of people involved in the production process, the quality of video production, and the amount of time dedicated. Assuming personnel costs account for 75% of total costs once facilities, equipment, and overhead are included, we estimate total costs per MOOC of $39,000 - $325,300, as shown in Table ES2. Costs per completer in the two cases where completion data were available were $74 and $272. By comparison, Bates (personal communication, April 29, 2014; Bates & Sangra, 2011, pp. 163-166) currently estimates the costs to develop a regular 3-credit online course delivered via a learning management system at $35,000 - $50,000. Table ES 2: Estimated Costs of MOOC Production and Delivery at Four Institutions Institution

Type of MOOC

Length of MOOC (weeks)

Total estimated costs per MOOC

Costs per completer

Teachers College, Columbia University

xMOOC

$38,980

$74

University of Manitoba

cMOOC

$65,800 - $71,800

American Museum of Natural History

xMOOC

$104,620

$272

Large Midwestern University

xMOOC

5-8

$203,770 – $325,330

* Completion data were not available for these MOOCs. See Cases 10, 11, 12, and 13 for sources.

The question for MOOCs over the longer term is whether variable costs can be contained by automating functions and substituting instructional support provided by expensive faculty members with less costly teaching assistants, part-time instructors, or peer-to-peer learning and assessment. Many MOOC developers anticipate that the costs of re-running a MOOC will be substantially lower than the costs of initial development. In Case 10, we show that the costs of re-running a cMOOC were 38% lower than the cost of the initial offering. However, given the intense level of instructor involvement in cMOOCs, this is unlikely to be a useful predictor for xMOOC re-runs where instructor involvement may be minimal or absent. A similar cost analysis for xMOOC re-runs is much needed. The Future of Higher Education and the Potential Role of MOOCs Many interviewees envisaged a future of more “unbundling” of educational services, more choice for students in how their education is delivered, greater price competition among providers, and greater Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

use of technology in education. Several questioned whether two- or four-year degrees are the most useful way to acquire an education, and suggested that while colleges and universities may continue to provide foundational skills and knowledge, the flexibility of online education will allow for more “just-intime” learning experiences throughout an individual’s career. Most interviewees believed that higher education will be increasingly pressured to shift towards competency-based models of credentialing. Several foresaw the possibility that the emphasis on earning a degree will dwindle in favor of an emphasis on demonstrating capabilities needed by employers. The possibility was raised that federal funding for postsecondary education could shift away from paying for access to college-based degrees, and towards paying for the demonstration of competency through individually accredited educational experiences. Specific ideas about the potential role of MOOCs in the future of higher education are summarized in Table ES3. Conclusions While much of the media hype surrounding MOOCs may be subsiding and only a small percentage of institutions of higher education are actually offering MOOCs (5% according to Allen & Seaman, 2014), it is clear that the infrastructure and effort that has been poured into such initiatives are not likely to evaporate overnight. Whether MOOCs as they currently stand persist into the future is certainly debatable, but there is no doubt that online and hybrid learning is here to stay and that MOOCs have catalyzed a shift in stance by some of the most strongly branded institutions in the United States and abroad. We ourselves expect that MOOCs or their derivatives will continue to play a role in the continuing education of working professionals, in experimentation with various types of blended or hybrid delivery models on-campus, and in efforts to help struggling students find low-risk options to build skills that allow them to test out of developmental education courses. MOOCs could potentially affect higher education in more revolutionary ways, but not without some significant changes to the status quo. For example, if MOOC providers are able to offer participants credentials of economic value (e.g., college or high school credits; verified certificates of accomplishment; virtual badges to certify skills or noncognitive traits), a market will be established for individual courses, which could be extended to a variety of non-degree-based educational experiences. Such a market would greatly benefit from a system for evaluating and accrediting each course or educational experience - one that is trusted by employers, educators, and funding agencies alike, and that reflects a learner’s ability to contribute productively to society. If funding agencies subsequently become willing to allow learners to apply financial aid to any such recognized educational experience, the landscape of higher education will be opened to more competition, leading to lower costs. A second way in which MOOCs may contribute to revolutionary change is in their potential to catalyze the development of true adaptive learning experiences. However, this will require a significant investment of resources, cross-institutional and inter-disciplinary collaborations, and clarification or change in regulations applying to data use and sharing. If the development of course markets is a trend that may threaten quality of the overall educational experience but lower costs, the development of adaptive learning could balance the shift by improving the quality of educational experiences delivered at scale.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Table ES 3: Ideas Raised by Interviewees about MOOCs in the Future of Education Ideas Raised by Interviewees about MOOCs in the Future of Education

Mentions among 83 interviewees

How MOOCs will evolve as courses MOOCs will serve as educational resources rather than as stand-alone courses

Many

MOOCs will be targeted at specific audiences

Many

MOOCs will offer revenue-generating services such as tutoring, face-to-face interactions, and study groups

Many

MOOCs will offer credentials of economic value such as high school or college credits, or badges that are recognized by employers

Some

Learning analytics data will catalyze the advent of personalized, adaptive, and masterybased learning

A few

Convergence of cMOOC and xMOOC pedagogy

A few

Continuous A/B testing will allow for iterative improvements in materials and activities

A few

MOOCs will incorporate computer grading of open-ended assignments

A few

MOOCs will be available on mobile devices

A couple

How MOOCs could affect the economic model of higher education Students will cobble together their own certifications

Many

Meeting demand for higher education in rapidly developing countries

A few

Faculty members as free agents

A couple

MOOCs-Inside platform and course markets

A couple

Replacing the course catalog and course â&#x20AC;&#x153;shoppingâ&#x20AC;?

One

How MOOCs can facilitate relationships among institutions Cross-institutional collaborations

Many

Life-long connections to alumni

A few

Community outreach

A couple

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Recommendations For college and university administrators Prior to embarking on MOOC-related initiatives, institutions should carefully consider their goals and whether MOOCs present a realistic and financially justifiable means to achieve them compared with alternative strategies. Administrators at academic institutions should work with representative faculty members to establish a strategy for engagement, and, where agreement is reached to move forward, a process devised for production or adoption of MOOCs. Data collection protocols should be devised up front to measure relevant indicators both prior to and post MOOC engagement in order to allow for an objective assessment of whether stated goals are being met. By tracking personnel time commitments with respect to MOOC-related efforts and accounting for displacement of other productive activities, such as teaching or grant writing, costs of such initiatives can be estimated and compared with the evidence of goal achievement to judge whether MOOCs are a worthwhile investment of resources. To broaden access to less educated audiences, institutions must identify multiple channels of communication to reach potential recruits, for example, social media networks, high schools, employment agencies, and community organizations. Additionally, courses must be designed to serve less self-directed learners by incorporating motivational features and optional instructional scaffolding that can address differences in participant preparation levels. Institutions using MOOCs to build or maintain brand must manage the potentially contradictory objectives of increasing access to their educational offerings and building brand by carefully selecting where increased access will be granted, and differentiating these offerings from their core business. To evaluate the impact of MOOC initiatives on these goals, institutions must collect and compare pre- and post-MOOC metrics on student recruitment and enrollment, faculty recruitment and retention, breadth and generosity of donors, and quantity and size of grant revenues. Costs of MOOC production and delivery should be compared to other strategies known to be effective at building and maintaining brand or increasing access in order to ascertain which alternatives are most cost-effective. To reduce the costs of higher education, MOOCs could be used to substitute certain courses across multiple campuses, for example, those that are fairly standard across campuses; courses that serve as continuing professional development for working adults or that fulfill ongoing professional certification requirements; and courses that help students place out of developmental education. Reduction of costs could also be realized if academic institutions are willing to shift more of the responsibility for instructional support away from tenured faculty and onto non-tenured instructors, teaching assistants (TAs), or outsourced personnel. Negotiations with unions must address issues of permissible class size and related instructor income, teacher workload, and assistance from TAs. To increase revenues from MOOCs, institutions must find ways to confer economic value on MOOC completion, for example, in the form of employer-recognized credentials. Careful consideration must be given to the targeted audience and types of offerings so as not to cannibalize existing programs. To maximize the potential for MOOCs to lead to improved educational outcomes through use of participant data, institutions should work towards standardization of data formats across delivery platforms and more uniform interpretation of regulations regarding data use for research and evaluation. To advance the possibility of developing adaptive and personalized learning, interdisciplinary and cross-institutional collaborations must be established among content experts, Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

instructors, researchers, instructional designers, programmers, and for ongoing financial sustainability, perhaps also for-profit content providers who have the capital to invest in this endeavor. For evaluators/researchers/research funders As a first step, it would be useful for a group of researchers and practitioners who are familiar with evidence-based best practices in online learning to recommend pedagogical strategies that can be usefully migrated to MOOCs. Conversely, MOOC researchers could benefit from soliciting the involvement of experienced online learning practitioners or researchers as they plan studies to experiment with MOOCs. Beyond attempting to improve engagement and completion of MOOCs, future research needs to assess the impact of MOOCs on educational and professional outcomes, for example, through: • • •

• •

Pre-and post-assessment of skills and knowledge, for example, through performance-based assessments Development of metrics to assess gain in cognitive and non-cognitive skills that can be applied outside of the MOOC environment Comparison of skill or knowledge acquisition through MOOCs vs. regular online or face-to-face courses Follow-up of post-MOOC outcomes such as sequential learning experiences or employment opportunities gained Broadening the types of learners represented in studies of MOOC activity and impact in order to avoid the presentation of results that are not applicable to the majority of learners.

To assess the sustainability of MOOCs and to help answer the question of whether MOOCs are a costeffective means to deliver education, researchers and evaluators need to: • •

Document how much costs fall for repeat offerings of MOOCs, how enrollment numbers change, and what the uptake rate is for any revenue-generating services Compare the costs of MOOCs to the costs of alternative delivery mechanisms, as well as the effectiveness of each alternative in addressing a common outcome of interest, such as increasing participants’ knowledge or skills.

For policymakers MOOCs introduce a number of problems and possibilities that may not be resolved or realized without policy attention or pressure, in some cases because they may not be in the interests of the institutions which currently maintain control over the issues. Issues that must be addressed include: • • • • • •

Establishing an accreditation system to evaluate MOOCs and other non-degree-based learning experiences to allow learners to accumulate a portfolio of credentials that serve as a viable supplement or alternative to a college degree Creating pathways for accredited MOOCs to be accepted for credit in higher education, or to satisfy government-mandated continuing education for professionals Conferring the ability to use public funds towards covering the costs of such credentials Clarifying what level of student services institutions must provide to MOOC participants Establishing how state funding applies to colleges offering MOOCs Applicability of FERPA to MOOC participants and data sharing among institutions Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

• • • •

Facilitating data format standardization across online learning platforms Permitting cross-institutional Institutional Review Board agreements In K-12, determining whether MOOCs can be completed for credit by high school students within and across states In K-12, determining under what circumstances teachers can offer MOOCs, for example, how cross-state teacher certifications will be managed; who will pay for the course content and for the enrollees.



Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Introduction Over the past few years, observers of higher education have speculated about dramatic changes that must occur to accommodate more learners at lower costs and to facilitate a shift away from the accumulation of knowledge to the acquisition of a variety of cognitive and non-cognitive skills (see, for example, Dede, Ed., 2013). All scenarios feature a major role for technology and online learning. Online education at the college level has been expanding rapidly over the last decade with students participating in single courses or even earning entire degrees without setting foot in a brick-and-mortar institution. According to Allen and Seaman (2013, 2014), the online enrollment growth rate in the United States (U.S.) has ranged between 6.1% and 36.5% in each year since 2002. Means, Bakia, and Murphy (2014) outline and document four major trends in how universities are using online learning: “self-paced, adaptive instruction and competency-based learning; blended learning; learning analytics; and MOOCs” (p. 46). The arrival of massive open online courses (MOOCs), which allow hundreds of thousands of students to participate simultaneously in a course, and are free and open to any interested participant, constitute a phenomenon that extends pre-existing initiatives to provide free, educational resources online, such as MIT OpenCourseWare (OCW), Stanford Engineering Everywhere (SEE), and Khan Academy. While the Open University in the United Kingdom has been providing distance education to large numbers of participants since 1971, and the Open Learning Initiative (OLI) at Carnegie Mellon University has been offering free, college-level courses online since 2002, the term “MOOC” was only coined in 2008 (Cormier & Siemens, 2010). Derived from “massively multiplayer online role playing games” (MMORPGs), the term was first used to describe a twelve-week online course offered by George Siemens and Stephen Downes, Connectivism and Connected Knowledge, at the University of Manitoba, Canada, to an audience of 25 students for credit and an additional 2,300 students who participated without paying and without receiving credit (Downes, 2008). In 2011, MOOCs made frequent headline news after an artificial intelligence course offered by Stanford University professors attracted 160,000 enrollees (Markoff, 2011; Waldrop, 2013). The structure and pedagogical philosophy of this course was quite different from the connectivist MOOCs offered by Siemens, Downes, and other Canadian instructors. To differentiate between the two educational approaches, the terms “cMOOC” and “xMOOC” were coined, “c” denoting the focus on connectivism and “x” denoting “exponential,” to describe the massive participation, or denoting “extension,” for example, HarvardX as an extension of the Harvard campus, and MITx as an extension of MIT (see Downes’ blog post, April 9th, 2013). Most of the MOOCs created at present are xMOOCs. Currently, the MOOC concept is so new that there has been little agreement as to what actually constitutes a MOOC and what educational or other objectives they can and should address. Norton, Sonneman, and McGannon (2013), Daniel (2012), Masters (2011), and deWaard et al. (2011) offered some of the earliest perspectives on these questions based on reviews of the early MOOC producers, available writings (mostly online or journalistic), and specific cases of MOOCs. The two sections of this report that present Origins of the cMOOC and Origins of the Modern MOOC (xMOOC), as recounted by the original creators of these courses, illustrate radical differences in goals and structure of these learning experiences, with the only commonalities being that they are scalable and technology-based. But, as observed by Lewin (2013), one universal impact of MOOCs is clear: “The intense publicity about MOOCs has nudged almost every university toward developing an Internet strategy.” Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Allen and Seaman (2014) report that, in 2013, five percent of 2,831 U.S. institutions responding to an annual survey about online learning were offering a MOOC, nine percent were planning to do so, and fifty-three percent were undecided as to whether to engage in this innovation. Larger institutions were more likely to be offering a MOOC than smaller ones, as were doctoral/research institutions compared with institutions offering less advanced degrees. The value and purposes for engaging with MOOCs have been less clear, and many undecided institutions are struggling with whether and how to join the race to provide education using this relatively untested method of pedagogical delivery. Some faculty members have objected to the introduction of MOOCs on the basis that they are costly and may stifle intellectual inquiry if all students are exposed to the same content and educational philosophy delivered by a small number of “master teachers” (Kolowich, 2013a, 2013b). Allen and Seaman (2014) indicate that the most common objectives for the MOOCs offered by 140 or so institutions in their sample were: increasing the institution’s visibility (27%); student recruitment (20%); innovating pedagogy (18%); and providing flexible learning opportunities (17%). Other objectives mentioned by less than 10 institutions each included reaching new students, supplementing on-campus experiences, exploring cost reductions, learning about scaling, and creating revenues. Surprisingly, increasing the quality of teaching and learning did not appear to be a priority. Peer-reviewed publications on MOOCs began appearing in 2008, (see review by Liyanagunawardena, Adams, & Williams, 2013), and some early indicators of student performance in MOOCs are emerging (e.g., Firmin, Schiorring, Whitmer, Willett, & Sujitparapitaya, 2013; Breslow et al., 2013; Champaign et al., 2014) but there is a dearth of rigorous studies investigating the effectiveness of MOOCs in addressing educational objectives. Researchers are still formulating ideas about how to assess effectiveness of different types of MOOCs. The Department for Business, Innovation, and Skills in the United Kingdom conducted a review of literature on MOOCs and other forms of online distance education (BIS, 2013), and uncovered two conflicting strands of discourse amongst learning practitioners. One strand focuses on “access, empowerment, relationship building and community” (p.4) and the potential for MOOCs to disrupt the traditional higher education system. The other strand argues that many of the benefits of MOOCs have already been realized in online distance education while challenges such as “quality of learning, accreditation, pedagogy, poor engagement of weaker learners, exclusion of learners without specific networking skills” (p.4) are not being adequately addressed by the MOOC format of delivery. Swan, Bogle, Day, and Matthews (2014) suggest a need to distinguish MOOCs according to pedagogical style as a first step before conducting research and evaluation of learning. They have created an instrument “Assessing MOOC Pedagogies (AMP)” that can be used to categorize MOOCs based on the various learning design strategies they employ. Swan et al. plan to use AMP to “characterize classes of MOOC pedagogies” (p.8) that can be linked to specific student outcomes such as retention and learning for particular audiences in various subject areas. Other researchers (e.g., Grover, Franz, Schneider, & Pea, 2013; DeBoer, Ho, Stump, & Breslow, 2014) point out that current discussions about the effectiveness of MOOCs are based on assumptions and outcome variables of previous learning environments and that there is a need to reframe the discussions and re-conceptualize the variables. It appears that few MOOC offerors are willing to address the apparently straightforward question of whether MOOC participants gain skills and knowledge as a result of their engagement in the courses. Part of this reluctance may stem from the fact that MOOCs are being pursued at many institutions for reasons other than improvement of teaching and learning. This being the case, alternative metrics are needed to assess the impact of MOOCs on these other objectives. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Given the ongoing alarm regarding uncontrollable costs of higher education (e.g., Bowen, 2013; Kelly & Carey, Eds., 2013), it would be reasonable to expect not only greater concern about the impact of MOOCs on the intended outcomes, educational or otherwise, but also systematic efforts to document the resources expended on their development and delivery. Beyond the estimates of the costs of MOOC production offered by the E-Learning Working Group at the University of Ottawa (see Boddy et al., 2013), there is little publicly available information on MOOC costs that is based on rigorous analysis. Lack (2013) observes that inattention to costs is pervasive in postsecondary education, and highlights one of the few exceptions in the field of postsecondary online learning: the National Center for Academic Transformation (NCAT) which helps institutions use “information technology to redesign learning environments to produce better learning outcomes for students at a reduced cost to the institution” (NCAT). However, the NCAT experience in itself offers a cautionary tale regarding the resistance of many colleges and universities to adopting technology-based innovations that apparently improve student outcomes, and undoubtedly reduce costs (see Miller, 2010). Carey (2010) attributes this seemingly irrational behavior to a lack of accountability for student learning and a lack of incentives to save money. Ruth (2013) explores the question of whether MOOCs can be used to help reduce college tuition and concludes that MOOCs may only contribute to lowering costs of higher education if combined with a reduction in labor costs, as experienced in successful implementations of NCAT’s course re-design model. Hoxby (2014), at Stanford University, assesses the economic value of MOOCs and concludes that while MOOCs may have some value for non-selective postsecondary institutions, they make little sense for highly selective postsecondary institutions. She argues that MOOCs, which she characterizes as being non-selective, offering standardized content and assessments, and little student-instructor or studentstudent interaction, could substitute existing online and on-campus courses offered by non-selective institutions because, based on data from various national surveys, she surmises that the characteristics of courses at non-selective institutions are not so different from MOOCs. She does, however, question the assumption that cost reductions, via economies of scale, will be realized through MOOCs because she expects that the most popular MOOC instructors will eventually need to be paid high salaries. At highly selective institutions, Hoxby perceives a few opportunities for MOOCs to replace courses that cover standard material, such as calculus or statistics. She asserts that, otherwise, MOOCs are incompatible with the typical characteristics of most courses at highly selective institutions, which often include aspects of the instructor’s research and provide relatively frequent opportunities for intellectual exchange. She notes that if faculty members share cutting-edge research through MOOCs, they risk the loss of intellectual property because of the size and anonymity of enrollments. She also suggests that it is risky for highly selective institutions to give away costly content for free, as this strategy is only likely to be successful for an “infant industry” under conditions of a natural monopoly, which clearly do not exist for MOOCs. Furthermore, Hoxby warns of the possibility that large investments in MOOCs may alienate existing students upon whom highly selective institutions rely for future donations. Hoxby proposes an alternative model for online education at highly selective institutions in which these institutions offer online courses for fee and for credit, but only to students from their own or other similarly selective institutions. This dual-tiered approach to delivering online education, while perhaps economically sound, is far removed from the movement to “democratize” education being led by other Stanford professors. It is curious that MOOCs have taken hold without much evidence as to whether they are effective in improving participant skills and knowledge, or in addressing other objectives, and without an idea of their economic value or resource requirements. As Means et al. (2014) observe, “Both irrational Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

exuberance and deep-seated fear concerning online learning are running high” (p.42). If decision-makers are to make rational decisions about engaging in MOOC production, it is critical to know whether MOOCs are both effective and cost-effective in delivering quality education or related outcomes. In this report we set out to provide clarity on a number of issues surrounding MOOCs and the purposes they serve. We queried a broad range of individuals knowledgeable about online learning, higher education, and MOOCs to establish what constitutes a MOOC, and to determine the goals of institutions and instructors offering MOOCs. After a brief review of our methods, we first present a detailed account of how interviewees, including administrators, faculty members, researchers, online platform providers, and other actors in and around the MOOCspace, define a MOOC. We describe characteristics of MOOCs and MOOC derivatives, and include accounts of the origins of the two major categories of MOOCs: cMOOCs (from the perspectives of George Siemens and Stephen Downes) and xMOOCs (article contributed by Andrew Ng and Jennifer Widom). Subsequently, we provide a brief summary of institutional approaches to MOOCs and a review of each of six major goals being pursued through MOOC initiatives, including whatever evidence we could glean regarding whether and how each goal is being measured and met. We include a number of case studies to provide detail on the MOOC-related activities of a few institutions in pursuit of these goals. Later sections of the report provide information on examples of research being conducted with MOOCs; the types of data available from MOOCs; the challenges of using these data for data mining and learning analytics; and the resource requirements and costs involved in the production and delivery of MOOCs. We present the results of cost analyses we conducted of MOOC development and delivery at four different institutions. We also present a compilation of ideas from our interviewees about how the MOOC phenomenon might evolve over the next five years, and their potential impact on higher education. We end with a few conclusions and recommendations with regard to maximizing the potential contributions of MOOCs to the educational endeavor. Throughout the report we use “Boxes” to help explain or elaborate upon specific ideas or concepts. Appendices include a summary table showing the types of institutions and professional roles represented in our study sample; our interview protocol; a brief account of what each institution in our sample is doing in the MOOCspace; a detailed cost analysis of online video production; sample pre- and post-surveys used to collect data from MOOC participants (courtesy of American Museum of Natural History and Penn State University); and a list of our interviewees. By providing this extensive array of information about MOOCs, we hope to improve transparency of the MOOCspace and to bring greater rationality to MOOC-related decision-making.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Methods Using a methodology similar to that employed by Bacow, Bowen, Guthrie, Lack, and Long (2012) to investigate barriers to online learning in higher education, we conducted a qualitative study comprising interviews with 83 individuals across a range of institutions. With the aim of eliciting perspectives on MOOCs from several angles, we sought out both public and private institutions of higher education, researchers, online learning platform providers, other for-profit education companies, and several additional stakeholders in the online learning space. Figure 1 and Table 1 show the distribution of interviewees across institutional type. Further detail on the MOOC-related activity of each institution in our sample and interviewee roles is provided in Appendix III, except where anonymity was requested. Appendix VI lists alphabetically by last name the interviewees who agreed to be identified. Figure 2 shows the distribution of interviewee roles and further detail is available in Appendix I. Fifty-one interviewees were administrators, faculty members, or both at public or private institutions of higher education, and thirteen were researchers. Throughout the report, we refer to interviewees as each individual specified on their Informed Consent forms, respecting several requests for partial or full anonymity. Interviewees were identified from the existing academic and journalistic literature on MOOCs, by reviewing the names of conference presenters and panelists for sessions on MOOCs or online learning in higher education, by researching the MOOC activities of institutions on the Internet, or by consulting with known experts in the field of educational technology. Additionally, many of our interviewees suggested other people for us to interview either at their own institutions or elsewhere. We contacted by e-mail individuals who appeared to be Figure 1: Types of Institutions Represented knowledgeable about MOOCs or online Other learning based on their position in institutions Other fordeciding whether and how to participate 11% Public profit in the MOOCspace, their experience universities education teaching or planning MOOCs, or their 26% companies 6% writings and research in this area. We contacted a total of 100 individuals on a rolling basis at 66 different institutions, 39 of which were colleges or universities. 11% Of the 100 contacted, 83 people at 62 institutions were successfully interviewed within the time-frame of our study. At Platform most institutions we interviewed one providers person, but at a few institutions we Private 8% interviewed two or more. Most universities Community 23% interviewees were based in the U.S., but colleges two were in China, two in the United 15% Kingdom, and several were in Canada. Seven of the people we contacted recommended we interview another individual at the same organization instead, six of whom accepted the invitation. Five people we e-mailed did not respond, but in each case we were able to identify another individual at the same institution who agreed to participate. Three people responded to our emails but did not schedule an interview time during the study period. At least one person from every Research organizations

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

college or university we contacted agreed to participate. Only two individuals from the same for-profit institution outright refused participation. Figure 2: Roles of Interviewees 2%

Administrators/executives

16%

Faculty members 9%

Administrators/faculty members

55%

Researchers 18% Other roles

Interviews were conducted between June 2013 and February 2014 and follow-ups by e-mail with interviewees to obtain updates and to verify information continued until time of writing. Slightly under half of the interviews were conducted face-to-face and the others were conducted by telephone or Skype. Interviews lasted an average of 75 minutes and followed a semi-structured interview protocol which appears in Appendix II. Interview questions addressed what the interviewee’s institution is currently doing with respect to MOOCs; institutional goals for pursuing MOOCs; the interviewee’s role in the process; how the interviewee and his or her institution define a MOOC; the characteristics of MOOCs, including structure, purpose, enrollment level, fees, and credentials offered; how any credentials differ from regular course credentials; educational objectives of specific MOOCs being offered; educational outcomes being measured; data being collected pre-, post- and during MOOC delivery; how MOOC-based research is being used to improve pedagogy on-campus; personnel and other resource requirements; cost drivers; how costs of MOOC production compare with face-to-face courses; how the interviewee sees the MOOC phenomenon developing over the next five years, and how this might help the institution meet its ongoing goals. In some instances, not all questions were relevant to the interviewee. For example, researchers were unlikely to be producing or delivering MOOCs themselves. Most interviewees granted permission for the interview to be recorded, and the resulting digital audiofiles were subsequently transcribed. All hand-written interview notes and transcriptions were coded using NVivo software using themes initially derived from the protocol and iteratively refined as more granular topics were identified. Interviewee perspectives were supplemented with a review of the MOOC-related literature and perspectives offered by presenters at various academically oriented, MOOC-related events: the “moocshop” at the 2013 Artificial Intelligence in Education conference; the 2013 MIT Learning International Networks Consortium conference; the 2013 MOOC Research Initiative 1;

Attendance for one author at this conference was sponsored by the Bill & Melinda Gates Foundation, for which we are grateful. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

and a 2013 workshop entitled “Will MOOCs Pass the Test? Evaluating These New Online Courses,” sponsored by American Institutes for Research and the American Council on Education. Table 1: Institutional Affiliations of Interviewees Type of institution Public universities Private Universities Community Colleges Platform Providers Research organizations Other for-profit education companies Other institutions** Total

Number of institutions represented* 16 14 9 5 7 4 7 62

Number of Interviewees 20 26 10 6 8 5 8 83

* Note that while only one person was interviewed at the majority of institutions, at a few institutions several individuals were interviewed, for example, to include one or two administrators, one or two faculty members, and one or two researchers **Other institutions: one museum (two interviewees); one K-12 school district; one educational technology advocacy group; one higher education association, one venture capital firm; one private foundation; one independent consultant.

Cost analyses reported in the section Resource Requirements and Costs of MOOCs were conducted using the ingredients method (Levin & McEwan, 2001) and were executed using the CBCSE Cost Tool Kit. We focused on estimating personnel costs incurred in the production of a variety of MOOCs and followed Levin and McEwan’s guideline that personnel costs typically account for 75% of total costs of educational interventions (Levin & McEwan, 2001, p.53). Wherever possible, instead of using actual salary levels of personnel at each specific institution, we use national average salaries to calculate the costs of replicating production of these MOOCs. This approach not only respects the privacy of the individuals involved, but, more practically, allows for a comparison of the costs across a number of institutions without introducing local pricing influences. National average prices and benefits rates were obtained from the CBCSE Database of Educational Resource Prices. A draft of this report was circulated to all 83 interviewees prior to finalization to provide an opportunity to suggest edits, provide updates, and verify information.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

What is a MOOC? Definitions and Characteristics In this section we first summarize our interviewees’ definitions of a MOOC, starting with their interpretations of each of the four components of the name: massive, open, online, and course. We proceed to review a number of MOOC characteristics including: course structure; populations served; enrollments and completion rates; credentials and fees; and platforms used. Finally, we summarize a few of the course structures derived from MOOCs.

Definition of “MOOC” The term “MOOC” (Massive Open Online Course) was coined by David Cormier in 2008 (Cormier & Siemens, 2010) to describe a twelve-week online course, Connectivism and Connected Knowledge, designed by George Siemens and Stephen Downes and offered at the University of Manitoba, Canada, in Fall semester 2008. Twenty-five matriculated students took the course for fee and for credit while another 2,300 participated as “open” enrollees. “MOOC” is a play on the acronym MMORPGs: massively multiplayer online role playing games. The section Origins of the cMOOC describes the pedagogical philosophy and strategies behind the development of the first “connectivist” MOOC from the perspectives of George Siemens and Stephen Downes (see page 31). The key goal was “for people to experience what it means to be part of a social, technical system of learning where the teacher’s voice is not an essential hub but, instead, a node in an overall network” (Siemens, formerly at Athabasca University). The structure and pedagogical philosophy of the open online courses offered at Stanford University in 2011 were quite different from the connectivist MOOCs. The article Origins of the Modern MOOC (xMOOC), contributed by Andrew Ng and Jennifer Widom, describes the sequence of events leading to the advent of the Stanford courses (see page 34). Whereas the cMOOCs facilitate learning through participant interactions with a network of individuals and encourage participants to create, share, and build upon each other’s artifacts (e.g., videos, blog posts), the Stanford-style MOOCs are primarily designed to deliver education at scale and involve more structured and sequenced direct transmission of knowledge. To differentiate between the two educational approaches, the terms “cMOOC” and “xMOOC” were coined, c denoting the focus on connectivism and x denoting exponential, focusing on the massive enrollments, or extension, for example, HarvardX being an extension of the Harvard University campus. The acronym “MOOC” has since been applied to a variety of online and blended courses. As Joseph Ugoretz, Associate Dean of Teaching, Learning and Technology, at Macaulay Honors College (CUNY) points out, we are “…at a place where almost any kind of online learning is called a MOOC, or if it's not called a MOOC, no one pays any attention to it....” Ugoretz further observes that the hype surrounding MOOCs has led to catchy acronyms being invented for course offerings that are similar to types of online courses that have existed for some years. These MOOC derivatives include the SPOC, the POOC, the DOCC, and the SMOC (see MOOC Derivatives below). “Massive” Interviewees expressed their understanding of “massive” in terms of actual numbers of course participants, the capacity for courses to enroll large numbers, or the capacity to obtain vast quantities of participant activity and performance data. Interviewees who suggested actual numbers of participant Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

enrollments mentioned numbers ranging from a few hundred to one hundred thousand, with 500 and 1,000 being popular benchmarks. A few felt that “massive” could apply to any class size that is too large to be accommodated in a face-to-face learning situation such as a traditional lecture hall or an auditorium. Others differentiated between MOOCs and typical online courses not only in terms of class size, but also with respect to instructional design of the course. Given the expectation of peer learning in MOOCs, a certain critical mass is necessary for the instructional design to be successful. For example, George Siemens defined massive as: Anything that is large enough that you can get sub-clusters of self-organized interests. Three hundred plus students could be one benchmark; another could be Robin Dunbar’s number of 150 people, which is the maximum after which the group starts to create smaller fractions. (Siemens, formerly at Athabasca University). Some interviewees focused more on the potential for courses to be massive as opposed to the actual numbers of enrollments. For example, they should be scalable at negligible marginal cost, and incorporate technological and pedagogical features that can support an infinite number of participants. A few interviewees suggested that the most significant contribution of larger class size in a MOOC is the "massive amounts of data that can be generated by these technologies." (Professor, East Coast University). “Open” The most generic interpretation of the word “open” with respect to a MOOC was the possibility for anyone with an adequate Internet connection to participate in the course. Many interviewees also interpreted open as free, that is, no fee is required for participation. However, most of these interviewees anticipated the imminent arrival of optional for-fee services such as credentialing or tutoring. Provided there is a basic enrollment option that remains free, interviewees will continue to perceive the courses as open, and massive enrollments are expected to persist. Some interviewees pointed to different interpretations of open as applied to access, content, and the online platform as described in Box 1. Box 1: What Does it Mean for a Course to be “Open?” Open access Interviewees presented different ideas about what constitutes open access to a MOOC. Karen Cator, President and CEO of Digital Promise, asserted that, to be considered open, a course should not require prerequisites or pre-tests that will limit participation. Judy Baker, Ph.D., Dean of Foothill College, CA, does not consider MOOCs to be open if participants must register to access the content. A few interviewees believe that courses are only truly open if they allow enrollment and participation at any time, that is, participation is not confined to the typical duration of a course. These interviewees were likely to be more expansive in their definition of what constituted a MOOC. For example, Elle Wang, a MOOC Researcher at Teachers College, Columbia University, included Khan Academy videos in her definition as they are more open than the typical xMOOC content that is only sometimes searchable on YouTube, and often inaccessible after the conclusion of the course. Others were more restrictive in their interpretation and felt that enrollment could occur any time within a Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

given window. These different understandings of openness have implications for the instructional design of MOOCs. For example, in order to help participants succeed, a MOOC without prerequisites may need to include additional scaffolding and links to self-study material for those who enroll without any relevant background in the topic area. If participants are not required to register, the course platform cannot track student progress. Absence of such longitudinal participant data precludes the development of adaptive learning mechanisms and inhibits the collection of learner activity and performance data that might be useful to researchers. Courses that are open to enrollment and participation at any time must be perpetually hosted and at least minimally monitored by the offering institutions. Participants can learn what they want, when they want, without the constraints of time or course structure. Such rolling course enrollment and completion is likely to render the experience completely asynchronous for most participants. Dr. Glenn Kleiman, Executive Director, Friday Institute for Educational Innovation, North Carolina State University, cautions against making the structure of an online course overly flexible. He believes that some structure is needed to facilitate professional exchanges that will occur only if participants are progressing roughly at the same pace through the course. Open content Some interviewees who had been previously involved with the Open Educational Resources (OER) movement objected to the use of “open” with respect to MOOCs because the meaning is not usually consistent with the “openness” of OER. Open content in the OER context implies legally open licensing, for example, with a Creative Commons (CC) license. Such open content is available for anyone to view at any time, to download, use, share, and “mash up” with their own materials. In cMOOCs, openness extends to participants being able to contribute resources and provide direction to a course, which eliminates the one-way nature of many instructor-student relationships. Truly open content is common in cMOOCs but less so in xMOOCs where ownership of content is often contentious and has to be worked out between faculty members, their employers, course platform providers, and would-be adopters of the MOOC content. Timothy Renick, Vice Provost and Chief Enrollment Officer at Georgia State University, noted that the University’s contract with Coursera specifies that faculty members cannot direct students to use MOOC materials from other institutions’ MOOCs as part of their courses without paying a licensing fee. Karen Vignare, Associate Provost, Center for Innovation in Learning, University of Maryland University College (UMUC), pointed out that when contracts for the delivery of MOOCs are signed between a third party platform provider and an institution, as opposed to directly with an instructor, the individual instructors are unable to choose whether their content is made open access. While many MOOC instructors strive to incorporate open content, it is not always feasible and some have good reason to remain protective of their materials. Dave Pritchard, Cecil and Ida Greene Professor of Physics at MIT, expressed some reservations about open content: The issue of the extent to which the material should be public, I feel some ambivalence on. I have developed many really carefully thought out problems that elicit a lot more student interaction than other problems. I don't want them out there with the answers so that all the students can just copy the answers. I'm happy to share them with other bone fide teachers. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

The Creative Commons license does not say that you can treat teachers any different from students. Then that raises a huge amount of problems with, "Who's a teacher?” How do you establish that you're a teacher? If you're a student taking this thing, you could masquerade as a teacher, and then plop up all the answers on the worldwide web. How do you prevent that from happening? (Pritchard, MIT). Open platform A few interviewees suggested that openness in MOOCs requires an open source delivery platform. In April 2013, Stanford University and edX signed an agreement to collaborate on development of the edX platform with a caveat that the platform would be open source by June 2013. The resulting Open edX initiative has now made the platform code and various related tools freely available under an AGPL license. In September 2013, Google agreed to offer a hosted version of the Open edX platform, thereby making it more accessible to institutions. Open edX carries the interpretation of “open” one step further in allowing institutions to retain control over their content and related platformgenerated data by creating their own instances of the Open edX platform. Candace Thille, Assistant Professor, Director of Open Learning Initiative, Senior Research Fellow, Office of the Vice Provost for Online Learning, Stanford University, is taking advantage of this capacity by working with Open edX to incorporate functionalities of the Open Learning Initiative (OLI) platform into the Open edX platform. This will allow OLI courses to be supported on the Open edX platform and render them easily accessible to more people. “Online” There was little disagreement in the interpretation of the term “online.” Almost all interviewees stated that MOOCs should be available via the Internet on a variety of devices to facilitate scale and to expand access beyond the traditional campus. However, experimentation with the MOOC structure has led to variations that include offline elements. For example, some instructors are using MOOCs as resources to flip their on-campus classrooms and others are “wrapping” MOOCs with their own materials for use oncampus (see Fisher, in press). Some MOOCs have been specifically designed to cater to a small group of on-campus students at the same time as being accessible to a mass of outside or “open” participants. Examples of the latter include the first MOOC offered by Siemens and Downes at the University of Manitoba, and University of Oklahoma’s online courses. Less formally, face-to-face peer-group meetings have occurred spontaneously in different parts of the world and some platform providers have organized face-to-face “meet-ups” to connect MOOC participants. Pearson VUE testing centers may soon be used to provide physical spaces for MOOC participants to sit for proctored exams. Interviewees generally anticipated a trend towards more blended applications of MOOC content with some specific ideas outlined in the section Where are MOOCs Going over the Next Five Years? (see page 152) “Course” Most participants agreed that, to be labeled a “course,” MOOCs should be bounded by time, that is, have a beginning and an end point; should provide a coherent set of resources; and follow a sequence of activities organized by an instructor in order to address specific learning objectives or goals. Siemens noted that without the element of “eventedness,” even the Internet could be conceptualized as the Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

worldâ&#x20AC;&#x2122;s first MOOC. The degree of synchronicity required for a course varied among interviewees, with most believing that asynchronous participation within set time periods (e.g., one to two weeks) was most conducive to learning in an online course. Reasons given for enforcing set start and stop points for a MOOC included motivating participants to engage regularly with the course content and facilitating discussions and peer grading. However, a few interviewees argued against such constraints, noting that they simply conform with seat-time requirements for the purposes of accreditation in traditional university or college settings. Ugoretz at Macaulay Honors College observed that such a time-bounded perspective can be frustrating for life-long learners. Some interviewees provided additional specific criteria for a â&#x20AC;&#x153;course.â&#x20AC;? For example, Ted Dodds, Chief Information Officer and Vice President for Information Technologies at Cornell University, suggested that a weekend training session does not constitute a course while an eight- to ten-week learning experience may do so. Others insisted on the need for assessments in order to gauge whether learning objectives are being fulfilled. Interviewees did not believe that certificates and credentials to prove completion were a requirement for a learning unit to be considered a course, although many felt that they should be offered.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Characteristics of MOOCs Structure The vast majority of existing MOOCs are xMOOCs. They are structured as weekly sequences of activities, over as little as two weeks and as much as sixteen weeks but, most frequently between six and ten weeks. Instruction is provided predominantly via several short lecture videos per week, typically each 10 minutes or less in length, sometimes supported by supplementary readings, and problem sets or other assignments. Videos are often punctuated every two-three minutes with automatically graded “inline” questions, usually multiple choice or short answer, to help participants formatively assess their own understanding. Assessments that count towards the participant’s final score are provided, usually weekly, in the form of auto-graded multiple choice or short answer quizzes, peer-graded assignments, and some auto-grading for computer code. Online discussion forums allow participants to engage with each other and course facilitators for technical and instructional support, or merely to create a sense of community. xMOOCs are generally delivered via third party platform providers such as Coursera, edX, and Udacity. Most are “cohort-based” in that they are offered over a fixed period of time, with participants being expected to complete activities within set windows of time. These courses and their associated materials often become unavailable for non-registered participants not long after their conclusion. Some xMOOCs are self-paced, remaining open indefinitely to participants. Box 2: xMOOC vs. cMOOC Characteristics •

xMOOCs Pre-determined, instructor-led, structured and sequenced weekly activities

•

Short, content-based videos, readings, problem sets

•

Quizzes (auto-graded), peer-graded assessments

•

Discussion forum participation optional

•

Delivered via third party platform provider (e.g., Coursera, edX)

•

cMOOCs “social, technical system of learning where the teacher’s voice is not an essential hub but a node in an overall network” (Siemens). Creation/exploration of topic area in “atelier” environment Unique products created by students (blog posts, images, diagrams, videos) Discussion forums, Diigo groups, Twitter and other social networking are key Facilitator aggregates, reviews, summarizes and reflects on activity in daily/weekly newsletter “Boot-strapped” platform and collaboration tools

Connectivist MOOCs are characterized by a more fluid structure that addresses an overarching instructional goal or question but is less directive with respect to process. Siemens describes the cMOOC experience as the exploration of a topic area and creation of artifacts in an “atelier” environment. Instructors may pose initial or weekly questions and challenges, and provide a variety of text-based or media resources. The success of the cMOOC is highly dependent on participant interaction, for example, via discussion forums, Diigo groups, or Twitter. Course outcomes are often unique products, such as blog posts, images, diagrams, or videos generated by participants using a variety of social media. The Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

role of the instructor is to act as a facilitator by aggregating, reviewing, summarizing and reflecting on participant activity in a daily or weekly newsletter. cMOOCs are usually delivered using easily available course delivery platforms, such as Blackboard Collaborate, enhanced with collaboration tools. Box 2 xMOOC vs. cMOOC characteristics summarizes key differences between the two major types of MOOC. The cMOOCs we encountered among our interviewees ranged in length from four to 30 weeks. In several cases, the courses have featured different instructors each week leading the activity or discussion. For example, Stephen Downes at the National Research Council offered the cMOOC CHANGE which featured 26 experts over a period of 30 weeks. Downes observed that while the course stimulated a great deal of discussion, it “felt too long.”

Origins of the cMOOC: George Siemens “In the 90s, I was at Red River College which became the first college in Canada to go exclusively laptop. Use of class time changed overnight. While teachers pretty much just substituted PowerPoints and display cameras for overheads, chalkboards or white boards, students started working in a totally different way. They were downloading music, texting, instant messaging, and doing some really creative things that they learned through their social systems. It became evident to me that when students were given their own tools of control, they acted in certain ways that you weren’t able to anticipate. With the one-to-one technology in place, I started to play with peer-to-peer systems like Groove to investigate how technological and social networks influence the teaching and learning process. Stephen Downes was similarly exploring distributed social and technological interactions. We shared a central vision that learners should own their own space of interaction. Our interpretation of openness is that, if you learn transparently, you become a teacher, because people observe your learning; they see what you are doing and they can emulate or adapt it. Unintentionally, you teach others who are at a similar stage. If you mentor a way of being, then there is more that happens in the learning process than the acquisition of knowledge. There is the acquisition of the ways you interact with peers in that space. You become something. You don’t just learn the knowledge of, say, a dentist; you become a dentist. That includes the acquisition of a variety of mannerisms, social standards, and norms that might never be embedded in a curriculum. In 2002, I started a “non-course” with my newsletter subscribers. I would kick off the week by asking one question and the mandate for participants was to discuss the question throughout the week, share resources, detailed opinions, and so on. This was all done on Yahoo groups. At the end of the week, I would review all the interactions and write an article that summarized the activity and discussions. I did not present content; the content was the by-product of the learning experience each week. Most participants were in higher education, mostly faculty plus 20 or so students. In 2004, I wrote a paper called Connectivism, a learning theory for a digital age about how teaching with technology shifted the control structures: how students were able to do things for themselves that I had to do for them in the past and how this had an impact on the design of the learning experience. I spent some time giving presentations on this notion of social network learning or connectivism, but eventually I thought it best to have people experience what I meant instead of talking at them. First, I ran a couple of online conferences on Connectivism to experiment with this idea. I used the Moodle discussion forum Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

and Elluminate as the synchronous platform. Others like David Wiley and Alec Couros were doing similar things. In 2007, I got together with Stephen Downes at the National Research Council in Canada and we agreed to run an open course on connectivism for people to experience what it means to be part of a social, technical system of learning where the teacher’s voice is not an essential hub but, instead, a node in an overall network. I had just developed a certificate program at the University of Manitoba called the Certificate in Emerging Technologies for Learning and this course became the central course in that. We made Connectivism and Connected Knowledge (CCK08) available online and announced it through Stephen’s daily e-mail, which had about 10,000 readers, and through a regular e-mail I sent out about tech trends, which had about 7,000 – 8,000 readers. We had 25 students enroll for credit and we allowed anyone else to join for free with no credit. We ended up with around 2,300 people subscribed. Before the course started, we had a few online meeting sessions with instructors who had taught online before or who had experimented with open courses: David Wiley, who is now with Brigham Young, had previously done a course with about 50-60 students; and Alec Couros who had done one with about a hundred students. We discussed what the experience could be like and how to make sure it would work well. Brian Alexander and Dave Cormier observed that what we were planning was like a massive open online role-playing game, so that’s how the term “massive open online course” got coined [see Cormier’s blog post on this]. Since then it just became known as a “MOOC” and we have been running up to three MOOCs a year. In 2011, I heard about the Stanford initiative and wrote a blog post in August saying that Stanford is running a MOOC. Tamar Lewin from the New York Times picked it up and I had an interview with her. I said “That’s a MOOC they’re running.” We had 2,300 students, but their scale changed everything. Our emphasis was that we wanted participants to create things. Our view was that you can’t take knowledge that is in the head of a person and deposit it in the head of another person, it has to be generated through active means. The pedagogical model has to emphasize creation and generation. It was the beginning of the “remix” culture where we start something and share it, someone else takes it and improves it, someone else improves it even more and maybe links to it to something else. You get a network innovation effect going. We believe that content should be a by-product of the learning experience, not necessarily a prerequisite. If you externalize how you see a concept by creating an artifact, someone can come by and re-mash it, re-hash it, build it, improve it, mix it up or identify any knowledge or conceptual errors. Our core assertion was that knowledge is a networked entity and learning is the process of forming, pruning, developing those networks. We received criticism because we tried to reduce the profile of the faculty members and encourage peer networks and structures. We never really emphasized that all students need to participate in a central discussion forum, even though we had a Moodle forum. Instead, the emphasis was on each person participating in a space that they owned, creating a personal learning environment and a personal learning network. Students would submit their blogs and their RSS feeds and we would pull out the RSS feeds from all the different participant websites and the Moodle forum, and create a daily e-mail newsletter that aggregated everything. Our course was so different from the current MOOCs because we strove to drive the students away from our central platform whereas edX, Coursera, Udacity bring you to their own spaces and determine when you can no longer access the materials.”

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Origins of the cMOOC: Stephen Downes “George and I had both been working for a long time in the area of what I was calling “learning

networks” and what he was calling “connectivism.” It's basically the same concept, and it's the idea that learning is a network phenomenon and that the organization of learning should therefore be network organization. For me, the origins of learning networks are based in the works that I did in philosophy in the 80s and 90s - it was called "connectionism" at the time, which is a self-organizing system. The heart of what I do is to try to create tools and methods to help people acquire learning for themselves, rather than have learning provided to them by some provider or institute. In 2007, George had run an online connectivism conference which was pretty popular. I'd participated in that. The University of Manitoba asked him to teach an online course on connectivism for the Certificate in Adult Education, which is a Manitoba-wide certification. I'd actually done work preparing a course for the same certificate a number of years earlier, called Introduction to Instruction, which was one of the very first online, web-based courses in Canada. He asked me if I'd co-teach this with him. The objective of the course became for the two of us to explain what it was that we meant, what we were up to when we talked about connectivism. It's wonderfully recursive. We decided to run the course according to connectivist principles even though this created some challenges for a course that was intended to be part of a university certificate program. One of the principles of connectivism is open learning, so all the content needed to be open. Another principle is diversity so we set it up such that people could use whatever system that they wanted, their blogger system, Twitter, Facebook, et cetera. I have a tool called "gRSShopper," that I'd been developing for a number of years, which allowed us to set up the course as a network. We would use gRSShopper to link all of the content together (see Box 11). We still expected 20 to 25 people, but 2,300 people signed up. Because we had set it up in this network structure, we didn't face 2,300 times whatever amount of time to do certain tasks. Any place where there would have been a bottleneck, we used the network instead of an administrative structure. For example, if students contribute content to a course and it's on the university website, you've got to check each piece of content, especially if it's an open course, to prevent them from, say, uploading pornography. Because people were putting the content on their own websites, it was not necessary for us to pre-screen it all. We posted the regular class assignments and the grading rubrics. Some people did them in their own blogs, but they weren't graded unless they were enrolled in the course. We actually made it clear: “You choose the manner in which you participate in the course. If you just want to follow along and read the newsletter, that's fine. If you want to set up a blog or a feed of some sort, and contribute, that's fine.” There was also a mechanism where you could put comments directly on the website. It was up to people how they wanted to contribute.”

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Origins of the Modern MOOC (xMOOC): Article contributed by Andrew Ng and Jennifer Widom Andrew Ng is the Director of the Stanford AI Lab and a Co-founder of Coursera. Jennifer Widom is Chair of the Computer Science Department at Stanford University. Online education has been around for decades, with many universities offering online courses to a small, limited audience. What changed in 2011 was scale and availability, when Stanford University offered three courses free to the public, each garnering signups of about 100,000 learners or more. The launch of these three courses, taught by Andrew Ng, Peter Norvig, Sebastian Thrun, and Jennifer Widom, arguably marked the start of the modern, instructor-directed MOOC (sometimes “xMOOC”). Each of these MOOCs offered learners the opportunity to watch online lectures, do machine-graded homework, and earn a “Statement of Accomplishment” if they passed the class. Two of these courses (Widom’s Databases course, and Ng’s Machine Learning course) were taught on a platform developed by Ng and a team of students, which eventually became the genesis of Coursera. One course (Artificial Intelligence, taught by Sebastian Thrun and Peter Norvig) was taught on a platform led by Thrun, which eventually became the genesis of Udacity. Since then, many other MOOC organizations, such as edX, FUN, FutureLearn, NovoEd, Iversity, J-MOOC, and others, have been started to offer similarly highly scalable online courses. This article describes the origins of the three 2011 Stanford MOOCs. Because these MOOCs drew inspiration from many sources, it is difficult to trace the origins of every idea. We will try to do so where possible to give due credit, but will otherwise focus on the chronology of actions, rather than ideas, in the drive toward highly scalable models of education. Community of researchers The modern MOOC launched rapidly into public awareness in 2011, but contrary to popular opinion, MOOCs were not an “overnight success”: The idea of highly scalable education had taken years of germination, false starts, and experimentation, culminating finally in the three highly-visible Stanford offerings. These MOOCs drew from a wide variety of ideas developed by a community of Stanford and non-Stanford researchers interested in online education. Key members of the Stanford group included Daphne Koller, who had been experimenting with and evangelizing blended learning (the “flipped classroom”) at Stanford for several years; John Mitchell, who led a team developing an on-campus learning management system; Bernd Girod, whose students had developed sophisticated lecturerecording technologies; as well as Dan Boneh, Steve Cooper, Tiffany Low, Jane Manning, and Roy Pea, who contributed significantly to early discussions. Most MOOC platforms today actually pursue two major online learning activities simultaneously: (i) offering highly scalable forms of learning to large numbers of learners, and (ii) offering blended learning to small on-campus classes. The origin of participation by these platforms in blended learning stems mostly from the work of Daphne Koller and John Mitchell, who had experimented extensively with “flipped classroom” teaching in Koller’s on-campus course at Stanford. Outside Stanford, Ng and Thrun both credit Sal Khan (khanacademy.org) as a huge source of inspiration. Ng was also heavily influenced by the work of lynda.com, and the community/forum design of StackOverflow.com. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

The drive to scale What made MOOCs “revolutionary” was their availability and, especially, their scalability. One of Stanford’s first attempts to offer scalable forms of education was started by Andrew Ng in 2007 together with the Stanford Center for Professional Development. About ten Stanford courses were videotaped, and posted online together with lecture notes and self-graded homeworks. Called the SEE (Stanford Engineering Everywhere) project, it offered a similar experience to MIT’s OpenCourseWare, except a driving tenet in SEE was that every course should offer a “complete course” experience: It should have a complete set of materials - including lectures, homework problems, and solutions - that the learners could work through by themselves. Even though the technology was primitive, SEE videos came to be viewed by millions, and their success inspired Ng and others to spend the next few years developing and iterating on different versions of online education technology. In the intervening years after SEE was launched and before the 2011 MOOCs, Ng launched a number of experimental websites, all designed with the explicit goal of giving every learner a good experience, no matter how many people participated. Screenshots of some of these websites are included below. These websites experimented with ideas such as tablet recording, short-format videos, accelerated video playback, programming exercises, user-generated content (specifically, a Wiki that anyone could edit), and in-video quizzes as inspired by Daphne Koller's blended-learning experiments. Many computer science faculty members contributed to these ideas, and it is difficult to Screenshot 1: Stanford Engineering Everywhere. give credit to them all, but notable for his (http://see.stanford.edu/) involvement was Dan Boneh, who contributed significantly to the design of the tablet recording setup, and even wrote code himself for OpenClassroom, one of Ng’s early platforms (openclassroom.stanford.edu). Boneh and Jennifer Widom (together with Ng) were among the most ardent proponents of using Khan-style tablet recordings, and contributed many of the little “techniques” that are now commonplace. Bernd Girod meanwhile developed a platform called ClassX for do-it-yourself lecture recording, which used computer vision for automatic slide synchronization and tracking of the lecturer. By allowing learners to pan-and-zoom in the video while it was playing eliminated the need for a camera operator. Several hundred hours of lecture videos self-recorded by Stanford instructors were uploaded to the system between 2009 and 2011. In this period, the Stanford teams made progress but also had several false starts. We learned lessons about both what would and what wouldn’t work. For example: • We tried recording lectures in a classroom, with the instructor lecturing as usual to a room full of students, and editing this content later to create video segments each around five to ten minutes long. We learned that this approach nearly always resulted in a worse learner Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

•

experience compared to having the instructor explicitly plan out each five to ten minute segment. Working with a high school, Ng experimented with website features that support a team of three to five physically co-present students sitting at a computer working on problems together. This use case was very unpopular as most students preferred to watch videos by themselves so that they could control the pacing. Even today, we are still uncertain how best to support physically co-present learners watching videos. During this period, we spent a significant amount of time fine-tuning the set of tools for inexpensively creating MOOC videos, taking inspiration from Salman Khan and from Lynda Weinman’s work.

Screenshot 2: OpenClassroom website, version 1. Designed for use within a high school class, with a high school teacher facilitator. (http://openclassroom.stanford.edu/ai/shcp.php)

Screenshot 3: OpenClassroom website, version 2, with content on a variety of computer science subjects. (http://openclassroom.stanford.edu) Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Screenshot 4: Machine learning (deep learning) tutorial site -- self-paced course in Wiki format. Later also gained community translations of content. (http://bit.ly/deeplearn)

Screenshot 5: ClassX system by Bernd Girod. System allows pan/zoom during playback, and features slide synchronization. (http://classx.stanford.edu/) The three 2011 MOOCs In 2011, the three inaugural MOOCs were launched. They came to be known affectionately by students as ml-class, db-class, and ai-class from the website URLs (ml-class.org, db-class.org and ai-class.com); they were on the subjects of machine learning, databases, and artificial intelligence. Ai-course was the first course to be announced, and ran on a platform developed by Sebastian Thrun, David Stavens, Mike Sokolsky, and a few others. The course featured short-format videos, in-video quizzes, multiple-choice quizzes, subtitles, and subtitle translations. Ml-course and db-course ran on a platform developed by four students (Jiquan Ngiam, Frank Chen, Chuan-Yu Foo, and Yifan Mai) working with Andrew Ng. These courses featured short-format videos, accelerated video playback, subtitles, in-video quizzes, weekly multiple-choice quizzes, machine-graded Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

programming exercises, and a discussion forum in which learners could post and answer questions, and vote questions and answers up or down. Inspired by earlier work of Jeff Ullman and his Gradiance system, ml-class and db-class further made extensive use of randomized quizzes, and of “mastery learning” in which students are encouraged to attempt an assignment multiple times until they get it right. Ng's team developing the ml/db-class platform set “Do What’s Best for Students” as a guiding principle, a mantra that has continued at Coursera.

Screenshot 6: The machine learning course (ml-class) taught by Ng in 2011.

Screenshot 7: The databases course (db-class) taught by Widom in 2011. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Launching these free online courses generated a substantial amount of policy discussion within Stanford. The issue of whether completion “certificates” could be offered may have sparked the most discussion. Ultimately, in a series of meetings with the University Registrar, Legal Office, and the School of Engineering Dean’s office, Ng and Thrun secured permission to offer “Statements of Accomplishment” instead. This term has stuck, and today, over 1,000,000 Statements of Accomplishment have been issued across multiple platforms. When building out these early MOOC platforms, one lesson we learned repeatedly was that if a piece of software is not designed from the outset to scale to a huge number of learners, then it will probably not work in a MOOC. The pedagogy of effective MOOC teaching is also very different from on-campus instruction, and one has to constantly keep scale in mind. It isn’t simply an issue of whether the software will stand up under the onslaught of 5,000 simultaneous users; rather, it pertains to the fundamental design of the platform and pedagogy. For example, in a class of 100 students, almost any discussion forum design will work, since each student could conceivably read what everyone else writes. But in a class with 100,000 students, unless the user interface is designed from the outset to handle the correspondingly larger volume of discussion, then there will be many repeated/redundant comments, making the whole discussion forum unusable. The ml-class/db-class platform modeled its discussion forum after StackOverflow.com, a website for discussion about software programming, which Ng had frequently used in his work. The ai-class platform initially did not have a discussion forum, but a group of students launched a separate forum site (aiqus.com), which subsequently came to be the de facto course forum, used for much of the course communication. Key to any scalable discussion forum is the ability to vote posts up or down, so that the most helpful questions and answers quickly bubble to the top of the list, and so that spam can be quickly rejected by the community. Word about the three courses in the summer of 2011 spread mostly via social media and the popular press. All three courses started on the same day, and ran for ten weeks. In the end, around 7,000 learners completed db-class and earned a “Statement of Accomplishment;” 13,000 learners completed ml-class; and 23,000 completed ai-class. After the initial MOOCs With the successful launch of the initial MOOCs, Ng and Thrun both started to explore ways to move their projects forward. Ng recruited Jane Manning to Stanford to help with faculty support, and began recruiting a number of faculty colleagues at Stanford and at UC Berkeley to teach MOOCs. Some of the early Stanford faculty members recruited included Dan Boneh, who taught a very successful cryptography course, Daphne Koller, and Scott Klemmer, who was later instrumental in helping to develop a system for peer grading. Dawn Song at UC Berkeley also played a significant role in thinking through possible paths forward. Bernd Girod, Daphne Koller, and John Mitchell were also key players during this period; indeed, Girod, Koller, Mitchell, and Ng had also previously collaborated on sharing resources across their earlier platforms (Girod’s ClassX, Mitchell’s Courseware, described below, and Ng’s OpenClassroom). Soon after, Koller joined forces with Ng and his team to form Coursera, while Thrun turned his project into Udacity. Today, Coursera seeks to be a platform that works with existing universities, while Udacity seeks to build the university of the future. Mitchell and Girod both took administrative leadership positions at Stanford around online education.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Blended learning Most modern MOOC platforms such as Coursera, Udacity, and edX support two related but distinct visions: scalable forms of education, and blended learning for on-campus teaching. The latter of these visions stems greatly from the work of Daphne Koller and John Mitchell at Stanford during the period 2008-2011. For many years, John Mitchell had been developing Courseware, a new LMS (learning management system) for use at Stanford and elsewhere. Although not designed explicitly to scale, the system had many features that served as inspiration for the early MOOCs, including machine-graded homework, a discussion forum, and sophisticated instructor dashboards.

Screenshot 8: Courseware website, built by John Mitchell and used for Koller's early blended learning experiments. Daphne Koller was able to use Mitchell's platform to experiment with her early ideas on blended learning. In 2010, Koller was inspired by a talk about YouTube, and realized that on-campus education can be more engaging and make better use of student and faculty time if students learn the core material online at home in video format, leaving class time to discussions or experiential activities. This set of ideas, now widely known as the “flipped classroom" was little known at the time. Koller started videotaping her own lectures and hosting them on Mitchell’s LMS, in order to "flip" her own classroom. In creating her videos she also invented the idea of “in-video quizzes,” which today remains one of the best-loved features of many MOOCs. Her on-campus course also included a number of auto-graded interactive assessments, and used Courseware’s discussion forum. Koller and Mitchell worked together to further his platform to experiment with blended learning in on-campus settings, for example, creating designs for instructor dashboards to assist just-in-time teaching. Some of these features have also served as inspiration for various MOOCs. Even though blended learning hasn’t received the broad popular attention that MOOCs have enjoyed, they remain a key motivation for universities to partner with platforms such as Coursera and edX. In Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

early 2012, when universities were joining the MOOC movement, some were joining more to experiment with blended learning than with MOOCs. Summary Even though the MOOC movement appears to have emerged suddenly in 2011, the developments and inspirations behind the movement were many years and people in the making. A huge community of Stanford and non-Stanford personnel contributed ideas, and there were many years of quiet germination and iteration before the courses and course format were ready to “go viral.” Perhaps ironically, none of Ng, Thrun, or Widom had heard the term “MOOC” at the time they were launching their initial courses, nor were they aware at the time of George Siemens and Stephen Downes’ groundbreaking work on “connectivist MOOCs” (cMOOCs). We are still uncertain who subsequently first used the term “MOOC” to describe these three courses. The term xMOOC has since been proposed to distinguish them from cMOOCs. This article covers only the origins of the modern MOOC at Stanford, through the end of 2011. Many, many people at numerous institutions have since contributed significantly to building up the MOOC movement, which today continues to accelerate. We are thrilled at the innovations and cultural changes that the three 2011 courses have sparked. Scalable forms of education give instructors the opportunity to have a much greater impact on the world than was ever possible before. We eagerly await what universities, instructors, companies, and others will invent in the future.

Acknowledgments: We are grateful to Daphne Koller for her input on the evolution of blended learning at Stanford, and to members of the ai-class team (Sebastian Thrun, David Stavens, Mike Sokolsky and Peter Norvig) for discussions about their course. Bernd Girod also provided many helpful comments.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Populations served Most MOOCs have been developed from existing undergraduate level courses. They are available to any member of the public who is suitably motivated and informed about the availability of such opportunities, who has access to high bandwidth Internet, and can read and comprehend at the appropriate level. The vast majority of MOOCs have been offered in English but this is rapidly changing as foreign universities develop their own courses and the major platform providers develop partnerships abroad to facilitate translation of existing courses and the creation of new, local-language MOOCs. In practice, the majority of MOOC participants are already well-educated with at least a B.A. degree. A few community colleges have been creating developmental education MOOCs targeted at a different set of learners compared to those enrolling in the typical xMOOCs offered by elite institutions. Specifically, they are aiming to serve struggling students who would normally need one or more semesters of developmental education courses before being eligible to enroll in credit-bearing courses that lead to a degree. Some MOOC offerors are targeting distinct groups of professionals. For example, North Carolina State University, New York Institute of Technology, and the American Museum of Natural History have each created MOOCs specifically targeting K-12 teachers and administrators. edX has been developing “white label” courses for corporations and Executive Education courses for which they will charge $500 per participant. In late 2013, Udacity announced that it would shift focus to work with corporations rather than with institutions of higher education (Chafkin, 2013). Enrollments and completion rates While xMOOC enrollments vary from a few hundred to over 150,000 per course, they regularly run into tens of thousands. cMOOCs have generally been smaller, with participants numbering in the hundreds and low thousands. However, many of our interviewees questioned the value of enrollments as a metric of success because of the ease of enrolling and the absence of costs or negative consequences for failing to participate actively. I don't think that the number of enrollees means anything other than that you successfully advertised the course, you got somebody to click on it. I think a much better criterion is, just the same as for my on-campus elective class, whether students who complete the second assignment pass the course. In our MOOC [Mechanics ReView] last summer, 75% of the people who attempted half of the second assignment earned certificates. My dean would be happy with that. (Pritchard, MIT). Completion rates for cMOOCs have not typically been reported because there is rarely a required set of activities for the open enrollees and products created are not graded in the traditional format. On the other hand, because xMOOCs involve a clearly defined set of activities with a precise grading structure, completion is easy to assess. Completion rates for courses offered by our interviewees ranged from around 3% to 15% of all enrollees. As an example, Pritchard’s MIT Mechanics ReView has been offered three times, twice on the LON-CAPA platform and most recently on edX. At first offering in Spring 2012, 2,500 participants enrolled and 67 completed the course, a 3% completion rate. In Summer 2012, the course was advertised more selectively to attract a specific audience of teachers and 800 people enrolled. Certificates were issued to 117 participants, yielding a 15% completion rate, with 90% of the completers being teachers. The edX offering attracted 15,000 enrollees and 1,030 of them earned Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

certificates, yielding a 7% completion rate. Some courses offer verified certificates of completion, for which participants must pay up front. The completion rate for the small subset of participants taking up this service is high. For example, in three Coursera MOOCs for which we obtained the relevant data, less than 1% of the enrollees signed up for Signature Track, but, of these, between 90% and 100% completed the course. It is unclear whether there is a trend towards lower or higher enrollment or completion for repeat offerings of a MOOC. We heard evidence pointing in both directions, but it is likely that this is largely controlled by varying marketing efforts and the narrowness or breadth of the audience targeted, as illustrated in the Mechanics ReView example. A systematic study of this question as more repeat offerings are made would help determine the long term stability of MOOCs and help institutions and investors in the platforms assess whether MOOCs are worth the up-front investment of time and other resources. Mark Lester, Head of UK Partnerships at FutureLearn, suggested that if participants could browse MOOC content without registering, only those genuinely interested in participating would enroll and the resulting completion rates would be higher. Joseph Jay Williams, Research Fellow at the Lytics Lab, Graduate School of Education, Office of the Vice Provost of Online Learning, Stanford University, suggests that inclusion of a mandatory pre-course survey would also serve to improve completion rates by deterring “browsers,” but, as several of our other interviewees pointed out, such open exploration has value in and of itself. Credentials and fees For participants successfully completing a MOOC, various institutions and platform providers have experimented with awarding credentials such as certificates of completion, “Signature Track” certificates, “Verified Certificates of Achievement,” Mozilla Open Badges, and American Council on Education (ACE) credit recommendations. Since their inception, both edX and Coursera have provided certificates of completion, often signed by the instructor, to MOOC participants who meet certain criteria such as a minimum level of performance on course assessments. This basic credentialing is generally free of charge while other options carry a fee. Signature Track: In January 2013, Coursera introduced the Signature Track service which provides identity verification for MOOC participants and verifiable electronic certificates for a fee of $30-$100 (see Fain, 2013). Anders (2014) reported that 200 of 584 Coursera courses offer this option, as of January 2014. Learners can also earn a Specialization Certificate, by completing a series of three to nine related Signature Track courses and a capstone project, for fees totaling around $250-$500. Similarly, edX is offering Verified Certificates of Achievement for $25-$50 for participants who agree to be supervised via web camera, and three sequences of courses that can earn an XSeries Certificate. According to Agarwal, CEO of edX, participants who pay for these certificates often donate an additional 10% to support the edX mission. Among the 29 MOOC-offering institutions we engaged with in this study, only two, the American Museum of Natural History and Vanderbilt University, had pursued the Signature Track option for one or more of their MOOCs by the end of 2013. Others were considering this as a next step but noted that it would involve solving administrative issues related to collecting revenues from non-matriculated course enrollees. Mozilla Open Badges: One innovative idea implemented by Cuyahoga Community College was that of using Mozilla Open Badges as part of the game mechanics in their Pre-Algebra CCC MOOC. Christina Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Royal, Chief Academic Officer and Provost at Inver Hills Community College, believes that badges can help motivate students in attaining their educational goals, and could help create a better linkage between student preparation and employer expectations. American Council of Education (ACE) Credit Recommendations: Since the 1940s, ACE has been providing credit recommendations for formal learning that does not take place in a college or university setting by identifying alignments with competencies instilled via credit-bearing courses. ACE began its service with military education and training and has expanded its evaluation and recommendation services to government agencies, corporate entities, and non-accredited education providers. ACE credit recommendations can act as a form of transfer credit if a college or university is willing to accept them (see Sandeen, 2013). Around 2,000 institutions across the U.S. consider ACE recommendations for credit. In February 2013, Coursera announced that ACE’s College Credit Recommendation Service had evaluated and recommended undergraduate college credit for four MOOCs: Pre-Calculus offered by University of California, Irvine; Introduction to Genetics and Evolution and Bioelectricity: A Quantitative Approach from Duke University; and Calculus: Single Variable from the University of Pennsylvania (see Kolowich, 2013c). In addition, University of California, Irvine’s Algebra MOOC was recommended for developmental math vocational credit. Participants in these MOOCs who wish to take advantage of the ACE credit recommendation pay Signature Track fees and must complete an online exam supervised by ProctorU. Coursera advertised the option to obtain ACE credit recommendations on their website and through e-mail announcements to the relevant MOOC participants. At least initially, uptake for this service has been minimal. According to Melissa Loble, formerly Associate Dean for Distance Learning at University of California, Irvine Extension, only 39 of 80,000 participants across the two MOOCs offered by the University with an option for ACE credit recommendations signed up for the service. Of these, 38 successfully completed the MOOCs. In Fall 2013, Pre-Calculus was offered for a second time and 37 enrollees requested ACE credit recommendations. Beth Smith, President of the Academic Senate for California Community Colleges, cautioned that credit recommendations for MOOCs, while potentially beneficial for community college students, may cause confusion if, for example, a student invests the time and effort to complete a MOOC which carries the same title as a regular course, but is only recommended for half the credits because it does not address equivalent content. Credits: Some institutions have already offered or are contemplating offering credits for MOOCs or MOOC-like courses, at least to a subset of participants who may, for example, already be enrolled at the institution. San José State University (SJSU) partnered with Udacity to offer several for-credit, online courses in Spring and Summer 2013. This “SJSU+ Augmented Online Learning Environment” pilot limited for-credit enrollment to 100 students per course, who each paid $150 per three- to five-credit course. An additional 10,000 or so participants enrolled in each course directly via the Udacity platform for free and without receiving credit. University of Oklahoma has developed online courses which count for credit for the University’s own students, but are also open free and without credit to non-matriculated students. Matriculated students pay tuition equivalent to the on-campus courses. Georgia Tech’s MOOC-based Online Masters of Science in Computer Science (OMSCS) program, being developed in partnership with Udacity, offers courses for full credit, at $134 per credit (see Case 5). A few institutions, including University of Maryland University College, Georgia State University, and Penn State World Campus are exploring ways to award credits to MOOC completers who are able to Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

demonstrate competencies equivalent to on-campus offerings. (see Box 3 for Georgia State Universityâ&#x20AC;&#x2122;s approach to this). We look at MOOCs as a great opportunity for informal learning. If the provider, either at the college, or Coursera, or Udacity has gone the extra steps of having ACE accreditation reviews of their MOOCs, then we will transfer and accept that content, providing our student shows evidence of competency. (Vignare, University of Maryland University College). Through its MOOC2Degree initiative, Academic Partnerships is developing MOOCs with partner universities to serve as entry points into full online programs. Students who complete the initial MOOC and enroll in the full program at the partner institution can obtain credit for the MOOC towards their degree program, subject to criteria established by the university. (See Case 4 for more details on MOOC2Degree). Offering credits for MOOCs can be contentious because, unless the MOOC is as rigorous as an oncampus equivalent, there is a risk of diluting the brand of the offering institution. Furthermore, unless participants pay equivalent tuition, it will undercut the institutionâ&#x20AC;&#x2122;s core business. Several interviewees asserted that credits should not be offered for MOOCs unless they provide the same rigor in learning, assessment, and identifiability of participants as face-to-face courses. A few observed that rampant cheating in online courses serves as a major deterrent to offering credentials. Platforms used Most of the xMOOCs offered by our interviewees or their institutions were offered via the Coursera and edX platforms, with a small number offering courses on both platforms. Other platforms mentioned included Blackboard CourseSites, Blackboard Collaborate, Udacity, Desire2Learn, Google CourseBuilder, NextThought, LON-CAPA (Learning Online Network with Computer-Assisted Personalized Approach), and Canvas Network. By being selective about which institutions may deliver courses using their platforms, Coursera and edX have successfully leveraged existing institutional brands to create their own brands, which draw the attention of millions of MOOC participants. However, some faculty members and institutions are choosing different platforms which provide additional functionalities. For example, Canvas Network by Instructure provides more options for math quizzes, allowing answers in the form of equations in addition to the multiple choice format common to other platforms. NextThought has developed a sophisticated structure to facilitate peer learning in contrast to the typical rudimentary discussion forums. Beyond our study sample, MOOC.List provides a directory of MOOCs offered by around 46 different providers, with Coursera, Saylor, Canvas, and edX courses being the most ubiquitous. Instructors offering cMOOCs cobbled together their own delivery systems by combining a variety of tools. For example, Downes and Siemens used Blackboard Collaborate and gRSShopper (see Box 11), a tool created by Downes to aggregate feeds from various social media tools. Matthew Martin, Professor of Communication Studies at West Virginia University, and colleagues used Drupal, an open source content management system, in conjunction with freely available social media resources.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Box 3: Accepting MOOCs for Credit: Georgia State University According to Georgia State University’s Complete College Georgia, 2013 Status Report, the University is working with the University System of Georgia to develop “principles and procedures for the support and credentialing of innovative means of learning, including MOOCs.” Timothy Renick, Vice Provost and Chief Enrollment Officer at Georgia State University, explains the thinking and practicalities behind this move:

“What we were saying about MOOCs is no different from an offer we make to students across the

board, which is to say: whether you are a vet coming in from military service, an adult learner, or an AP student, you can make the case to us that you have certain competencies. One reason that this is relatively non-controversial on campus is because we are serving students that are low-income and struggling to pay their bills to get through college. We are trying to make it easier for them to do so. So MOOCs allow a less expensive and more flexible avenue to getting credit. Ninety percent of our students have jobs, so we want to keep open opportunities for self-paced learning and learning outside the classroom. We don’t automatically take ACE recommendations, despite the fact we are trying to make things easy for our students. Our faculty are reluctant to turn the ability to determine what the functional equivalent is for a Georgia State course over to an external body. We do this all the time for things like AP credit, we have standard tables and if the AP people say that the student got a 4 then we give them college credit for that course. We are developing mechanisms of various sorts. In some cases we have found that there are standard, well-accepted, and well-established national exams that exist for certain subject matter and we do offer CLEP exams through our testing centers so students can come if they are claiming competencies in an area where there is a CLEP exam. We also have some academic departments that, long before MOOCs, developed their own measures, often exams, or interviews, or project presentations, but mostly exams. In cases where there is no CLEP exam or home-grown exam, we encourage the departments to develop a measure for these students and others that come afterwards. The takers of our offer to consider MOOC-based competencies have been few yet - just a couple - so the workload for the departments has been relatively light. But I think there would be a workload issue down the road if this was to become the norm and hundreds of students came in to say they had these competencies. We would face pressure to find a way to resource that because right now it is fairly intensive work to develop the instruments to measure whether the student has the competency. But I could envision a future where, if this continues, this sort of assessment becomes not unlike what we do with credit transfer articulation. Right now to transfer credit, especially from outside the state, is very work-intensive. It requires someone in the department sitting down with the syllabus of the course, making a judgment about how that relates to other courses the department currently offers, finding an equivalency, then filling out the paperwork electronically to let the central office know it has been approved. Then it is easy for subsequent students who take that particular course and want to transfer the credit to Georgia State. So something similar could happen in this space where if we develop good measures, we could even share those with other institutions in the state that have similar core courses.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

At the moment, students do not have to pay for this service but we are having a senate subcommittee look at that. Because the space is small we haven’t been pushed that way yet. They do have to pay something like $50 to take the CLEP exam, but we don’t have a structure yet if they want to take some sort of graded assessment in these other areas, but we will develop some model and I expect it would be about the same or maybe a bit more because it has to be personalized more. If the workload becomes significant we’d have to look at some release time or some kind of compensation for faculty working on this.” Source: Timothy Renick, Vice Provost and Chief Enrollment Officer, Georgia State University.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

MOOC Derivatives As MOOC producers experiment with various design aspects of these courses, derivative course structures with associated acronyms have proliferated. We heard about several of these from our interviewees, but many others have also been written about in the popular press and many more are likely to arise in the future. Examples include: MOOC-Ed: These MOOCs are specifically targeted at teachers and offer professional development. SPOC: Small Private Online Course (see Fox 2013). These closed courses implemented, for example, by Fox and Patterson at the University of California, Berkeley, integrate MOOC materials into an on-campus course, allowing local faculty to incorporate activities that enhance the learning experience for their students. “Wrapped” MOOC: Similar to SPOCs, some instructors have re-designed their courses around a MOOC offered by another institution, “wrapping” the MOOC with their own content, activities, readings, and assignments (e.g., Doug Fisher, Director of the Vanderbilt Institute for Digital Learning, Vanderbilt University, see Fisher, in press). A wrapped MOOC differs from a SPOC in that the former enables students to participate in the global learning cohort of the MOOC, as well as participating in their local cohort (see Fisher and Fox, 2014, p.23). In contrast to flipped classrooms, MOOC materials may be incorporated into class time, rather than being assigned as homework. “White label” MOOC: These courses offered by edX may have massive enrollments but are only available to employees of a particular company or to members of an organization. Mini-MOOC: These are online courses that are openly offered using technology and pedagogical features that can accommodate a large number of students, but in practice enroll only a few. An alternative use of the term is to describe very short, educational experiences delivered online to large numbers of participants, akin to a webinar. DOCC: Distributed Open Collaborative Course. The DOCC concept was created by FemTechNet in reaction to MOOCs. The DOCC involves students and instructors from multiple institutions who engage in networked learning through a collaborative open course (see Jaschik, 2013 and DOCC 2013). Participatory meta-MOOC: In January 2014 Cathy Davidson at Duke University launched a MOOC via Coursera entitled History and Future of (Mostly) Higher Education. Simultaneously, three related face-toface courses were offered at Duke University; University of California, Santa Barbara; and Stanford University. The MOOC was developed as part of the FutureEd Initiative sponsored by HASTAC, a virtual learning network. Davidson labeled the course a “meta-MOOC” because, among other topics, it addresses the possibilities and pitfalls of MOOCs in Higher Education. SMOC: Synchronous Massive Online Course. The SMOC was introduced by the University of Texas at Austin, which offered Introduction to Psychology to on-campus and non-enrolled students simultaneously. The instructors lectured live on the Internet twice a week. POOC: Personalized Open Online Course. POOCs that offer adaptive learning are more of an idea than a reality at present. The success of these courses will depend on collecting and processing massive amount of data from the students in order to personalize the learning based on individual progress. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

How and Why are Institutions Engaging with MOOCs? Colleges and universities have adopted several different stances towards engaging with MOOCs. Some are actively developing MOOCs and may be termed “producers,” some are using MOOCs developed by other institutions in their programs and could be termed “consumers,” and a few are doing both. Others are adopting a “wait-and-see” approach, and some have considered MOOCs and have either decided against any form of official engagement, or have not met with interest from faculty members to pursue them. While many people assume that MOOC production is limited to “elite” universities, we found this not to be the case. While it is true that MOOCs have “catalyzed big brands into what is fundamentally a distance learning game” (Lester, FutureLearn), other institutions, including a few community colleges, have concluded that they know how to serve their particular students best, and have created MOOCs tailored to the needs of their own student populations. “Consumers” of MOOCs are integrating MOOCs created by other institutions into their course offerings in flipped classrooms (e.g., San José State University), or as wrapped MOOCs (e.g., Vanderbilt University), or simply as supplemental resources for their students. Instructors at the University System of Maryland are experimenting with a variety of approaches to embedding MOOCs created by others into their classes (see Griffiths, 2013; Ithaka S+R, 2013). A small number of institutions have declared willingness to consider MOOCs for credit (e.g., Georgia State University, University of Maryland University College). A few universities such as Vanderbilt University and University of California, Berkeley could be categorized as both producers and consumers, with some instructors creating MOOCs and others using MOOCs created by their own or other institutions in their classes. Of the 39 colleges and universities that participated in this study, 28 had already offered a MOOC or integrated MOOCs into their courses, or were in the midst of doing so. Most fell into the producer category and were developing xMOOCs, with only a small fraction offering cMOOCs. Institutional approaches varied from tightly coordinated and controlled, top-down efforts in which senior administrators hand-picked academic departments and instructors to offer MOOCs; to more democratic processes in which a committee solicited and reviewed proposals from faculty members; to more grassroots approaches where the administration took no official position but individual instructors developed and offered MOOCs of their own accord. In some cases the institutional approach towards MOOC production evolved rapidly from an ad hoc process to establishment of formalized teams dedicated to various functions such as videography, instructional design, course support, and contractual negotiations. Appendix III summarizes the MOOC-related activity of each institution we engaged with, except where an interviewee requested anonymity. Our interviewees identified a variety of institutional goals for engaging with MOOCs which we organized into six categories as shown in Table 2 below: extending the reach of the institution and access to education; building and maintaining brand; improving economics by lowering costs or increasing revenues; improving educational outcomes for both MOOC participants and on-campus students; innovation in teaching and learning; and conducting research on teaching and learning. In subsequent sections we address each goal in turn, describing how the goal is being addressed, any evidence of success, and obstacles encountered.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Table 2: Institutional Goals for Developing and Delivering or Using MOOCs Number of institutions offering /using MOOCs stating this as a goal

(n=29)*

Private Universities (n=10)

Public Universities (n=15)

Community Colleges (n=3)

% of all interviewees who raised this as a goal (n=83)

Extending Reach and Access

65%

42%

Building and Maintaining Brand

41%

25%

Improving Economics

38%

29%

Improving Educational Outcomes

38%

20%

Innovation

38%

19%

Research on Teaching and Learning

28%

18%

Institutional Goal

% of total

* Includes one museum in addition to the universities and colleges, explaining why two of the “% of total” numbers are higher than expected from counting the colleges and universities alone.

Our findings are somewhat consistent with other investigations regarding the goals of institutions offering MOOCs except that in two other studies branding appeared to be more important than improving access to education. A 2013 study, To MOOC or not to MOOC: Strategic lessons from the pioneers, conducted by Cathy Sandeen, Vice President, Education Attainment and Innovation, American Council on Education (ACE), and Dave Jarratt, Vice President of Marketing at Inside Track, identified similar goals to the ones we found from a survey of administrators at nine institutions of higher education, and a separate survey of 108 faculty members who had offered MOOCs. According to Sandeen, “A common motivation was reputational, and also for outreach and dissemination of knowledge; and another was to provide an incubator for new ideas.” These respectively fit with our branding, access, and innovation goal categories. Respondents in Sandeen and Jarratt’s study also mentioned the objectives of lowering costs, increasing revenues, and improving outcomes for students, but they did not mention MOOCs for conducting research on teaching and learning. Sandeen and Jarratt did not quantify their results so we cannot compare the frequency with which goals were mentioned by our respective samples. In their 2013 annual survey of online learning in higher education, Allen and Seaman (2014) asked representatives of 2,831 colleges and universities in the U.S. to indicate the “Primary objective for your institution’s MOOC.” Only five percent of their respondents had offered a MOOC and, among these, the most common objective was to “increase institution visibility,” selected by twenty-seven percent of this subset. The second most important objective, selected by twenty percent of the MOOC producers, was to “drive student recruitment.” Both these objectives fit within our second most popular category of building and maintaining brand. We cannot directly compare frequency with which each goal was mentioned because Allen and Seaman’s respondents were limited to a single choice of objective, while Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

ours could mention as many as they wished. Additionally, while the options offered in Allen and Seaman’s survey covered four of the goal areas we heard about from our own interviewees, they did not offer choices that would correlate directly with improving educational outcomes, or conducting research on teaching and learning. However, it appears that branding was most important to their sample while increasing access and reaching a wider audience was most important to ours. Very few of their respondents, less than three percent, selected the objectives of reducing costs or generating revenues. The Alliance for Higher Education and Democracy (AHEAD) at the University of Pennsylvania recently conducted a poll of administrators, faculty members, and other personnel at institutions of higher education in the U.S and found that, among the approximately 44 respondents at institutions offering a MOOC, 57% strongly agreed that MOOCs may be a potentially effective mechanism for “Raising Institutional Profile” (AHEAD, 2014), which we would equate with branding. Fifty percent of the respondents strongly agreed that MOOCs can help improve access around the globe, 40% that MOOCs can help improve access in the U.S., 34% that MOOCs can improve pedagogy, and only 19% that they can reduce costs. Interestingly, and tellingly, confidence in the potential for MOOCs to reduce costs was higher among respondents from institutions that had not yet offered MOOCs. For all other goal options offered by AHEAD, MOOC offerors appeared more confident in the potential for MOOCs to achieve the goals than MOOC abstainers. Steven Mintz, Executive Director of the Institute for Transformational Learning at the University of Texas System, highlighted differences in goals among members of the edX consortium, noting that for some members the primary goal is branding, “being relevant,” or helping solve the political problem that their enormous endowments are being spent on relatively few students, whereas “state systems like my own hope to use MOOCs to expand access and affordability, and to potentially use MOOCs for credit.” In general, for less wealthy institutions, return on investment is a greater priority. Several of our interviewees expressed frustration at the lack of agreement across administration and faculty members within their own institutions regarding the goals for engaging with MOOCs. We heard a number of reasons to explain why some institutions are not embracing MOOCs. Some do not accept the notion that mass delivery of education will allow for the quality of learning that can be fostered on a small scale. For these institutions, quality is a priority over quantity. Others doubt that MOOCs can help them fulfill their mandates, for example, Royal at Inver Hills Community College observes that "Through the lens of innovation, MOOCs are an important initiative with great potential. Through the lens of college priorities such as retention, persistence, and completion, they are probably not." Several institutions are simply being cautious and financially circumspect, letting the early adopters or “MOOC pioneers” spend the time and money to figure out what works so that they can adopt best practices later. A few are pragmatic in the realization that their brands and MOOCs cannot compete with those of elite institutions. Some are reluctant to give up significant amounts of control, content, and data to third parties, and are either planning to figure out how to offer MOOCs themselves, or are waiting for alternative consortia to coalesce in which they might wield more influence. Other institutions lack the resources for MOOC development and are hard pressed to justify cutting back other initiatives in order to divert resources to MOOCs. In particular, a few question how they will be serving their paying students well by expending such significant amounts of time and effort to educate the rest of the world, especially when most MOOC participants already have a college education and their own students are still trying to earn one. For example, a senior administrator at a leading UK university remarked: “We are quite dubious about conducting expensive experiments in the area of Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

education when we have our own students paying £9,000 a year for the education we provide them, as opposed to learners outside the university.” These concerns appear to be validated by the reports we heard from a few interviewees that on-campus students are already beginning to question why they are paying tuition to underwrite the production of MOOCs, and yet they cannot earn credit for them. In some instances, the college or university administration has not taken a formal position on MOOCs but individual faculty members, the equivalent of Moore’s (2014) “technology enthusiasts,” are experimenting on their own, either offering MOOCs they develop themselves without institutional resources and support, or integrating MOOC materials from other institutions into their courses. In these situations, the goals of individual faculty members are primarily to advertise their subjects and expand their reach to a wider audience. In a few cases, MOOCs serve as a source of data for research. Even when an institution does offer support for MOOC development, some faculty members prefer to act independently without being subject to institutional restrictions, and without being limited by the institution’s technology capacity, or lack thereof. Martin at West Virginia University also suggested that MOOCs offer a far less restrictive outlet for faculty members to disseminate their work and expertise than publishing companies which are increasingly demanding rights to content.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Goal 1: Extending Reach and Access Extending the reach of the institution to a wider audience and improving access to education was the most commonly identified goal for offering a MOOC, mentioned by 65% of the institutions in our study that were offering or using MOOCs, and 42% of our interviewees overall. These interviewees included administrators and faculty members from seven private universities, eight public universities, three community colleges, and a museum. The frequency with which this goal arose is consistent with the message offered by each of the three most prominent MOOC platforms. Coursera, edX, and Udacity each highlight in their mission statements access to a world class education for all as a goal of their enterprises: Coursera is an education platform that partners with top universities and organizations worldwide, to offer courses online for anyone to take, for free. We envision a future where everyone has access to a world-class education. We aim to empower people with education that will improve their lives, the lives of their families, and the communities they live in. (Coursera). Through our institutional partners, the XConsortium, we present the best of higher education online, offering opportunity to anyone who wants to achieve, thrive, and grow. (edX). Our mission is to bring accessible, affordable, engaging, and highly effective higher education to the world. We believe that higher education is a basic human right, and we seek to empower our students to advance their education and careers. (Udacity). Some institutions claimed to be pursuing an altruistic goal of reaching the masses globally with high quality educational experiences, while others presented more defined goals of reaching a specific population, such as providing continuing education for alumni, or solving a particular challenge related to access. For example, Martin at West Virginia University, offered a cMOOC on social media with several colleagues from the Department of Communication Studies with one goal being to provide continuing education to alumni who had not had opportunities to take courses related to social media. Reaching the masses Many interviewees spoke of MOOCs as a means to fulfill an instructional or personal mission to make education universally available: The reason I am so passionate about this is because I want the American dream sustained. I don't see how we are going to get there unless we up the level of knowledge, skills, and expertise that are broadly disseminated in the general population. I see MOOCs as a vehicle to do that. (Professor, East Coast University). We want to make education available globally. Ideally all people in all places would be able to get a quality education. We want to provide a course of [our institution's] quality to outside people. (Senior Administrator, East Coast University). We want to provide education for free to anyone who is interested....We like the idea of “The Walking Dead MOOC” because we feel it does what MOOCs initially set out to do, which is to provide education to anybody that could be interested at any level; and that it’s open and free education; and that it’s fun and interesting and not necessarily about credit. It’s about the experience. So we feel that if we can just get one more person to be excited and learn something Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

about a topic they haven’t known about before, we would consider it successful. (Melissa Loble, formerly at University of California, Irvine Extension). The University as a place-based institution has public engagement as part of its mission. We have wonderful events and speakers on campus and the ability to teach in a public mode. I think this is using technology in a radical way to extend that. (Harper, University of Oklahoma). I won't try to make the argument that MOOCs are a panacea, or applicable to all types of classes in all settings. However, for introductory engineering and science classes, I am comfortable asserting that it is an important aid in democratizing education, and particularly for our institution, a way for people who really want to get access to our education, to do so. (Development Officer, Highly Selective University). Part of Columbia's mission is to educate the world and spread knowledge. (Sreenivasan, formerly at Columbia University). Several instructors were motivated to offer MOOCs because of their passion for their subject and desire to make it accessible to the general public. The faculty member for the “Geospatial Analysis and Mapping” MOOC was very committed to having topography understood at a lay level, and why that's increasingly important in our world of digital positioning; why people need to understand more about maps. He had the personal quest of reaching thousands of people. He said at one point, "I'll be teaching in a single MOOC more people than I'll ever teach in my entire career." He was very motivated by that. (Lawrence C. Ragan, Director, The Center for Online Innovation in Learning, Penn State World Campus). My goal is to teach as many people as possible about viruses, which is unusual for a researcher....Between the blog, YouTube, iTunes, and Coursera I have massive involvement of people. It is really important that scientists reach out to the public with what they are doing. (Racaniello, Columbia University). (See Case 1). Improving access to community college through developmental education MOOCs Community colleges have a very specific goal of improving access for students who want to attend college but do not have the necessary preparation in math and English. Burck Smith, CEO of Straighterline, an online course provider, believes that MOOCs can offer a “no-risk failure model” for struggling students. Instead of increasing the cost to institutions by adding more remedial services, MOOCs may provide a no-risk, low-cost option that might encourage and motivate some at-risk students. A small number of community colleges including Mt. San Jacinto Community College, CA, Wake Technical Community College, NC, and Cuyahoga Community College, OH, have created and offered developmental education MOOCs with the aim of saving students time and money by setting them on an accelerated path to credit-bearing courses, and more timely completion of a degree. Some students wanting to attend community college do not have college-level math and keep failing the introductory level classes. They either cannot afford to retake the classes, or are prohibited from doing so by the new policies. If they can take a math MOOC and help get themselves to college level in math, they may be more likely to succeed first time in the creditbearing courses, especially as there is less pressure from being judged in a MOOC. So MOOCs can improve access and allow students another chance to succeed. (Online Learning Coordinator, Large California Community College). Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

To increase the likelihood that students would benefit from the experience, the designers of Mt. San Jacinto’s Crafting an Effective Writer MOOC included a week-long unit on “how to be successful as an online learner” (Patricia James, Dean of Distance Education, Mt. San Jacinto Community College, CA) and made 14 people available during the course to respond to student inquiries and requests for support. However, assessing whether these MOOCs reach their target population or help students test out of developmental education has not been easy because student identities in a MOOC are difficult to match against on-campus identification systems, thereby making it difficult to track the students’ post-MOOC course-taking trajectory. Not all interviewees were optimistic that MOOCs can help struggling community college students, with some pointing out that expecting students at the developmental education level to “self-remediate” may not be realistic. One of the challenges with developmental education and MOOCs is the level of guidance and support given. One of the lessons we learned is that even when you structure learning in a very methodical way, people inherently have different levels of motivation and desire to move forward at a certain pace. In general, we need to move towards more competency-based frameworks where students can move forward at their own pace as long as they complete the competencies, whether this takes two months or two years; and you pay tuition until you complete it. MOOCs have a framework that can be very helpful in supporting that, but there still has to be a level of guidance and support, where you just can’t have one faculty member teaching 10,000 students and expect that there will be a level of guidance and personalization at where we are at today. (Royal, Inver Hills Community College). Beth Smith, President of the Academic Senate for California Community Colleges, notes that students often believe that online courses are easier than face-to-face courses, and are not adequately prepared to do the necessary work. She also cautions that the amount of reading required in an online course can be problematic for some community college students. Shanna Jaggars, Assistant Director at the Community College Research Center (CCRC), Teachers College, Columbia University, similarly asserted that, to benefit from MOOCs, students must have relevant background knowledge and specific interest in the subject; be self-directed and strongly motivated to learn; and have strong skills in time management, metacognition, and information literacy. She cautions that this constitutes a small proportion of all community college students, while the majority requires more structure, guidance, and encouragement than is presently provided by MOOCs. Past research by Jaggars and others at the CCRC (see Online Education and Instructional Technology papers at CCRC) has not found online learning to be helpful in improving graduation rates for community college students in general, but a more recent study by Shea and Bidjerano (2014) indicated that students taking some of their early courses online were more likely to obtain a community college credential than students who took classes only oncampus. MOOCs to overcome resource constraints In addition to the generic goal of improving access, some institutions are utilizing MOOCs to overcome specific constraints in order to improve access to courses and degree programs. A few interviewees suggested that MOOCs can alleviate current infrastructure constraints for education both domestically and in rapidly developing countries where the existing physical plant cannot accommodate the growing demand for postsecondary education. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

For the Online M.S. in Computer Science program, a primary goal is to provide access to students who are not able to take on-campus courses, and to accommodate people we don’t have room for. Last year we had 1,400 applicants and over half were eligible. We could only accept about 140, so we felt there was an underserved population and we wanted to see how big that is. There was some concern that the online program would decrease demand for the on-campus program, but, in fact, the number of applications for Fall 2014 increased by about 30%, and the number of new students starting this Fall will probably be at least 40% greater than the Fall 2013 class. (White, Georgia Tech). I am on the Global Engineering Deans Council. In China, the number of students enrolling in college is going up almost 20% at a compound annual growth rate. Over 40% of these students are enrolling in engineering. You have all this incredible growth in the number of engineering students and nobody there to teach them. That is a need that MOOCs could address. (Peercy, University of Wisconsin-Madison). MOOCs can potentially ease the pressure on oversubscribed programs or courses. Catheryn Cheal, Ph.D., Associate Vice President and Senior Academic Technology Officer at San José State University, CA, noted that the MOOCs being developed by the university in conjunction with Udacity, such as Introduction to Psychology and Java Programming, are all “bottleneck” courses. Many students are blocked out of these required courses due to capacity constraints, and are delayed in making progress towards degree completion. A senior executive at a nonprofit, online learning company indicated that bottle-neck courses at state universities and community colleges may be the next target for MOOC production. We are looking at big growth in developmental curriculum, remedial curriculum, the lower-level courses that are log-jammed, that are pinch-points in the curriculum in community colleges and undergraduate education: statistics and pre-calculus, calculus, micro and macroeconomics - the courses with massive enrollments. We have started with high-end, elite courses and the next phase will be the courses where large, state-funded, land-grant systems cannot find space for all the students who need to take them to complete degrees. (Senior executive at a nonprofit, online learning company). Ann Kirschner, University Dean at Macaulay Honors College, NY, believes that MOOCs can increase access to instructors skilled in specialized domains and niche subjects. I think MOOCs will help because not all institutions can be expert in everything. So if students come to us to study a language or specialized part of physics for which we don’t have a faculty member, instead of going to Stanford for a semester they can take a MOOC. If the Amazon of MOOCs existed, I could tell a student to study George Eliot with Professor X at Cambridge, to supplement resources and to develop an education collective that could only exist in a networkdistributed 21st century world. (Kirschner, Macaulay Honors College). Kirschner also suggested that MOOCs can provide flexibility for students to create their own programs with courses from various institutions to best match their abilities and interests. Indeed, Coursera has recently begun offering certificate-bearing sequences of courses, and several universities offer similar course “series” (Kolowich, 2014). MOOCs, as with online learning in general, can improve access to education by providing flexibility in time and place of study, either for a commuter population oncampus or an off-campus audience beyond city, state, or national borders. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

I have been thinking about the best use of our students’ time. Could we take some of their requirements and move them online to give them greater flexibility? We have the special pressure of a commuter population. Could we find ways to give them flexibility and opportunities to interact online as students which they don’t have now? I am very interested in experimenting. (Kirschner, Macaulay Honors College). We are interested in growing an international audience - people who never actually come here. That’s a target population for the museum and this could be a good way to get them in. (Kinzler, American Museum of Natural History). MOOCs are also being used to address constraints of “traditional” online courses which are often restricted in size to the same degree as on-campus courses so as not to overextend the instructor. For programs like ours that have ambitions of growing larger, that's a real concern. You don't have the mechanisms to scale to large enrollments when you're considering every class can only have 20 students. Part of our motivation for going into MOOCs was to examine how can you serve large enrolling classes, efficiently, effectively, but at the quality that we want to brand as a Penn State quality experience. We were really looking at these MOOCs, I would say primarily with the question: can you teach online courses at what we call a Penn State quality experience, and have them reach fourteen thousand, or fifteen thousand, or forty thousand students? That's a real challenge. (Ragan, Penn State World Campus). Evidence of improving access through MOOCs Having established that improving access to education and reaching a wider audience was the most important goal across our sample of institutions, we review here the evidence that this goal is being achieved. It is clear that each of the major MOOC platforms has attracted a substantial number of participants: around two million unique users for edX as of March 2014, and over seven million for Coursera. In January 2014, Coursera reported over 22 million enrollments, with many participants enrolling in multiple courses. These enrollees were spread across 571 courses and were based in 190 different countries (see https://www.coursera.org/about/community). While these numbers are impressive, only a small fraction of those who register complete or even start a course. Ho et al. (2014), using data from 17 HarvardX and MITx MOOCs offered between Fall 2012 and Summer 2013, found that 841,687 people registered for the MOOCs but 35% of them never engaged with the content, and 56% engaged with less than half of the content. Around 43,000 or 5% of participants completed their courses. A typical registrant was “male with a bachelor’s degree who is 26 or older” (p.2). This finding is in line with Christensen et al.’s (2013) claim that 79% of MOOC participants have a bachelor's degree, based on a survey of participants in 24 MOOCs offered by the University of Pennsylvania on the Coursera platform. With respect to global reach, 72% of the registrants in the HarvardX and MITx MOOCs were from outside the U.S. and 2.7% were from countries on the United Nations’ list of Least Developed Countries (Ho et al., 2014). Among the 34,779 participants in the University of Pennsylvania sample, 31% were from non-U.S., OECD countries; 15% were from BRICS (Brazil, Russia, India, China, and South Africa); and 20% were from other developing countries. Foreign participation is rising as MOOCs are offered in languages other than English. For example, edX offers courses in French, Hindi, and Mandarin, and recently announced a partnership with the Queen Rania Foundation in Jordan to offer Arabic language Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

MOOCs via the Edraak MOOC portal. Earlier this year, Coursera announced a partnership with Fundación Carlos Slim to offer courses in Spanish. The data suggest that MOOCs are providing educational opportunities to many individuals across the world. However, most are already well-educated and only a small fraction of these participants fully engages with the courses. Among our interviewees, we found mixed views as to whether MOOCs have succeeded in improving access to their institutions’ offerings and reaching wider audiences. Some had no doubt that MOOCs had already expanded access to their courses: Using the technology in their courses has brought real benefits, both in terms of the effectiveness of the material and the reach of the course, the number of students they can serve without watering things down. (Fox, University of California, Berkeley). I will argue that we are now reaching hundreds of thousands of students a year, when previously we could only touch thousands of students. Also, for our subset of the education world, STEM, I personally argue that MOOCs are a valuable platform to extend our reach globally. Due to funding and other restrictions, we can only accept 10 percent international students into our undergrad program; now we have a way to reach those students. (Development Officer, Highly Selective University). Others acknowledged that the initial expectations that MOOCs would “democratize” education had not been realized: The idea of MOOCs is that we are going to educate the children in China and Africa. A surprising number of people in the Financial Engineering and Risk Management MOOC were people who work on Wall Street. A number of Ph.D.s were taking it too. (Sreenivasan, formerly at Columbia University). The argument for MOOCs solving the access problems of developing countries is that it is a solution looking for a problem, rather than the other way round: identifying what the best solution would be on an analysis of local conditions. It’s trying to fit a square peg into a round hole. (Bates, Contact North). We know that there is still something about MOOCs that isn't particularly helpful, or isn't particularly appealing, to people who have not gotten a degree. (Vignare, University of Maryland University College). Quantifying the impact of MOOCs offered by a particular institution on improving access to its courses is difficult unless institutions accurately document the level and diversity of participation in their programs both before and after offering MOOCs. For the most part, institutions were not yet making a concerted effort to document these potential changes. One exception was a large Midwestern university where an evaluator described the thinking about how to capture the effects of MOOCs on the University’s goal of expanding its reach: Our criterion for the second round is that the course must address and ideally solve a problem that would be relevant to people off-campus. We are taking a more targeted approach in what audience we are trying to reach, for example, people from the armed forces or college dropouts, and then this allows for a benchmark of success: whether we are able to attract that audience. Another goal is to increase enrollments. We draw well from certain states and high schools, but poorly from other areas and groups, so if we can start an interaction with those Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

people and understand their interests and how we could engage them, we think it will pay off longer term in revenues from new programs we create for them. There may be delayed benefits if they sign up for a program later so we would need to track this by asking what motivated them to sign up and whether they had participated in a MOOC we offered. (Evaluator, Large Midwestern University). At Georgia Tech, David White, Assistant Dean for Academic Programs and Executive Director of the Online M.S. in Computer Science (OMSCS), College of Computing, provided a solid example of how MOOCs can expand the university’s audience. He reported a significant difference in the types of applicants to the MOOC-style OMSCS program being developed in conjunction with Udacity, as compared with the usual applicants to Georgia Tech. While typical on-campus students do not work full time, 75% of the 2,359 OMSCS applicants for Spring 2014 were full-time working professionals, mostly in computer or Information Technology fields. White also reports a difference in geographical origin of the OMSCS applicants: More than 80% are U.S. citizens and permanent residents compared with 75% international students who apply for the on-campus program. Most U.S. applicants are probably people who cannot afford the regular program or cannot uproot from a job and family to move to Atlanta. We think that international students may prefer to apply on-campus because they are often looking to work in the U.S. and this allows them to get a student visa and do summer internships. (White, Georgia Tech). Stan Silverman, Professor and Director, Technology Based Learning, New York Institute of Technology, described success in using targeted MOOCs to improve teachers’ access to and use of specific technology tools. For example, he offered a short course for educators associated with New York State Teacher Centers on how to use Blackboard Collaborate as a pedagogical tool. Subsequently, he was able to monitor the number of requests for use of the tool and found that, compared with the usual number of requests, the post-MOOC uptake rate increased significantly, with more teachers using the tool in their professional work. Barriers to improving access through MOOCs There are many reasons why MOOCs have not fully met expectations with respect to improving access to education. According to Grossman (2013), most lay people outside of higher education are not yet familiar with MOOCs. Several interviewees pointed out that simply providing “free” educational resources does not necessarily make them accessible to everyone. The assumption of MOOCs is that we're now bringing content to the world for free, and everybody has the same access, so therefore we're giving everybody the same access to this high quality content. Yet we know from some of the work we've done in New York State and some other studies from around the world that the other side of the equation isn't the same. Not everybody has access to the Internet or devices to play on the Internet. Who's going to pay for that in this model, and make sure that everyone has access to this “information highway?” (Kolb, RPI). Dr. Tony Bates, Research Associate at Contact North, Ontario, observes that the end user in developing countries may experience both infrastructure constraints and cultural barriers to engaging in MOOCs. In many locations, Internet bandwidth is inadequate for MOOC participation which involves downloading materials and watching videos. Buying additional bandwidth is very expensive in countries like China and Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

may be prohibitive for would-be MOOC participants. Dr. Li Fengliang, Associate Professor, Institute of Education, Tsinghua University, China, observed that even his own university’s Internet access is fairly expensive and not very fast, which makes downloading materials and watching videos inconvenient. Adaptations have been made in certain countries to overcome bandwidth limitations. For example, according to Saad Rizvi, Senior Vice President, Efficacy, at Pearson, in Pakistan where the infrastructure outside of cities cannot support MOOCs, the government is partnering with FutureLearn and Coursera to transfer MOOC content to CDs. Rizvi also reported that ZayaLabs, a technology startup, provides tablets and a router loaded with MOOC content in a backpack to schools in India, Indonesia, and Mongolia. The set-up takes the content offline and allows participants to engage with it in a classroom in the absence of any wireless infrastructure. As platform providers such as NextThought and Coursera move towards mobile access to their content, some of the infrastructure constraints may be alleviated. Less easily addressed were some pessimistic views about the capacity of online courses, massive ones in particular, to reach any but the most dedicated students. Li Fengliang at Tsinghua University pointed out the most fundamental obstacle to MOOC participation: “I don’t think it’s easy for people to learn faceto-face, let alone through a MOOC.” Renick, at Georgia State University, observed that access to educational resources is only one part of the equation to guarantee success for the majority of college students, with the need to motivate students, track their progress, and guide them along a trajectory towards completion being equally important. Christopher Newfield, Professor of Literature and American Studies at the University of California, Santa Barbara, suggested that education is more complex than simply providing access to content: If you can get to an Internet café in your village, in a low-income or medium-income country... get access to interesting people, who are now on TV, or on YouTube....Going from that, to the interactive, cognitive, developing process, structured knowledge, and highly evolved and analytical processes, habits of thinking and habits of analysis that are robust and complicated and enable originality, that's a whole other thing. Anybody who's been teaching for a while knows that you can't just put it on the screen and get that. If that were true, then we'd be a nation of geniuses and all our problems would be solved, all our social problems as well as our scientific ones. (Newfield, University of California, Santa Barbara). Paul Sechrist, President of Oklahoma City Community College, similarly observed that there are limits to what can be taught online: “For example, the biotechnology research assistant program where they are teaching students how to work in a sophisticated biotechnology facility: you have to have a laboratory to practice those skills.” Regulatory barriers to using MOOCs for improving access to education In addition to logistical and pedagogical barriers to improving access to education through MOOCs, a number of regulatory issues also pose obstacles. For example, community colleges and state universities must carefully negotiate around regulatory and administrative issues related to access and state funding. Dr. Douglas Hersh, formerly Dean of Educational Programs at Santa Barbara City College, CA, noted the need for his own and other colleges to comply with the requirements of two accreditation agencies: Western Association of Schools and Colleges, and the Accrediting Commission for Community and Junior Colleges. In particular, these agencies demand that all students, whether on-campus or online, receive “equivalent student services” and “regular effective contact,” both of which are hard to satisfy with MOOCs. Hersh pointed out that it would be prohibitively expensive to provide student Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

services including academic counseling, library services, tutoring, and proctoring to thousands of MOOC participants. Beth Smith, President of the Academic Senate for California Community Colleges, raised other “unresolved issues” such as the fact that community colleges collect apportionment or state support based on student enrollments in their classes. As MOOCs are offered for free, it is unclear how this will affect the revenue received from the state. Will colleges have to differentiate between the on-campus students for whom the state is paying and the out-of-state or international students? Furthermore, Smith points out that the faculty member unions will need to agree on contractual issues regarding class size, teacher workload, and expectations for assistance from teaching assistants. Laura Kalbaugh, Dean at Wake Technical Community College, Raleigh, NC, also notes that while it might be desirable to allow students to earn credits for MOOCs, articulation agreements with universities would need to be rewritten to ensure that students could transfer these credits to the four-year colleges. Additional issues will arise if MOOCs become part of the K-12 education world. Lawrence Paska, Director of Social Studies, Harrison Central School District, Harrison, NY, and a former State Education Agency Technology Policy Coordinator, observes that: “In K-12, you have to remember that local taxes pay for local educators. The problem for K-12 will not be about the technology, but the barriers will be [related to] how adults have organized education.” Paska raises several questions that will need to be resolved including: • • • • • • • • •

How will cross-state teacher certifications be managed? Who will the teachers work for and how will they be paid? Will teachers become “for hire” or independent consultants? If so, who would evaluate an independent consultant? What will be the teacher’s rights and responsibilities? Who will pay for the course content and for the enrollees? Will federal and state aid be available for these courses? How will a teacher’s course-load change from the current 150 students? Will MOOCs allow for credits and, if so, who is certifying the credit and how does this work across districts and states?

Regulatory issues also arise in offering MOOCs internationally. For example, in order to comply with U.S. Department of State economic and trade sanctions regulations, Coursera and Udacity recently restricted access to their courses in Cuba, Iran, and Sudan (see Curley, 2014). It appears that despite a declared commitment to providing global access to free education, many obstacles remain in fulfilling this goal. Victor Vuchic, formerly Program Officer for Education at the William and Flora Hewlett Foundation, who questions Silicon Valley’s “blind faith that technology helps everybody,” offered a verdict on how well MOOCs improve access to education: I read this article a year ago when the first MOOCs came out; a journalist said “My 9th grade son took two Stanford MOOCs and completed them.” I thought, “Right now, every wealthy parent just hired tutors for their kids to sit and complete four Stanford MOOCs to put on their college application. The bar at every elite school just went up and none of the poor kids have any chance to do that…” It is true that technology is going to give people that have nothing, something. People across India now can have Wikipedia and CDs of Sal Khan videos. Technology raised the Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

floor - but it is going to blow the roof off the top. (Vuchic, formerly at William and Flora Hewlett Foundation). We concur with Vuchic that, overall, the evidence suggests that MOOCs are currently doing more to increase gaps in access to education than to diminish them.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Case 1: Virology for the Masses: Vincent Racaniello, Columbia University Virology I: How Viruses Work was among the first five MOOCs offered by Columbia University and one of the first in a series of two MOOCs offered by Professor Vincent Racaniello, based on the video recordings of his on-campus course Virology. The MOOC Virology I: How Viruses Work was offered over 11 weeks starting in August 2013, and the sequel, How Viruses Cause Diseases, was offered over 12 weeks starting in January 2014. In addition to the video recordings, the MOOCs draw on Professor Racaniello’s virology blog (virology.ws) and his weekly podcast This Week in Virology. Professor Racaniello describes his motivation for using technology to reach the masses as follows:

“My goal is to teach as many people as possible about viruses, which is unusual for a researcher. I was

teaching and I realized that I can explain complicated things quite easily. That is a gift for sure. You have to use it. Same with the knowledge. I find no reason to keep it to myself. If I can teach 180 students a year but can reach thousands going beyond the classroom, then why not?”

I have been a basic researcher my whole career here but I always liked teaching. I gave lectures all over New York: Cornell and Rockefeller and so on. A few years back I was involved in writing a textbook on Virology. Having written that, I got a very broad view of virology. That is the background. Then blogging came to be available for the masses. In 2004, I started blogging - might as well share some knowledge of virology with the world, I thought. I started writing just in a general way that goes to inform the public about viruses. I developed a following on that blog over the years. Today, I get two or three thousand hits a day. Not bad for a highly technical subject. Then I started to podcast. I thought that it was another medium I could use so I got all this audio equipment. I do three different podcasts every few weeks. My colleagues and I have conversations about topics in virology, parasitology, or microbiology. They are called "This Week in Virology/Parasitism/Microbiology.” It is a totally different audience than blogs. Some people love to read, other people like to listen. The virology podcasts, for example, get 80,000 - 100,000 downloads a month so we have a big audience out there. Then four years ago I decided...Columbia does not have a Virology course. It is one of the best institutions in the world and does not have a virology course. I submitted a proposal to do one and it was accepted. The first year I had 30 students. The following three years I had 60, 90, and 180 students. From the beginning I decided to record the lectures. Just for the students so that they are able to listen to it. It is a complicated subject and I don't expect them to get it all in the lecture. So I thought of recording them and posting them. I started posting them on iTunes U. A lot of people who have seen the iTunes U lectures said, “You should do a Coursera course now and reach more people.” So I viewed it as an expansion of iTunes in a way. Although, in two years I got 100,000 enrollments on iTunes and Coursera has 26,000 students enrolled, so it is less than iTunes. But in iTunes it is hard to know how people are using it. You don't get feedback. With Coursera, you can see how many people are taking the quizzes. There is a discussion board so you can sense who is engaged. Part of the reason why people are engaged is because they rarely had access to a professor outside the college. They [professors] are too busy to bother with e-mails and engaging with the public. I am really happy to do that. I think it is important. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

I don't do Coursera for Columbia students. However, my ability to do Coursera is based on the Columbia course. I develop the course and use the lectures from it. I stay in front of my class and lecture and I am inspired by seeing them [the students]. It makes me a better lecturer. Many people record their lectures for Coursera with nice lighting and camera work. I think that it is flat. The best part is that the participants can see that I am in front of the class; they can hear students asking questions. For me it gets me excited. I think complicated subjects like science are best taught by people who are doing the science. Certainly, other people can teach virology, but they don't bring the same depth of experience and passion. I think that is what the students feel and the public feels: the passion. You cannot have it if you just learned from a textbook. When possible, scientists doing the research should be doing the teaching. That is my mantra: to bring what I do in the lab to the public. They like the insight from the lab. They pay for the research so they need to know what is going on. They can't read it in the journals unfortunately as it is behind a pay-wall. So that is what we do. We discuss the literature so that people have access to it. My goal is to be the earth’s virology professor. One of my Spanish colleagues said “You can't be the earth’s virology professor because you just talk in English.” So I said, “You can translate it if you like.” So she got her students to translate it into Spanish. It would be great if it could be translated into Russian, Chinese, and Portuguese also. Italian I should be able to do myself. My goal is to take this course I develop here and give it to the world. I think everyone has a right to knowledge. The more people in the world we make smart, the better workforce we have. It is very clear that knowledgeable people will contribute to the better health of the world. Even if they can't afford it [education], they should have access to it. Giving it to them is just going to enrich the world. In my case, in the subject of virology, it is very relevant as everyday people in the world are faced with issues of virology. For example, Polio virus has been isolated in Israel in the last few months. I have been getting emails asking what this means. What do we have to do? That is the kind of real-life stuff I try and teach people. Every aspect of virology is important, not just how we get sick and preventing it, but also how so much of our genome is virus and what does that mean. I end up turning people on to the subject. People say I want to be a virologist because of your course. If I can inspire one person to be a virologist, that is the best thing I can do.” Source: Vincent Racaniello, Professor, Columbia University.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Case 2: Developmental Education MOOCs: Cuyahoga Community College’s Pre-Algebra CCC Cuyahoga Community College, OH, (Tri-C) received a grant from the Bill & Melinda Gates Foundation in November 2012 to create and facilitate a developmental education math MOOC, Pre-Algebra CCC. The MOOC was targeted at students performing at the lowest level of developmental education in math. Students placed at this level need three semesters of developmental education courses before they can enroll in college-level math courses. The goal of offering the MOOC was to leapfrog students out of at least one level of developmental education, reducing the delay in earning credits towards a degree. For community colleges, helping underprepared students through the hurdle of developmental education courses is an important aspect of improving access to college education and ensuring successful completion of a degree program. According to Kalbaugh, Dean, Wake Technical Community College, Raleigh, NC, if students become mired in developmental education courses, they may not reach the level of credit-bearing courses at all or may not persist through to completion because they have already expended substantial time and resources in college. Therefore, helping students advance to credit-bearing courses sooner is the key to keeping them on track for successful completion of their degree program. Belinda Miles, Provost and Executive Vice President of Access, Learning, & Success, Cuyahoga Community College, OH, points out that the number of students who must repeat a developmental course has been very high at Tri-C. As an “Achieving the Dream” college, Tri-C has been working on finding ways to improve student outcomes and shorten their time in developmental sequences. Tri-C’s Pre-Algebra CCC MOOC was hosted on CourseSites, the open platform offered by Blackboard. The four-week long MOOC was offered in March, April, May, and June 2013. The MOOC was different from most xMOOCs in two ways. Firstly, it used only pre-existing open educational resources, including videos from Khan Academy and TeacherTube, freely available exercises, and OpenStudy to facilitate collaboration among students. Secondly, the entire MOOC was structured as a game with a wraparound story. Custom graphics were donated by an artist. The game mechanics included leveling-up, that is, students could not get access a higher level until they had completed the previous one. The bar for leveling-up was set high at 80 percent. According to A. Sasha Thackaberry, District Director, eLearning Technologies, Cuyahoga Community College (Tri-C), OH, gamers in general fail 80% of the time and succeed the other 20%, and yet they develop a remarkable persistence, resilience, and sense of agency - the very skills community colleges need to foster in their students. Miles sees gamified learning experiences as a "safe-failure environment." Students might not feel comfortable failing 80% of the time in front of their peers and teachers, but when they are working independently and trying to beat a machine or game, they are motivated. Typically, students who are working on this type of college-readiness course experience a great deal of math anxiety. The Tri-C instructional designers believed that the game environment would be a good choice for this population. After the first four facilitated iterations of Pre-Algebra CCC, the course has been kept open on CourseSites (https://tric.coursesites.com/), but without facilitation from Tri-C faculty. Tri-C is collecting data to ascertain student success in the unfacilitated version. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

A total of 1,372 students signed up for Pre-Algebra CCC during the four facilitated iterations, with enrollment rising from 138 in the March offering to 537 in the June offering. Students were asked to complete a survey before being given access to the course materials. The survey queried the students’ demographics, background experiences, and reasons for taking the course. Eight hundred and nine of the enrolled students completed the survey and were able to participate in the course. Eleven percent of the initial enrollees completed the course and among those who actually participated, 18% completed with a score of 80% or above. Tri-C set the passing bar higher than the typical 70% for passing math courses because of historical Tri-C data indicating that students who earn only a “C” grade in the first-level developmental education course are far less likely to be successful at the next level. TriC wanted to ensure that students completing Pre-Algebra CCC would also be successful in future developmental courses. Among the 147 Pre-Algebra CCC completers, 13 took a math placement test before the MOOC as well as after. Of these, seven performed well enough on the second placement test to skip one developmental education math course and three were able to skip all three required developmental education courses and move directly to credit-bearing courses. While the numbers are small, they indicate that the goal of reducing the number of full-length, developmental education courses students need to complete was certainly feasible, saving students both time and money. Miles points out that the participants in developmental education MOOCs represent a totally different population from those participating in MOOCs created by universities. The latter, often stronglybranded institutions, attract many college graduates who, according to Thackaberry, “live in a highly networked environment themselves” and “find out about MOOCs with relative ease.” In contrast, 70% of those Pre-Algebra CCC participants who recorded their highest level of education in the pre-survey had not yet earned any degree and, most likely, have relatively few sources of information regarding educational opportunities. Tri-C is determined to provide access to “self-remediation opportunities” for those who are not already connected to “informal academic learning environments.”

Sources: Belinda Miles, Provost and Executive Vice President of Access, Learning, & Success, Cuyahoga Community College, OH; A. Sasha Thackaberry, District Director, eLearning Technologies, Cuyahoga Community College, OH; Laura Kalbaugh, Dean, Wake Technical Community College, Raleigh, NC.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Goal 2: Building and Maintaining Brand For institutions of higher education, building and protecting brand serves to attract and retain students, faculty members, and partnership opportunities with other institutions, funders, and alumni networks. Interviewees from 41% of the institutions in our study sample that were offering or using MOOCs stated that branding, positioning, or attracting students was a strategic goal for the initiative. These interviewees represented seven public universities, three private universities, and a museum. None of the community colleges mentioned branding as a goal for engaging in MOOCs. Branding implies different strategies to different types of institutions, depending on their current positioning. For flagship state universities in the U.S. that have already established local appeal, MOOCs allow for more national and international recognition. For elite institutions, being at the forefront of a highly publicized innovation serves as a signaling mechanism to protect their global ranking among the top universities. Additionally, the act of offering free education to the world helps to soften their exclusivity and wealth, and justify often disproportionate research funding received from public entities. Community colleges are less concerned about branding than improving access to education for an underprepared student population, but Judy Baker at Foothill College notes that offering MOOCs could help “dispel myths about community colleges and help people understand what community college means.” For European universities which must generally educate domestic students for free or at very low tuition rates, MOOCs may help attract international students who are an important source of income. Institutions that were employing MOOCs as a vehicle to expand their brand made strategic decisions to showcase programs, specialties, and research capabilities in which they believed they were leaders or wanted to be among the first to stake a claim to world-class expertise. For example, Ragan at Penn State World Campus suggested that offering the MOOC Epidemics - the Dynamics of Infectious Diseases was “a way to carve out that space” and make “a claim to a domain.” Administrators often call on “star” faculty members to offer the institution’s first MOOCs. Martin at West Virginia University described a cMOOC on the topic of social media that was hosted by multiple faculty members in the Department of Communication Studies as a means to promote the department and improve awareness of what it does to the general public. Deborah Keyek-Franssen, Associate Vice President for Digital Education and Engagement, University of Colorado System, used the term “evangelist courses” to describe the MOOCs being offered to promote the “awesome things we are doing and want everyone to know about.” Lester at FutureLearn similarly believes that MOOCs provide a useful vehicle for “broadcasting” cutting edge research to experts around the world, both enhancing the originator’s reputation and serving to attract collaborators to help extend the work. According to Nick Hathaway, Vice President for Administration and Finance at the University of Oklahoma, if an institution claims to offer a premier program, failure to offer an online version in the near future will incur reputational risk, especially if another university offers an online course in that domain and “champions themselves as the center of the universe for that subject.” Sir John Daniel, and Lester at FutureLearn made related observations regarding the need to protect brand by staying at the forefront of new trends. The Open University is doing this because it's kind of like MOOCs are the territory of what the OU is all about…It would be a lost moment if the OU didn't participate in MOOCs. It's kind of what it does. (Lester, FutureLearn). Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Reputations are at stake, especially for the lesser institutions. The situation at the University of Virginia is a case in point where the President, Teresa Sullivan, was fired and then quickly rehired, with concerns cited by the Board that included not moving quickly enough into online education. (Daniel). The one museum we invited to participate in this study, the American Museum of Natural History (AMNH), NY, emphasized branding as its primary goal for offering MOOCs, with the specific aim of highlighting its capacity to offer high quality, structured, educational experiences led by instructors who are experts in their field (see Case 3). ...obviously we care what people learn but we also deeply care about their perception of the museum as a valuable entity in their lives…How is this helping the museum count more in their assessment of their learning continuum?…Most people in our course didn’t know that we existed before, or that our resources existed….That’s the main benefit for the museum. (Kinzler, AMNH; Steiner, AMNH). Some interviewees offered less flattering perspectives on the branding potential of MOOCs, with a few suggesting that most institutions pursuing them were “being sheep” and others accusing some MOOC instructors of being primarily interested in being “recognized in the parking lot,” “defending their textbooks against interlopers,” or expanding the market for their textbooks. Bates suggests that "Ivy League universities are trying to use MOOCs in such a way that they now ‘own’ online learning in ways that enable them to leverage their brand.” At worst, MOOC initiatives were viewed as “a demonstration of offensive, neo-imperialist hubris.”(Daniel). MOOCs as tools of recruitment Building brand is a direct pathway to increasing recruitment and enrollment of students in tuitionearning courses and programs. MOOCs offer participants a “frictionless,” that is, low-risk, no-cost introduction to a program area, and the offering institution can simultaneously identify candidates who might perform well as degree-earning students. For example, at the University of Colorado Boulder, College of Engineering professors who offered engineering courses as MOOCs were planning targeted recruitment efforts that would involve sending information packets about the program to top performers in the MOOC. Similarly, one of the goals for the Geospatial Analysis and Mapping MOOC offered by Penn State World Campus was to increase awareness and interest in the area to lead people to the Master’s Degree in Geoscience Studies. Ragan at Penn State World Campus mentioned that, after the MOOC ran in July 2013, there was a substantial increase in inquiries and interest in this program, with traffic on the program website tripling. An evaluator at a large Midwestern university is analyzing participation in the university’s MOOCs and gathering survey feedback from MOOC learners regarding their potential interest in other courses and programs, with the aim of creating attractive offerings that will lead to increased enrollments at the university from outside the state. “An extensive end-of-course survey posed many psychographic questions, many of them open-ended. We got responses indicating that participants wanted more sequenced courses.” Similarly, Ragan at Penn State World Campus noted that course surveys are being used to elicit ideas for future offerings that would be appealing to MOOC participants. Academic Partnerships, a for-profit, online service provider and technology “enabler,” has formalized the route from MOOC to degree with its MOOC2Degree initiative which offers an initial course from a Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

partner university’s online program free to non-enrolled participants (see Case 4). If successful in the course, these participants can earn credit for the MOOC by enrolling in the full program at the offering university. Currently, two state universities are offering MOOCs through this initiative. The College of Nursing at the University of Texas at Arlington has piloted an offering of its course Enhancing Patient Safety through Interprofessional Collaborative Practice through MOOC2Degree. Successful completers may apply to the college’s Registered Nurse to Bachelor of Science in Nursing (RN to BSN) online program. The University of Cincinnati is offering Innovation & Design Thinking which can lead to entrance into the Master in Business Administration or Master of Engineering online degrees. Hersh, formerly at Santa Barbara City College, views MOOCs as “virtual land grabs,” that is, “opportunities for universities to brand themselves to the rest of the world to increase their student base and to develop additive revenue.” He notes that, prior to the advent of distance education, many universities attempted physical expansions abroad, but that with “distributed management” and foreign cultures, these satellites were often unsuccessful. Hersh suggests that MOOCs may provide a more manageable alternative. In addition to recruiting students, some institutions emphasized the need to showcase themselves as the leader in academic content areas and in pedagogical innovation so as to attract and retain faculty members who expect “to play in the same league as their competitors.” (Mintz, University of Texas System). MOOCs also offer an opportunity to maintain connections to alumni, who are an important source of funding. Several interviewees commented that MOOCs provide opportunities to forge mutually beneficial partnerships with other educational institutions - what Lester at FutureLearn described as “coopetition.” They also facilitate relationships with corporate partners and funding agencies, potentially leading to increased grant revenues. For example, the business school at the University of Colorado Denver, offered the Global Energy Management (GEM) MOOC in order to create brand recognition for the School of Business in the energy field and to reach out to energy professionals in the region who could offer potential for collaboration or funding. Mintz at the University of Texas System believes that the university’s investment in edX has been worthwhile in establishing it as a serious player in the realm of educational technology and innovation: It has made us competitive for national grants that we would not have been competitive for otherwise. It put us in conversations that we wouldn't have been in. We are working closely with MIT, Berkeley, Harvard, Georgetown, Rice, and others in ways we would not have been working. (Mintz, University of Texas System). Some institutions, recognizing that existing brand is a key factor in the success of MOOCs, are saving their dollars for other initiatives where they have more competitive edge: I am a realist about the branding issue… it would be very difficult for a MOOC from Georgia State to become among the definitive MOOCs in some major field like introduction to psychology when there is Duke and Harvard and Stanford and Michigan and others who are producing their own versions of this course. This is not to denigrate Georgia State, I am very proud of the institution and the commitment we make to our students, and the success they’ve had. But it’s being realistic and if we have $2mm, we can put it in programs that will change the lives of hundreds or thousands of students, or we can develop one or two MOOCs that likely will soon be Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

overshadowed in the marketplace by the production of better branded institutions. (Renick, Georgia State University). Isolating and measuring impact of a new initiative on brand can be notoriously difficult. An evaluator from a large Midwestern university suggested the possibility of capturing the effect of offering MOOCs on brand and reputation by comparing historical data on applications and admissions: “We could look at how many minorities get involved, how many people from other countries, from our state and other states. We can look at whether applications and admissions go up or down overall and for certain groups.” While we heard substantial anecdotal evidence and many aspirations with regard to the positive impact of MOOCs on brand most institutions are only just beginning to think about how to capture and quantify branding-related benefits to determine whether they outweigh the costs of investing in MOOCs. As institutions think about extending their brand through MOOCs, one complication that arises is the question of where the brand actually lies. Do participants opt for a course because it is on a particular platform or because it is offered by a particular university? As MOOC platform providers such as edX and Coursera open their platforms to a wider range of institutions, some of the initial cachet of belonging to these consortia is being lost. A senior administrator at a leading university in the United Kingdom expressed strong reservations about partnering with one of the MOOC providers for fear that it might dilute the institution's own very strong brand. We would be hesitant about getting involved with any of the current big providers of MOOCs because we think the governance challenges involved in associations with edX or FutureLearn in the UK are considerable for a University like ours which has a very strong brand identity. We don’t dilute or give out that brand identity lightly so there would be a lot of bridges for us to cross to be associated with the current providers. We are not going to be railroaded politically into getting into MOOCs. (Senior Administrator, Leading UK University). A final challenge in using MOOCs to build and maintain brand is the clear tension that Chris Dede, Wirth Professor in Learning Technology at Harvard University, observes between building brand and increasing access: “The problem the MOOCs are underscoring is that Harvard has contradictory goals of increasing its footprint and protecting its brand. Brand has value by being limited. The two goals paralyze each other.”

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Case 3: MOOCs to Increase Visibility: American Museum of Natural History MOOC Initiative In early 2013, Coursera invited the American Museum of Natural History (AMNH) in New York City to create three science-oriented MOOCs to be delivered in the Fall of 2013 through Coursera’s Teacher Professional Development program. AMNH successfully launched three 4-week long MOOCs targeted at science educators: Genetics and Society: A Course for Educators, The Dynamic Earth: A Course for Educators, and Evolution: A Course for Educators. While the museum had several goals for this initiative including improving teachers’ science content knowledge and use of AMNH materials in their classrooms, their primary goal was to increase the museum’s visibility, especially with science educators and international audiences. Some indicators of success in achieving this goal include the following: •

At time of offering the three MOOCs, AMNH was one of only three museums worldwide represented in the Coursera Teacher Professional Development program, accounting for three of the 44 MOOCs available.

•

A total of 39,685 people registered for the three courses.

•

Around two-thirds of these registrants were international, representing 110 countries. This contrasts significantly with the almost purely domestic population of participants in the museum’s regular online courses.

•

Sixty-two percent of the survey respondents had no prior experience with the museum or its content and were therefore “new customers.”

•

Forty-two percent of the MOOC participants who responded to surveys at the start of each course were educators, the target population for the AMNH MOOCs.

•

While the vast majority of survey respondents learned about the AMNH MOOCs from the Coursera website, some heard about them from a friend or colleague, an online social network, a search engine, blog or press article, AMNH’s own website or from their school or school district.

•

The promotional videos for each of the three MOOCs were discoverable on YouTube outside of the Coursera platform and were viewed between 450 and 864 times.

Sources: Dr. Ro Kinzler, Senior Director, Science Education, American Museum of Natural History, NY; Dr. Robert Steiner, Director, Online Teacher Education Programs, American Museum of Natural History, NY; Maria Janelli, Senior Manager, Online Teacher Education Programs, American Museum of Natural History, NY.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Case 4: MOOCs for Recruitment: Academic Partnerships MOOC2Degree Initiative One of the most strategic approaches to recruiting new students into university programs through MOOCs is being led by Academic Partnerships (AP), a for-profit entity that works primarily with public universities in the U.S. and universities abroad to: • • • •

Assist faculty members with conversion of their traditional degree programs and courses into an online format. Recruit qualified students into these programs. Help with retention of students through to graduation. Expand the reach of partner universities globally through an established worldwide university network.

AP's mission is to utilize technology to expand access to high quality, post-secondary education at scale. AP’s Executive Vice President, Dr. Justyna Dymerska, believes that MOOCs provide the capability to disseminate knowledge at a very low cost. However, she cautions that the current model of providing MOOCs for free is not sustainable for public universities that are already burdened with increasing costs and shrinking funding. To “capitalize on the potential of MOOCs while addressing the bottom line,” AP launched the concept of MOOC2Degree in January 2013. Under this initiative, the initial core course in an online degree program that AP represents is converted into a MOOC. These MOOCs are free, open, and “full-calorie courses,” that is, they have the same academic content and instructor as the closed, credit-bearing, online course. Students who successfully complete a MOOC2Degree course can apply to the full degree program to which the course belongs, and, if admitted, receive credit for the MOOC based upon criteria established by the partner university. According to Dymerska: A MOOC is an example of an innovative disruption within the higher education model. It increases the branding and makes our partner institutions more competitive in a very dense, competitive higher education market. The partner universities see it as a practical approach, not only for increasing local market share, but for global growth as well. It fits the mission of expanding access....From a more operational perspective, they get a free look at a student and a student's performance in a course....Essentially it's a filter for the university. Students who carry on into a degree program are high performance students, they're mature and they want to complete a degree. From AP's perspective, "MOOC2degree is a great recruiting tool for students, allowing universities to cast a broad net. We let people take the online version of a course and those who wish to continue do so, and for those who don't, they've had a great learning experience." (Dymerska, Academic Partnerships).

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

To date, the following institutions have launched or planned a MOOC under the MOOC2Degree program: • University of Texas at Arlington College of Nursing created the MOOC Enhancing Patient Safety through Interprofessional Collaborative Practice leading to the RN to BSN (Registered Nurse to Bachelor of Science in Nursing) online program. • University of Cincinnati created the MOOC Innovation & Design Thinking leading to the Master in Business Administration or Master of Engineering online degrees. • Lamar University (Proposed). • University of West Florida (Proposed). The College of Nursing at University of Texas at Arlington (UTA) initially teamed with AP to convert their traditional RN to BSN program into "an innovative, highly-accessible, affordable, and scalable [online] program” (Poster, Mancini, & Ganji, 2013). The RN to BSN online program enabled flexible scheduling and competitive tuition, addressing two barriers identified for working nurses with diplomas or associate degrees who wish to pursue advanced degrees. MOOC2Degree was proposed by AP as one of the ways to recruit students to this program and the course Enhancing Patient Safety through Interprofessional Collaborative Practice was selected to be offered as a MOOC. The target audience was practicing nurses, with a view to providing them an opportunity to determine whether online learning works for them, to experience the effort required, and to make an informed decision about committing to a full degree program. The MOOC2Degree option lets them try out the first course without any financial commitment to a program. To receive credit for the MOOC, students must: • Complete the MOOC2Degree course with a score of 80% or higher • Complete an online proctored exam within seven days after the course ends with a score of 70% or higher • Enroll in the RN to BSN program after fulfilling the admissions criteria. Learners who wish to receive Continuing Education Units for the course pay the continuing education provider $25 to obtain a certificate for the 45 hours of continuing education credit. In August 2013, UTA’s Enhancing Patient Safety through Interprofessional Collaborative Practice MOOC went live with a “soft launch” to pilot the course structure and processes. On the start date, approximately 300 learners were enrolled. While AP did not disclose how many of the enrollees were actually registered nurses, they reported that all of the registered nurses who responded to a participant survey within the MOOC expressed interest in applying to UTA College of Nursing’s RN to BSN program and receiving academic or continuing education unit (CEU) credit. By the time the MOOC closed, 50% of these registered nurse survey respondents were moving forward with the application process. AP will receive a percentage of the revenues from tuition paid by these students.

Sources: Dr. Justyna Dymerska, Executive Vice President, Academic Partnerships. Poster E. C., Mancini M., Ganji, D. (2013, August 5). MOOCs and more: Expanding online access for nurses. EDUCAUSE Review Online. Retrieved from: http://www.educause.edu/ero/article/moocs-and-more-expandingonline-access-nurses Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Goal 3: Improving Economics: Reducing Costs or Increasing Revenues Thirty-eight percent of the institutions in our sample that were offering or using MOOCs, and twentynine percent of our interviewees overall, claimed that a goal for their MOOC initiatives was to lower costs or increase revenues, or both. Among the 29 institutions currently offering or using MOOCs, this goal was mentioned by representatives of eight of the 15 public, four-year universities, two of the ten private, four-year universities, and one of the three community colleges, suggesting that it was more important to public universities than private ones. There was widespread acknowledgement that the current expenditures on MOOC development could not continue indefinitely without financial justification. The States used to cover about 75% of students’ tuition but now it’s more like 25% in most states; at least that’s true in Michigan where I was before. As that is happening, we [the San José State University administration] feel it is incumbent upon us to research and to figure out how we are going to survive as an institution. (Cheal, San José State University). We’d be crazy not to be thinking about cost-effectiveness given that state funding has dropped and will probably not rebound to earlier levels. (White, Georgia Tech). Our primary goal is to find cost-effective, scalable ways of providing professional development. (Kleiman, North Carolina State University). Online tools will fundamentally increase the value students get from residential education. I feel this will address many of the vexing questions about the costs of higher education. (Sarma, MIT). Some interviewees suggested that the notion of cost savings through MOOCs arose primarily from policymakers looking for a silver bullet to reduce state education outlays: …given the severe budget crunch that California is under and the severe crunch in higher education, [the governor is] convinced that online learning is part of the answer and that it's needed. He has been heavily promoting it. (Means, SRI). A lot of politicians and business leaders really thought that you can essentially just do this and solve universities' cost disease: do to university what has been done to lots of other industries, with digital technology. Getting people out of it. Getting excess labor costs out of it and get a better quality product. That was the claim: better education for less money. The promise was solving the quality-cost tradeoff. (Newfield, University of California, Santa Barbara). For high-level policy makers, the possibility of bending the cost curve does seem to be the dominant reason why they are interested. (Professor, East Coast University). Many interviewees expressed concern at the lack of sustainability for MOOC initiatives given the heavy burden on faculty time and other institutional resources. If MOOCs are going to demonstrate a genuine return on investment, they must produce educational resources that can make a genuine contribution to learning across the campuses, for example, by developing interactives, simulations, and other educational materials, and develop a next-generation platform that can support state-of-the-art teaching innovations. (Mintz, University of Texas System). Here is the conflict that we have today in higher education: cost is going up. I believe it is up either four or five times since the 80s. It went up by 8.3% in 2012. Government subsidies are Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

down. Just last year in 2012, they went down by about 9%. The middle class is tapped out with one trillion dollars in student loan debt, more than all consumer debt in the United States. This unfavorable landscape is accompanied by declining enrollment, which I believe last year went down by 2.3%. When we assess the MOOC concept and look at it from this perspective, if a university provides credit for a MOOC, then the model becomes problematic because it takes a significant bite from the operating revenue of a university. Giving credit for the MOOC is actually increasing the cost of delivery. Basic economics here, when you think about MOOCs, is that if you reduce the price of your product in any sector, not only in higher ed., but any industry, you have to lower the cost of production for it to be sustainable. In our view, universities have been defying gravity here by marking courses, their core product, to zero. This is unsustainable. We see the MOOC in its current form as a catalyst that shortens the time of sustainability of today's already broken university business model. Increasing access cannot be sustained by widening their deficit. We see MOOCs as potentially threatening to the university’s mission unless they are tied to degree programs and are used in the context of increasing paid enrollments. (Dymerska, Academic Partnerships). Burck Smith at Straighterline categorizes MOOCs, along with other online education interventions, as “marginal cost-pricing models,” and similarly points out a conflict between this model and the way universities and colleges currently manage their finances. What you're seeing is these entities are pricing individual courses much closer to their true cost of delivery. It really doesn't cost much to deliver that on a marginal-cost basis. There are some fixed costs for sure. But one of the central tenets of econ 101 and microeconomics, is that the price of a good ultimately equals its marginal cost of delivery. As a categorical definition, all of these are marginal cost-pricing models. That does not fit at all with how higher education is constructed. If you look at colleges who are offering online courses, they are pricing them oftentimes ten to thirty times higher than their true cost of delivery, making big profits. That subsidizes the face-to-face environment, or subsidizes other parts. What the “unaccreditable sector” is doing, is simply pricing the individual units very close to the cost of delivery. (Smith, Straighterline).

Potential Cost Savings from MOOCs In spite of recognition that MOOCs have, so far, proved to be a significant drain on time and money for institutions, interviewees offered several possibilities for eventual cost savings to institutions: • • • • • • • • •

Re-using MOOC materials multiple times Sharing MOOC materials across instructors and campuses Developing common courses to offer across institutions Replacing on-campus courses with MOOCs Faculty time savings Reducing the need for facilities Recruitment efficiencies Less costly student support services provided by non-faculty members Increasing student throughput Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Each of these ideas is described in greater detail below. Not surprisingly, given the infancy of MOOC initiatives, we were hard pressed to identify examples of actual cost savings realized to date. On the contrary, we were able to gather significant evidence of the high costs of development and delivery of MOOCs which we present in a later section: Resource Requirements and Costs of MOOCs. Re-using MOOC materials multiple times Several interviewees argued that, after the initial investment in creating online materials, re-using them with limited revisions would improve cost-efficiency: … but over time, certainly, once the higher costs of development are over, it would certainly pay off if the materials don’t have to be changed very often. (Cheal, San José State University). How you repurpose that MOOC afterwards has to be part of the cost-effectiveness equation. (Fisher, Vanderbilt University). James, of Mt. San Jacinto Community College, contrasts the costs of offering a developmental writing course on-campus with the costs of re-running the MOOC, Crafting an Effective Writer, recently developed by the college: If this MOOC helps our students, it is going to save us money. We wouldn't have to have these courses on-campus. It costs us about $6,000 every time we offer it on-campus to pay instructors and tutors. There is a cost to re-offering the MOOC, but not a huge one. We already created it. There is not going to be huge change. We paid our tech 20 hours at $45/hour to renovate the course for the second offering, to make revisions and keep track of things. The cost of running it is not that great. The cost savings are greater. (James, Mt. San Jacinto Community College). Ragan at Penn State World Campus, based on his experience with regular online courses, is less certain about being able to re-use MOOC materials without significant adjustments each time. In particular, he notes that designing formative and summative assessments and strategies to engage students in interactions and dialogues around the content are “rarely correct the first time out.” In the best of all worlds, you design it once, develop it once, and deliver it four times or five times before you pull it back up under the rack and rework it. The truth of the experience has been for us always in online courses is, the first three or four times you teach the course, you deliver the course, there are significant modifications made on the revision. You run it one time, you figure out what didn't work. You come back, you retool it, and so forth. You're automatically into a revision cycle that can be as much as a half of the initial investment. It's not until about the third or fourth time until it begins to stabilize some. By then, your content's old. (Ragan, Penn State World Campus). Sharing MOOC materials across instructors and campuses Several interviewees suggested that MOOCs could be used across multiple instructors and multiple campuses. In some instances, the MOOC materials developed by one instructor could, with appropriate licensing, be embedded in a course delivered by another instructor. In other cases, one single course could be offered online across multiple campuses with no local instructor. MJ Bishop, Director, Center for Innovation and Excellence in Learning and Teaching at the University System of Maryland, observed that in these latter situations it would be necessary to resolve logistical issues such as how tuition and credits are transferred. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Rebecca Griffiths, Program Director for Online Learning at the research organization Ithaka S+R, which has conducted several notable studies of online learning, envisions cost-efficiencies in repurposing MOOC materials for multiple contexts. Amy Collier, Director of Digital Learning Initiatives, Stanford University, concurs and Guthrie, President of ITHAKA, provides a specific example of how this might happen. We think it does not make sense for every instructor to create their own set of MOOCs or regular online materials to flip their courses or to try new formats. There should be some efficiency in repurposing materials created by one set of instructors and using them in a lot of courses - the marginal costs of using them are close to zero. Once you have made the investment in what should be very high quality materials, there is an argument for using them in as many contexts as possible. (Griffiths, Ithaka S+R). A lot of MOOC materials can be re-purposed, re-sequenced, reconfigured for on-campus students, for public audiences, for professional audiences, without too much of a cost impact. (Collier, Stanford University). Let's say that I'm a computer science professor. I already know how to teach programming, but I don't teach Python. I can have my students take the Python MOOC from Rice University. Then I can add something to that MOOC, maybe create some other assignments. That provides me with an opportunity potentially to offer a better learning experience at significantly less cost. (Guthrie, ITHAKA). Interviewees from some university systems see the potential for cost savings if educational materials can be shared across the system campuses or re-used many times over without significant changes. I think we're more interested in creating repositories of open educational resources and finding places where we can enhance our efficiency by sharing resources across the system. (Bishop, University System of Maryland). However, as the University System of Maryland is discovering in its current experiment to integrate MOOC materials into their courses (see Ithaka S+R, 2013), it is not so easy to take apart the components of a MOOC for selective assimilation into an on-campus course. Bishop described the significant efforts required to “unbundle” MOOCs so that faculty could, for example, embed the videos in their regular learning management system. Developing common courses to offer across institutions Lawrence S. Bacow, President Emeritus, Tufts University, MA, is concerned that MOOCs may only add to the already high costs of higher education. He believes that institutions that are genuinely interested in lowering these costs should focus on how to use MOOCs locally rather than worry as much about the “student sitting in Timbuktu.” He and others suggest a strategy for sharing resources. Another potential saving for institutions: to offer a richer mix to students without having to staff up. For example, offering different languages like Pashtun or Urdu by pooling resources of various institutions and providing a common language course online. (Bacow, President Emeritus, Tufts University). MOOCS could save individual colleges time and money by avoiding the recreation of the same courses, especially developmental courses that help students prepare themselves for creditbearing courses. (Online Learning Coordinator, Large California Community College). Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

If multiple institutions create content and make it available to others with a Creative Commons license, then there is potential for savings. But if everyone is creating the same thing we are losing the potential for economies of scale. I am not sure why 500 institutions are all creating English 101. (Royal, Inver Hills Community College). However, Vignare at University of Maryland University College cautions that college and university educators are more adept at creating new content than re-using existing content effectively. She argues that there is already an overabundance of content and that “…we have to figure out better ways to use current content and focus more on the teaching and learning…” in order to improve college attainment for the majority of students. Burck Smith at Straighterline noted that, a number of years ago, major commercial providers of educational content invested a great deal of effort in developing high quality, online courses but found very few academic institutions willing to purchase these materials instead of developing their own. Replacing on-campus courses with MOOCs Lester of FutureLearn and Dr. Paul S. Peercy, Dean Emeritus, College of Engineering at the University of Wisconsin-Madison, suggested that MOOCs could replace entry level courses, especially those that already attract several hundred students. However, Lester believes that some face-to-face supplement will still need to be offered on-campus. Bishop at the University System of Maryland expects that MOOCs could replace some aspects of existing courses. MOOCs in their current form appear to be good at knowledge-level content delivery because they’re more interactive than textbooks. So, why not go ahead and give up those parts of our large, entry-level courses, but then bring students to campus to talk about the higher level concepts? At the same time, eliminating redundancies across courses may result in some cost savings. (Bishop, University System of Maryland). Griffiths at Ithaka S+R expresses lower confidence in MOOCs as a substitute for introductory courses, but entertains the possibility that they could replace some less populated senior classes. There is also the question of what kinds of courses MOOCs are most appropriate for. Most of the work we have done has been around large introductory courses which are expensive to provide as they are often taught in several sections by tenured faculty and are often a bottleneck for students. But I am not sure this is the best place for them because they are such an important building block in a progression. On the other hand, there may be a place for them in senior-level classes that are hard to fill, especially in small departments with few full-time faculty. There is also the potential to embed modules of content in introductory level courses, more so than MOOCs as complete courses. (Griffiths, Ithaka S+R). There has been fear among many faculty members that MOOC master courses created by professors at elite universities to be used by other universities and colleges will make their jobs redundant. Acemoglu, Laibson, and List (2014) argue against the notion of MOOCs creating “winner-take-all superstar teachers” (p.1). Based on a theoretical model of human capital accumulation they predict that webbased education technologies (including MOOCs) will result in more equal distribution of educational resources. Their model is based on the assumption that scalability of MOOC resources created by skilled faculty members can be exploited only if the technology is complemented with face-to-face discussions with instructors. As the face-to-face instructor is freed from lecturing and allocates more time to other teaching activities, the quality of overall instruction will increase, thereby benefiting students. In Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

addition, if this shift helps teachers become more effective and productive, it may have a positive impact on the wages of less skilled lecturers. Faculty time savings Over a period of several years, Peercy at the University of Wisconsin-Madison oversaw a shift to more online content in a flipped classroom model for the University’s engineering department. He has no doubts that there have been time-savings involved for course instructors which could be realized using MOOC materials for the same purpose. Let us take a three-credit course taught by a highly paid professor…typical numbers are: two hours of preparation, Monday, Wednesday, Friday; one hour of lecture, Monday, Wednesday, Friday; two hours of office hours, Monday, Wednesday, Friday; two hours of discussion time, Tuesday, Thursday. Add those hours up: 17 hours. Now put that course online. No lecture preparations: that saves you six hours. No lectures: that saves you another three hours. No office hours: that saves you another six hours. The debate students have is whether they need two face-to-face sessions with you or three…we did three. You have taken the faculty content workload from 17 hours a week down to three. Now what does the faculty do with those other 14 hours? Some volunteer to develop and teach another course…they can spend the time in consulting, writing proposals, doing research. (Peercy, University of Wisconsin-Madison). Bacow, President Emeritus of Tufts University, offers a different perspective on flipped classroom models, suggesting that while they may improve learning outcomes, they do not lower costs. If you think about the educational production function, the most efficient part of the production function is putting one faculty member in front of 300 students. What is hideously expensive is all of the sections to ensure that students are learning something. If you just eliminate the lecture by flipping the classroom but create more sections, now students are watching the lecture online but you don't reduce the costs. One of the hypotheses in the flipped classroom model is that if you take the professor out of the lecture hall the professor has more time to engage students. But you cannot have meaningful conversation with 300 students. This necessitates more sections. Sections are really expensive. Firstly, they require lots of space; secondly, lots of people to teach the; thirdly, the need to staff large number of sections of introductory courses causes many institutions to over-scale the size of their graduate programs in that field. TA-ships subsidize graduate education. If we count the full costs of graduate education, sections become exceedingly expensive. The second order consequence of this is that if we subsidize the production of a good, we end up over-producing it. We will end up with more doctoral students in a given field and wind up with disequilibrium in the marketplace. The big cost savings would come when we can use technology to reduce the number of sections. That is only going to happen when we develop interactive education programs which do a good job of substituting what goes on in a section. (Bacow, President Emeritus, Tufts University). Reducing the need for facilities Bacow suggests that MOOCs and other online offerings can lead to savings in facilities expenditures, in particular by incorporating virtual laboratories. Armando Fox, Professor, Electrical Engineering and Computer Science Department and Faculty Advisor to the MOOCLab, University of California, Berkeley, provides a specific example of how this may play out in practice. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Anant Agarwal has done some interesting things with virtual laboratories in the circuit and design field. In addition to sections, labs in scientific fields are very difficult to maintain and staff. Reducing labs is a way to reduce costs. (Bacow, President Emeritus, Tufts University). We have 240 students this semester. We're doing an experiment where the lecture hall for the class only holds about 100, because we know that, after the third or fourth week, people stop coming to lecture because they know the lectures are going to be posted on the Internet. We just took the stigma away; we said, "Look, if you don't want to come to lecture, don't come. You won't all fit anyway." There are people who like coming to lecture. The lecture hall is still full: 100 people, pretty much, are showing up, out of the 240 who are registered in the course. The rest of them are, I assume, watching at home, or maybe they're not watching at all. (Fox, University of California, Berkeley). Recruitment efficiencies Lester of FutureLearn believes that MOOCs can act as a cost-effective alternative to recruitment agents, especially in attracting international students. What's key is there's cost-effectiveness from a recruitment point of view and from a learning productivity point of view. The one for recruitment is straightforward. If you can build the systems right, and you can track people from one to the other, then you can see whether there's a return on investment, whether itâ&#x20AC;&#x2122;s effective to use this as a recruitment device. If you want to attract international students right now you have to use recruitment agents. (Lester, FutureLearn). While the MOOC2Degree case (see Case 4) illustrated earlier offers a feasible mechanism for recruitment via MOOCs, it has not been determined whether this is cost-effective. Less costly student support services provided by non-faculty members Daniel suggests that student support functions for online courses, including MOOCs, can be provided by personnel who are less costly than faculty members. He suggests that savings could be achieved by using already existing MOOCs and outsourcing personnel. Randy Best at AP [Academic Partnerships] thinks that universities can deliver online courses at half the cost they are delivering face-to-face courses now. Online courses can help reduce costs of student support. At AP they bring in part-time folks for tutoring. For MOOCs, most of the cost is for development, according to Tony Bates. The ratio needs to change with reduced cost of development and increased cost of student support. This can be done by offering credit for already existing courses, like those offered by MITx that are designed and ready to use, in partnership with organizations like AP who already have the infrastructure needed for student recruitment, support, and evaluation. (Daniel). A partial outsourcing model has been pursued for the MOOC-based Online M.S. in Computer Science program that Georgia Tech is developing with Udacity (see Case 5). Each course is staffed by a Georgia Tech instructor and a Head Teaching Assistant (TA) serving an estimated 127 students in Year 1, rising to 456 in Year 3 (based on projections from Spring 2013, although the actual average course size for the five courses offered in Spring 2014 was 115 students). The Head TA, while selected by Georgia Tech, is employed by Udacity. In addition, Udacity-based Course Managers, also approved by Georgia Tech, help students with practical issues, each serving 390 - 395 students. Georgia Tech graduate students, serving Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

40 students each, work as TAs to help with grading and responding to academic questions. Georgia Tech and Udacity are working on ways to manage the workload, including more use of automatic grading, as enrollments increase. Courses are initially taught by a faculty member, but re-runs may be taught by a non-tenured instructor or the Head TA who is expected to be a full-time professional and expert in the relevant field. It is not clear yet whether this student support model will be sufficient. According to Dymerska, in the Academic Partnerships model for online courses, each Virtual Teaching Assistant (the equivalent of a Head TA) works under the direction of the faculty member of record and serves 60 - 75 students at most, an average of two sections, with additional “retention specialists” providing supplementary counseling and mentoring. Increasing student throughput Several interviewees suggested that if students can earn credit for completion of a MOOC, the possibility of eliminating bottlenecks in over-subscribed courses may help accelerate student “throughput.” Another big opportunity in reducing costs: we found in our “Barriers to Online Learning” paper that a big motivation for some institutions to be able to offer online learning is to enable students to take courses that they are closed out of. As a result, their time-to-degree increased. If you can use online education to improve the throughput of students, the cost of education will fall dramatically. However, it does not show up in some conventional measures of cost. If you look at tuition it would not show up but if you look at cost per degree granted it is there. Lots of students who cannot matriculate have to stay for an extra year and are held out of the job market. So there are increased opportunity costs. If you use technology to broaden access to gateway courses there is a chance to reduce costs. (Bacow, President Emeritus, Tufts University). A small number of community colleges are experimenting with MOOCs as a way to lower costs to students and improve student retention by reducing the number of developmental courses that students must complete before being eligible to enroll in full, credit-bearing courses. The goal is to allow students to bypass classes that cost money but do not give transfer credits. It also takes you a year to get to college level wasting time and financial aid money. (James, Mt. San Jacinto Community College). While these colleges may appear to be sacrificing revenues by offering free, online courses to help students place out of fee-earning developmental courses, they expect both students and institution to benefit in the long run, especially as community college funding moves towards a performance-based model rather than depending primarily on enrollment numbers. …there is some research out there that says that if we can get them through developmental or to bypass developmental, we'll recoup that in serving that student for a longer period of time in curriculum classes. (Kalbaugh, Wake Technical Community College).

MOOCs as a Source of Revenue While a small number of interviewees (less than five percent) speculated that some institutions of higher education are pursuing MOOCs as a potential source of revenue, only two representatives from state universities explicitly stated revenue generation as a goal. Most institutions, at least for now, appear to be treating MOOCs as an investment. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Overall, the expense of engaging in this activity and developing the content is something that's seen as a strategic investment for our future. (Hathaway, University of Oklahoma). We decided early on that revenue generation would not be a primary reason [to do MOOCs]. We believe that, long-term, this has to find a way to be sustainable as a technology investment. It would be great if it turns out to generate some revenue, but nobody at the campus leadership level believes that this is going to be a major revenue generator, or that this is the golden unicorn that's going to swoop in and solve the university's financial pressures. If anything, for the first two to four years, we assumed that this is an investment: that this is going to be a revenuenegative activity because we have to train faculty how to do it. (Fox, University of California, Berkeley). Potential current and future sources of revenue mentioned by interviewees included: • • • • • • • •

Offering credit and charging tuition Creating new courses and programs Drawing MOOC participants into full-tuition degree programs Increasing class sizes Licensing fees for use of MOOC materials by other institutions Fees for additional services Grant revenues Matchmaking for employers

These ideas are each described in greater detail below with examples provided of instances in which revenues are already being generated or are expected. Offering credit and charging tuition There are a few instances in which MOOCs or MOOC-like courses have been offered for credit and tuition is charged. For example, the SJSU+ Augmented Online Learning Environment pilot courses at San José State University were offered as free, non-credit-bearing courses to the public, and to a smaller number of formally enrolled students who earned credit and paid $150 per three- to five-credit course. Georgia Tech’s Online Master of Science in Computer Science (OMSCS) program, launched in Spring 2014 was due to charge $134 per credit. In each instance where the courses are offered for credit, the offering institution has provided some degree of supplementary face-to-face instruction or online support for course participants. However, based on our review of the financial and enrollment projections for Georgia Tech’s OMSCS program (see Case 5), we do not expect that that the revenues will exceed the high fixed costs of MOOC development over the initial three year period. We did not have access to financial projections for San José State University’s MOOC initiative that would allow a similar review. At the University of Oklahoma, 20 credit-bearing, online courses have been developed with on-campus students as the primary audience in a flipped classroom model. These courses carry the same credits as the traditional courses and charge the same tuition. The courses are also open as MOOCs to any member of the public who pays no tuition, receives no credit, and does not attend on-campus sessions. According to Mintz at the University of Texas System, the Austin campus will experiment with a similar strategy later this year: Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Among the strategies that we will explore is a value-added model that will provide enrollees with more support from faculty and teaching assistants. (Mintz, University of Texas System). Straighterline offers a useful working model for online course revenue generation whereby students pay a $99 monthly fee for access to any course. All courses carry ACE credit recommendations so that students can apply for transfer credits at degree-granting institutions and, in some cases, articulation agreements guarantee acceptance of the courses for credit. In addition, qualified professors can charge $50-$100 to provide supplemental materials and student support for a Straighterline course. Jake Reynolds, General Partner at Technology Crossover Ventures, points to a similar strategy employed by lynda.com which charges $25 - $37.50 per month for task-oriented tutorials on topics such as software training. Emily Schneider, Student Director at Stanford University’s Lytics Lab, suggests another possible model of income generation for MOOCs that is used by Spotify, Netflix, and other entertainment services where consumers are willing to pay small amounts of money over time to have access to a wide array of materials. Schneider cautions that it will take a concerted effort among universities and platform providers to wean the public away from the current expectation that information on the web should be free. If there is a common desire to keep MOOCs free for those participants who cannot afford any fee, Schneider suggests that public or foundation funds will be necessary to support the endeavor, possibly supplemented with “some sort of tip-based, Kickstarter-style model , or pay-what-you-will at or above a recommended price.” One senior administrator at a large state university, frustrated that a major MOOC platform provider will not permit partner institutions to charge for their MOOCs, suggested that his/her institution might offer MOOCs using the Canvas Network platform and charge $5 per participant. Creating new courses and programs Several interviewees suggested that the advent of MOOCs and their capacity to reach different audiences has created the impetus to develop new, fee-based online programs and courses. The best known examples of new revenue-generating courses and programs developed around the MOOC concept are the above-mentioned courses developed for San José State University’s SJSU+ Augmented Online Learning Environment Pilot Project with Udacity (see Box 6 and Firmin et al., 2013), and Georgia Tech’s Online M.S. in Computer Science program also being developed in partnership with Udacity (see Case 5). At least two institutions were actively collecting ideas from MOOC participants about courses and programs they might be interested in pursuing. A few interviewees raised the possibility of developing MOOCs for the employees of large companies, similar to the “white label” courses that edX is already offering. Drawing MOOC participants into full-tuition degree programs We heard several anecdotal reports about increased interest in an institution’s regular courses subsequent to a MOOC offering, but only found one documented instance in which MOOC participants were being drawn into an existing, full-tuition program. University of Texas at Arlington’s school of nursing offered a MOOC through MOOC2Degree (see Case 4 above) and an undisclosed number of participants were in the process of applying to the university’s program. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Increasing class sizes Several interviewees claimed that using MOOC materials to flip the classroom has allowed more students to enroll in a class. For example, at University of California, Berkeley, where the MOOC course model has been adapted on-campus to what Fox terms a “SPOC” (small private online class), automation of certain aspects of the courses has allowed class size to grow, purportedly without sacrificing quality of the experience for the students. The biggest benefit of the SPOC in our setting is that we've been able to multiply by almost a factor of five the number of students who can take our class. The enrollment of my course has gone from 50 or 60 students about three years ago to 240 now. We're not reducing the quality of the course, we're not reducing the students’ evaluations of the course but we are accommodating more of them. We're getting more students through the system faster, students who no longer have to be waitlisted for a class they need. Historically, we were limited by the number of assignments we could grade and the number of TAs we could hire. Now the TA ratio hasn't changed but the TAs are spending their time quite differently. They're not grading assignments and exams because we have automation to do a lot of that. They're not rehashing what was said in the lecture because students can go watch that stuff online if they miss a lecture, or if they didn't get it the first time around. They're not creating finger exercises because we've included those along with the lectures; it's the same as the peer learning questions that we use. We put those online as well so students can review from them. The TAs can focus on higher leverage activities. (Fox, University of California, Berkeley). What is not clear, however, is whether these students are simply being diverted from other programs at the University, with no net increase in the numbers of students served by the institution as a whole, and hence no overall revenue increase. Licensing fees for use of MOOC materials by other institutions We came across several instances in which one institution had obtained a license to adopt all or parts of a MOOC created by another institution. However, because these were mostly experimental situations, no actual fee was exchanged. We only heard an actual dollar amount associated with such licensing from one university: The University System of Georgia office has a contract with Coursera that provides pricing for what it would cost if our faculty want to use the materials from one of their MOOCs. It is scaled by volume so the price actually goes down as the volume goes up. It is around $30 a student. (Renick, Georgia State University). It appears that the time and effort required to negotiate licenses for small numbers of users may not be worth the amount that can be feasibly charged. A licensing strategy may become worthwhile only at the level of mass use, for example, the recent agreement between the Kingdom of Jordan and edX where the former has contracted to pay a licensing fee to offer edX courses on the Arabic MOOC portal, Edraak.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Fees for additional services While Courseraâ&#x20AC;&#x2122;s Signature Track certification is already a source of revenue for both Coursera and for its partner institutions (as is presumably the case with edXâ&#x20AC;&#x2122;s XSeries Certificates), the level of uptake will need to increase substantially to exceed the costs of developing and delivering MOOCs. The American Museum of Natural History found that less than 1% of enrollees (or 20% of completers) in its MOOCs signed up for Signature Track. However, Pang Wei Koh, Director of Course Operations at Coursera, notes that Yahoo recently announced it would reimburse any employee enrolling in Signature Track courses. We expect that the uptake for Signature Track and XSeries certificates will increase as soon as more employers confer economic value on them by considering these credentials in hiring and promotion decisions. Other services are being considered such as tutoring, proctoring, and supplementary materials. Through our partnership with Coursera, we are experimenting with a couple of things. One is our Pearson VUE Centers where you take your GMATs and GREs. They've struck a partnership with Coursera to provide some certification or offline testing. (Rizvi, Pearson). Vuchic, formerly at the William and Flora Hewlett Foundation and a proponent of Open Educational Resources, while arguing that content should remain free, suggested various supplemental services that can be offered for fee: The model that grows the fastest, as we have seen in past years on the Internet, is when content is free and you monetize various things around it. If you charge at the point of content then suddenly every user has to go through a transaction to even engage in your system. Then you lose 80% of your users. You can charge for certification, services, data access, dashboards; you can charge a classroom fee for the teacher if they want reporting on quizzes every single night. Maybe even student-level data if they want that. (Vuchic, formerly at William and Flora Hewlett Foundation). In some MOOCs, the instructor recommends a self-authored textbook, a controversial strategy among students, and proponents of open educational resources. Grant revenues Several interviewees believe that MOOCs will improve eligibility for grants. In some cases, MOOCs are expected to act as a dissemination route for research, helping to satisfy a frequent requirement of funding agencies for grantees to demonstrate broad impact beyond the immediate participants in the research. For example, if a new curriculum for K-12 teachers was designed based on the latest research findings in a particular discipline, a MOOC could be used as a form of professional development to provide a large number of teachers with a scaffolded opportunity to implement the curriculum in their classrooms (Keyek-Franssen, University of Colorado System). In other cases, MOOCs serve as a target of research. We would like to put on MOOCs as a way to disseminate research products and then measure the reach the research product has as a result to show granting agencies what we're doing. It would be great to measure the increase in grant funding that we received based on this innovative way to get our research product out to the larger community. (Keyek-Franssen, University of Colorado System). Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

A large part of what we are doing is a research enterprise and there are funding sources to support that. That wonâ&#x20AC;&#x2122;t cover everything and is not a long-term model but has moderate sustainability across time, like the OLI initiative. (Collier, Stanford University). Matchmaking for employers While we did not come across any colleges or universities that were actively pursuing a match-making system with employers in which the employers would pay for some of the costs of MOOCs in exchange for connections to strong performers, a few interviewees hinted that this might become a possibility. Overall, it appears that revenue streams for MOOCs are slowly building but we do not expect the costs of MOOC production to fall significantly given the highly labor-intensive nature of the process. While these costs may be amortized across multiple uses and multiple years, they will still be additive costs to the institutions creating MOOCs. Unless these MOOC producers can offer credentials of economic value in order to attract fee-paying participants, or can use MOOCs to replace traditional offerings more efficiently, most likely by reducing expensive personnel, they will not be able to afford ongoing participation in the current MOOC experimentation. Free, non-credit bearing MOOCs are likely to remain available only from the wealthiest institutions who can subsidize the costs from other sources of funds.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Case 5: Georgia Tech’s Online Master of Science in Computer Science Program In 2012, Georgia Tech and Udacity announced the Online Master of Science in Computer Science (OMSCS) Program which would allow students to earn a degree for around $6,000 instead of the typical $45,000. An infusion of $2 million in private funds from AT&T is subsidizing the start-up costs. AT&T employees were encouraged to apply for admission and made up 20% of the initial student population in Spring 2014. The originally anticipated expenses and revenues for the program are publicly available online and indicate that revenues from the initiative should exceed costs by Year 3. While noting that both Udacity and Georgia Tech have made it clear that these financial and enrollment projections were speculative, and that actual plans are constantly evolving, we replicated these financial projections using the published assumptions in order to assess whether the model appears feasible as a mean to reduce costs and/or increase revenues for institutions of higher education. According to preliminary staffing plans from Spring 2013 (summarized in Table 3 below), a faculty member assisted by a Course Developer (for initial offering) or Head Teaching Assistant (TA) (for re-runs) will develop and teach each of the 20 courses planned for the program. In these initial plans, projected enrollments rise dramatically from 1,780 in Year 1 to 24,611 in Year 3. For the first year, eight new courses would be developed, each serving 127 students. Six new courses would be added in Year 2 with each course serving 282 students. Six more would be developed in Year 3, each serving 456 students. Once a course is developed, it will be re-offered three times each year in Spring, Summer, and Fall. In Year 3, for example, six new courses would be offered and 48 re-runs. In practice, course development has progressed faster than expected, and enrollment growth is being managed conservatively. Twelve courses are expected to be complete by Fall 2014, and four more by Spring 2015. After the first offering of each course, either the faculty member or a non-tenured lecturer will teach the re-runs along with a Head TA, selected by Georgia Tech, but employed by Udacity. In addition, Course Managers (CMs) at Udacity will provide technical and logistical support, at a CM to student ratio of around 1:390. Quality Assurance staff and managers will be Udacity employees. While auto-grading will be used wherever practical for course assessments, some classes require group projects that were, in the Spring 2014 semester, hand-graded by the instructor and graduate student TAs. Work is underway to optimize the grading of such projects. Courses are likely to be “refreshed” every three years on average. According to White, rapidly changing topics such as computer networks might need updating every year while more gradually changing fields such as theory and algorithms may last several years. With respect to the financial projections, we replicated the published spreadsheets, corrected a few minor computational errors, and added a very conservative estimate of costs for the Head TAs/Course Developers which appear to have been omitted (we assumed $10,000 per course, but if these individuals hold an M.A. or Ph.D. as planned, the cost will be much higher). These projections also do not include the graduate students serving as additional TAs in Spring 2014. We summarize projected program costs and revenues in Table 4, showing both the published numbers and our revisions. We stress that we did not conduct a prospective cost analysis of this program ourselves, only a review of the listed resources, cost assumptions, and projections. We do not therefore purport to be verifying the original estimated costs. The projections show an average cost per course of $226,000 - $284,000, but these numbers include both new courses and re-runs. While re-runs will not have development costs associated with them, as class size increases they will require additional instructional support staff. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Table 3: Preliminary Instructional and Support Staffing Plans for Georgia Tech’s OMSCS Program Instructional Staff and Support for OMSCS courses

Year 1

Year 2

Year 3

Projected total number of students enrolled in this year

1,780

10,147

24,611

New courses/Re-runs

8/6

6/30

6/48

Projected average number of students per class

127

282

456

Head TAs (1 per re-run course)

Course Developers ( 1 per new course)

Number of Udacity Course Managers (CMs)

Ratio of Students to CM

391

390

395

Average number of CMs per course

0.4

0.75

1.2

Ratio of Students to Quality Assurance staff member

1,953

1,977

1,951

Ratio of students to manager of CMs and Quality Assurance staff

6,509

6,590

6,505

Total no. of courses offered this year (new + re-runs)

Faculty members/lecturers

Sources: David White, Assistant Dean for Academic Programs and Executive Director of the Online MS in Computer Science, College of Computing, Georgia Tech; Online Courses Hosting Agreement Georgia Tech and Udacity.

Total projected program costs rise from around $3 million in Year 1 to around $15 million in Year 3 (using our revised estimates). If the enrollment projections hold, this will allow the costs per 36-credit degree to fall from an initial $12,820 to $4,212 by Year 3. Assuming revenues of around $150 per credit, this will yield a net loss for the program in Years 1 and 2, but a net gain of $34 per credit in Year 3 (excluding the university’s technology fee and the $2mm from AT&T). However, we consider a number of assumptions and plans in the projections to be at risk: • • • •

•

The expectation that 24,000+ students will be enrolled in Year 3 appears highly optimistic. The expectation that dropout rates will be as low as in Georgia Tech’s traditional program is not supported by data from other online programs where dropout rates are higher than for oncampus students. The very high ratio of students to Teaching Assistant and Course Manager in each course seem unlikely to provide adequate support for either the lead instructor or the students. The plan to develop only 20 courses in total and to re-run each one three times per year following their creation does not allow much choice to students who need 12 courses to complete the degree, and assumes that new students can be recruited into courses that are rerun every semester. More generally, the financial plans, even if they pan out as expected, do not accommodate any significant contribution to the indirect costs of Georgia Tech as a whole, with each student paying a fraction of the tuition paid by regular students.

While such a program could serve as a loss leader for the institution, attracting publicity and approval from policymakers, most institutions can ill afford the running costs of such initiatives. We suspect that AT&T will be a greater financial beneficiary of the program than Georgia Tech itself, having essentially arranged exceptionally cheap training for its employees through a tax-deductible contribution. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Table 4: Summary Projected Costs and Revenues for Georgia Techâ&#x20AC;&#x2122;s OMSCS Program Non-highlighted items were derived from the published financials. Highlighted entries show our own recalculations with minor computational errors corrected and the addition of a Head TA/Course Developer that we believe was mistakenly omitted. Estimated/projected item

Year 1

Year 2

Year 3

Projected total number of students enrolled in this year

1,780

10,147

24,611

New courses/re-runs

8/6

6/30

6/48

Projected average number of students per class

127

282

456

Estimated total costs of program

$3,138,800

$7,520,200

$14,399,700

Head TA/Course Developer added at $10,000/course

$3,375,165

$8,127,983

$15,311, 332

Cost per course offered (including new + re-runs)*

$224,200

$208,894

$266,661

Head TA/Course Developer added at $10,000/course

$241,083

$225,777

$283,543

Estimated cost per credit to GT and Udacity combined

$331

$141

$110

Head TA/Course Developer added at $10,000/course

$356

$152

$117

Estimated cost per credit to Georgia Tech only

$104

$35

$19

Head TA/Course Developer added at $10,000/course

$129

$47

$26

Student services cost per credit

$45

Total degree cost for 36 credits

$11,920

$5,072

$3,952

Head TA/Course Developer added at $10,000/course

$12,820

$5,472

$4,212

$145

$149

$151

(Loss)/profit per credit**

($186)

$41

Head TA/Course Developer added at $10,000/course

($211)

($3)

$34

Total no. of courses offered this year (new + re-runs)

Revenue per credit from tuition + fees**

*The published numbers do not allow for a separation of costs of re-runs from new courses. ** Excluding revenues from technology fee and the $2,000,000 grant from AT&T in Year 1.

Source: http://s3.documentcloud.org/documents/703593/udacity-gtrc-amendment-5-13-2013.pdf

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Goal 4: Improving Educational Outcomes Thirty-eight percent of the institutions participating in our study that were offering or using MOOCs, and twenty percent of our interviewees overall, expected MOOCs to lead to an improvement in educational outcomes, some directly within the MOOC format and some indirectly through the transfer of new strategies and techniques to on-campus teaching. This goal was mentioned by representatives of five public universities, four private universities and two community colleges. Ironically, most of these interviewees were at institutions that would be considered among the elite. My real issue is teaching people. I know how to teach people in my research group to be experts. I am now seeing how un-expert our whole undergraduate system really is, how we teach students to do problems in a certain format by some special shortcut way, where they don't really understand it as we think they do when they do it. I want to really try to learn how to teach more people to be experts. And I want to use the MOOC to do it, to accelerate the process. (Pritchard, MIT). Professor Sanjay Sarma, Director of Digital Learning at MIT, expresses concern about current teaching practice and optimism for the role MOOCs can play in rethinking it. We have looked at Eric Mazur’s research about brain activity during class and see that the way we are teaching now may not be the best. In some ways MOOCs are a Trojan horse to help our colleagues rethink the way they teach. We have to deal with some cultural barriers, for example, professors think lectures are pretty good. (Sarma, MIT). Sarma reported that one of MIT’s goals for using MOOC resources and technology is to allow students to come into class more prepared so that instructors can re-deploy attention to one-to-one teaching. Fox, at University of California, Berkeley, concurs that MOOC materials can be used to optimize instructor time. If you can create materials in a way that instructors can spend more of their time on the things that differentiate them, working with individual students or adding some of their own supplementary materials and ideas to provide a gentler on-ramp to a subject, that's a better use of their time. (Fox, University of California, Berkeley). Interviewees mentioned several ways in which MOOCs may lead, directly or indirectly, to improvement in educational outcomes: • • • • • • • • •

Motivating instructors to rethink pedagogy Course re-design “Chunking” lectures and interspersing questions Fine-tuning instructional materials Providing instant feedback Gamification and badging to increase motivation Outreach to participants to encourage persistence Adaptive learning/personalization/mastery-based learning Using MOOCs in K-12 to prepare students for college

We elaborate on each of these strategies below, providing examples of where they have been used. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Motivating instructors to rethink pedagogy For a number of interviewees, the most significant impact of MOOCs has been on the motivation they have created for instructors to rethink how they teach. As Siemens observed, “Regardless of format, the passion of teaching is ignited for faculty teaching MOOCs.” In addition to students learning the specific content, methods, tools, and skills of the academic discipline on which a MOOC is based, additional objectives of cMOOC instructors included providing an “atelier” environment where students and instructor work together in “a social and technical network” to create artifacts, and a permanent digital space to store creations. Several interviewees reported that MOOCs have prompted even typically intransigent faculty members into reconsidering their teaching styles and have facilitated faculty professional development, which might otherwise be resisted. A lot of the professors have not had to teach students of such diverse backgrounds before so they probably think about audience and what is needed for the students to be successful more deeply than ever before. This is common for online courses in general. I have been in faculty development for 10 years now and we call this “guerilla-style” faculty development because when you ask a faculty member to teach online, it is often the first time they think about their course design. Because they are willing to get input on the technology-related issues, you can subvertly throw in some ideas about course design, addressing different student audiences, writing learning outcomes. But you can see with MOOCs that faculty are really thinking about what they want students to get out of the experience, especially as they are not being motivated by course credit. “How do I motivate them, how do I think about who they are, what does it mean to be successful, what will they take away from this? I think they now think differently about even their on-campus, Stanford students.” (Collier, Stanford University). Guthrie at ITHAKA also believes that instructors are changing the way they design and deliver courses as a result of their experiences with MOOCs, and has no doubt that instructors feel that their teaching has improved, but he points out a lack of evidence that better learning experiences are actually being created for students. As he illustrates, change in practice is not inherently positive: If I sit down to write a speech and I have to write down every word, I have to do a very different thing than when I just stand up and start talking off the cuff. I have to think about it much more carefully; it's much harder work. That doesn't necessarily mean my speech is better, but I definitely understand the material better, I have to frame it very carefully. I think faculty members are having that kind of experience when they shift over to delivering a MOOC. (Guthrie, ITHAKA). Course re-design Many interviewees recounted that faculty member engagement in MOOCs has led to re-design of their courses. Ugoretz at CUNY observed the inherent capacity for online experiences to provide instructors with feedback on their teaching: One of the things I find most powerful about online learning is that, for the teacher, the usually ephemeral information is concrete, and it's right there. You can look back at it to modify your teaching. I've walked out of a face-to-face class thinking "That was a great class!" or "That really didn't go well." I may remember some of it, but I don't remember exactly where it went wrong. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

With online discussion, you can go back and you can see, "OK, here's where I said the wrong things," or "Here's where I should have steered us back, or let it go further in the other direction.” (Ugoretz, Macaulay Honors College). Often, design changes are made initially to conform to the typical xMOOC format of multiple, short videos with embedded questions, weekly quizzes, and assignments. However, many instructors have found that the process pushed them to reconsider how to present concepts and materials. They have also discovered that some of these changes can be adapted for on-campus courses. Sarma at MIT promotes the concept of “modularity,” noting that shorter courses are both easier to create and to complete. Courses created for the edX platform have generally been shorter than on-campus classes and have led to the “modularization” of some on-campus courses. Box 4, Case 6, and Case 7 provide examples of how on-campus courses were restructured as a result of the instructor either teaching a MOOC or integrating one with his class. The most popular strategy we encountered for re-structuring instructional time was “flipping” the classroom, a concept that significantly pre-dated MOOCs. MOOC material is made available for students to work on independently while face-to-face instructional time is optimized by engaging in problem-solving and discussion. For example, Harper at the University of Oklahoma re-designed his on-campus course Law and Justice putting short videos, quizzes, an interactive textbook, and discussion components online, while in-class time was focused more on discussion and debate. Another common, but pre-existing, strategy being introduced in many courses is peer learning. Guthrie at ITHAKA believes that the most powerful learning innovation that MOOCs present is the ability for “many to many” communication and that leveraging this capability may produce the most learning gains for participants. Siemens notes that large enrollment numbers permit self-organizing “sub-clusters” of interests and introduce a broad variety of interests. The University of Oklahoma has been working with NextThought to deliver online courses that exploit the potential of online communities to foster sociallyconstructed learning (see Case 8). Experimentation with large-scale peer assessment techniques have led to improvements in peer-to-peer rating strategies with rubrics developed and iteratively revised to maximize clarity and consistency of grading. James at Mt. San Jacinto Community College helped design the college’s Crafting an Effective Writer MOOC which included three assignments that were reviewed by peers and a culminating assignment that was peer-graded. She observes that “The rubric has to be incredibly detailed for it to work.” Some interviewees were less sanguine regarding the current status of peer grading and its appropriateness. For example, Michael Cennamo, Educational Technologist, Columbia Center for New Media Teaching and Learning (CCNMTL), Columbia University, remarked: “We are not ready to take that idea to thousands with no credit, no risk, no liability.” A number of interviewees suggested that the integration of MOOC materials created by other faculty members into their courses, a concept often termed “wrapping” or “embedding” a MOOC, in a fashion parallel to the use of textbooks, will eventually become more accepted practice and will benefit students (see Fisher, in press). Fox at University of California, Berkeley coined the term SPOC (small private online courses, see Fox, 2013) to describe one version of this application of MOOCs. SPOCs are a big part of the equation and are not getting enough airplay. This is going to change the way we think about curricular technology transfer. As instructors, we don't all use the textbooks that we wrote. Most of us use other people's textbooks. We don't feel that we have Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

abdicated control of the course just because we didn't write the textbook. It's taken us a fair amount of time to get acclimatized to that point of view, but I think we've got to acclimatize to this one too: that if you use somebody else's MOOC materials, that doesn't mean that you've given up your control of the course. It means their materials are really good and you think you can do a better job helping students if you work with those materials than the ones you had before. That's an unsung but really interesting opportunity. I don't think it's going to be free, but I'd like it to be very cheap for an instructor to just click the clone button on a MOOC and say, "I want a version of this to use in my own classroom." (Fox, University of California, Berkeley).

Box 4: Rethinking the Design and Delivery of Engineering Software as a Service at UC Berkeley

“When we did the ‘Engineering Software as a Service’ MOOC the first time around, we were advised by

our colleagues, ‘Don't just record your hour-and-a-half-long lecture.’ The hour-and-a-half-long lectures from previous runs of my course have been online for years. But my colleagues had pioneered a format of 10-minute segments with frequent self-assessment questions for the students. We thought that if we were going to do the work to create these questions, we should find a way to have the on-campus students benefit from them as well. We decided to use peer learning. We restructured the lectures oncampus with the understanding they were going to be captured live and edited after the fact. Now we talk for 7-10 minutes and then we stop and ask a question. The students discuss it with their peers, and they vote on the answer. This has kept the students much more engaged in lectures, so much so that even if I never taught another MOOC again, I'm going to keep doing my lecture-planning this way. It turned out that was a much better format for the live lectures to begin with. Now, you could argue that the peer learning literature has said for years that this is a good idea. So why do you need MOOCs as a kick in the pants to do it? Because in MOOCs you are forced into certain design decisions that you wouldn't necessarily be forced into in a campus class, and the realization is that the new design actually would have been good for the campus class if you had the initiative to try it. In fact, the peer learning strategy on-campus has worked out so well that I now feel bad for the MOOC students because they just get a self-assessment question. The students in the classroom get a chance to discuss that question with their peers before they decide on the answer, which study after study shows helps them learn the material better. We're actually in the process of doing an experiment right now to see if that aspect can be brought back into the MOOC. Students could have small, virtual chats within the MOOC with colleagues in a virtual chat room and then answer the question together. We're going to see whether, in an A/B test, students who answer the question that way are more likely to not only get their question right, but also do better on those questions in the assessments and the quizzes. We're doing a dry run of this in a MOOC now. And we hope to do a production run with an in vivo experiment in the spring.” Source: Armando Fox, University of California, Berkeley.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

“Chunking” lectures and interspersing questions Rebecca Petersen, Director of Research and Educational Initiatives at edX, reports that early indications of video-watching behaviors show many MOOC participants will watch only three to five minutes of a video clip, regardless of the actual length of the clip. Many MOOC videos, which are rarely longer than 10 minutes, are now interrupted at regular intervals with “inline” questions to check the viewer has grasped the foregoing content or can apply it to a problem. Udacity takes this strategy to the extreme with questions interspersed as often as every two minutes. Numerous MOOC instructors, including Fox at University of California, Berkeley (see Box 4) and Racaniello at Columbia University, have taken the strategy of “chunking” lectures and interspersing questions back into their live classrooms. One thing I like to try is to ask questions during my live lectures using clickers. I did not really see the use until I did my Coursera course. I liked the idea of pausing and making sure that everybody knows what is going on. So it is like a video quiz question, but live. It is a pause. In Coursera you break the lecture into chunks so that students can digest a chunk. So I can do that in my lectures too: stop and see if they really get it by seeing the results right there. It will slow things down but it will be better for their comprehension. I am going to do that in Spring. I think I am also going to be aware of thematic chunks in my lecture because of this. In fact, I was thinking of posting the Coursera videos. They are ours to do as we want. Right now I post lectures that are an hour and a half. What if I post 15-minute chunks [for the on-campus students]? Maybe somebody busy can watch for 15 minutes. That may help them learn it better. I could cut it up right away for the Columbia course and it will be all ready for Coursera. (Racaniello, Columbia University). Fine-tuning instructional materials In some cases, rather than re-designing an entire course, instructors are fine-tuning instructional materials, for example, substituting a textbook and other publisher material with open educational resources. Several community college interviewees noted that the move towards free texts that began with the Open Educational Resources movement, and is being adopted in some MOOCs, was a significant benefit to their students who often struggle to pay for textbooks. Sarma at MIT described the development of “instrumentation” to measure activity in MOOCs and allow adjustments to improve instruction, for example to identify concepts that students appear to be stuck on, as evidenced by multiple video rewinds or attempts at a question, and then revise the content to provide alternative explanations. But Jaggars at CCRC notes that the current turnaround time for analyzing most MOOC platform data is too long to allow for mid-course corrections. Loble, formerly at University of California, Irvine Extension, listed several adjustments being made to both the university’s “traditional” online offerings as well as to future MOOCs as a result of participant feedback in the initial MOOCs. These include posting more videos of the professor; being more explicit in expectations and directions to students; and establishing an online “getting-to-know-you area.” Another lesson learned was that the MOOC medium does not work for all content. Providing instant feedback Fox, who has run his Engineering Software as a Service MOOC several times and has built sophisticated auto-graders to assess student code, believes that the MOOC format affords students more frequent Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

opportunities to demonstrate mastery and receive instant feedback than were available when all assignments were graded manually. He acknowledges that there is still room for improvement. For example, the auto-graders are not yet capable of providing feedback regarding style of coding, as opposed to simply whether it is functionally correct. At MIT, Professor Michael J. Cima’s experiment with Introduction to Solid State Chemistry, described in Case 6, is a testament to the power of regular and instant feedback. While many observers are skeptical of the applicability of auto-grading to anything but multiple choice or numerical assessments, Ryan Baker, Associate Professor at Teachers College, Columbia University, NY is more optimistic. I actually don’t share the skepticism about computer grading: computers make different errors from humans but when you look at agreement between computers and humans for best practices text scoring work, it’s actually about the same as agreement between humans and humans. (R. Baker, Teachers College). Gamification and badging to increase motivation Faculty members and administrators involved in designing and delivering Tri-C’s Pre-Algebra CCC MOOC (see Case 2) found the MOOC format lent itself well to one of the more important tasks in serving their students: encouraging persistence. One way in which this was accomplished was by introducing elements of game mechanics into the course so that students could progress through four increasingly challenging levels of activities. However, as Thackaberry noted, gamification is expensive to execute. What we did was kind of the low-rent version of game mechanics, using Adaptive Release and Blackboard. Actually, it mimics some of the most successful use of educational gaming, because it doesn't overwhelm the user, so that the student can focus on the simplicity of conquering the content, rather than learning how to navigate in a virtual world. That's also good news for us in terms of expense. We're not Sony PlayStation. We can't spend $5 million developing an immersive game. (Thackaberry, Tri-C). While several other interviewees mentioned gamification as a strategy to engage participants in online environments, only two other institutions described actual examples of gamification in their courses. A second strategy Tri-C used to encourage persistence and “good online citizenship” in the Pre-Algebra CCC MOOC was the incorporation of Mozilla Open Badges. Royal, who was instrumental in the development of the Tri-C MOOC before moving to Inver Hills Community College, describes the reasoning behind incorporation of badges: The concept of badging is a great one, we see it in a lot of social aspects within our society and I think there is an opportunity to use the same principles in educational activities to help encourage, support, and motivate students to complete whatever their goals are for their educational path. It has become pretty standard for the millennial generation and people entering college to have digital representations of themselves: having avatars or profiles. So badging seems like a natural extension to that. (Royal, Inver Hills Community College).

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Outreach to participants to encourage persistence A third strategy to encourage persistence in Tri-C’s Pre-Algebra CCC MOOC was the implementation of motivational outreach and engagement activities by the MOOC instructors. According to Thackaberry, from one iteration of the course to the next, faculty members increased the number of special announcements and e-mails including “video snippets and other attention-grabbers.” In some cases, these were directed to all students while, in others, messages were targeted to individual students who had become inactive in the MOOC, in an effort to persuade them to return. Miles describes these efforts as purely motivational rather than content-related: “We were emphasizing greater engagement with the content, or directing students to the content, or supporting them through the content.” Course size was small enough to make it possible to monitor and interact with individual students: just over 300 students were active in the fourth and final facilitated iteration of the course. These motivational tactics led to an increase in the course completion rate from 12.5% in the first iteration, to 19.3% in the last, based on the number who completed a mandatory pre-survey. If it were possible to replicate these motivational strategies in larger MOOCs, it is likely that the completion rates would increase, at least for those participants who start with the intention to complete the course. In Goal 6: Research on Teaching and Learning, we report on several research efforts working in this direction. Adaptive learning/personalization/mastery-based learning One of the pedagogical attractions of online learning is the possibility of providing adaptive learning experiences, that is, providing a learning trajectory for each individual student that responds to ongoing assessment of how that student is performing. For example, if a student is struggling with a concept, the learning platform may direct him to a prior learning experience and only return to the original trajectory once he demonstrates a grasp of the prior concept. Students that master the targeted skills and content quickly can complete the course at their own pace without being held to a “seat-time” model of learning. Stephen Laster, Chief Digital Officer at McGraw-Hill Education, asserted that particularly for MOOCs which are open access, there must be recognition that participants will begin the course in various states of readiness and bring with them many different learning styles. He sees value in providing a preassessment that determines the prior knowledge and skills of participants. Such an assessment could be used to advise participants whether they are ready to take the course or to direct students to more scaffolded material. By allowing more accurate targeting of the course to participants, the completion rates may improve. Petersen at edX indicated that a few MOOC developers are indeed using preassessments for learners to judge for themselves their skill level relative to the content being presented. But she added that, for now, it is up to the learners to find their way through the materials; it is not the system that responds to a learner’s performance and skips him ahead or re-directs him to polish a skill. Jaggars at CCRC questions whether true adaptive learning will ever be affordable: I'm skeptical that a for-profit company would be willing to invest in the huge amount of research and development necessary to create truly high-quality learning software. Most companies seem to invest just enough to get the product to sell, not to get the product the best it can be. They may think their return on the additional investment would be too small to be worth it. Some of the best learning software we’ve seen was developed at research institutions with support from foundations, but I’m not sure whether a philanthropic development model is sustainable. (Jaggars, CCRC). Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Based on her assessment of the current state of adaptive learning environments, Griffiths of Ithaka S+R suggests that truly adaptive learning is still an aspiration rather than a reality: If 10 is the ultimate realization of adaptive learning environments to achieve their full potential with feedback loops and continual improvement, we are probably at a 2 with the tools we have now; they are pretty primitive. I don’t doubt that it will happen, but we are just in the really early days of figuring out at what level these tools can make the most impact. With the technology we are using now, we can just change the sequence of the learning objects in a way that is supposed to reflect the way that student learns, what he already knows, and what he needs to focus on. That doesn’t seem to make much difference to the student’s instruction: they are getting through the same material. It is not a transformative experience. The instructor can choose to turn on the adaptive feature or not and it’s not that big a difference. (Griffiths, Ithaka S+R). Using MOOCs in K-12 to prepare students for college A few interviewees believe that MOOCs may be successfully blended into advanced courses at the high school level to help prepare students for college. For example, Pritchard at MIT believes that MOOCs could be integrated into Advanced Placement classes, Baccalaureate programs, or “A” levels in the United Kingdom. There have already been a few reports of MOOC participation by high school students, both as part of their school work and outside of it. For example, McGrory (2013) reports that high school students in Florida began to participate in MOOCs after Governor Scott “signed a law allowing MOOCs in subject areas with end-of-course exams, including algebra, geometry and biology.” According to McGrory, Broward College, FL is offering a MOOC on college-level math, reading, and writing with half the participants being high school students from Broward County. St. Petersburg College, another community college in Florida, has offered a remedial math MOOC in which high school students from Pinellas County participated. It is not clear, however, what the impact of such MOOC participation has been for these students. Participants at the recent NYSCATE New York State Online Learning Forum (February 6th, 2014), suggested that while the costs to produce a MOOC would be prohibitive for regular high schools, there may be possibilities for existing MOOCs to be pursued by high school students in independent studies, and for teachers to participate in MOOCs for professional development. In New York State, school principals may decide whether credit can be earned for participation in an online course, but Paska, of Harrison Central School District, raised a number of policy challenges that limit the extent to which MOOCs can currently be used within existing K-12 curricula. For example, in New York State, while online courses may be offered by districts, they may not be used to reduce teaching staff, and the students must interact with a district-based teacher of record. Paska indicated that most states cap the teacher’s student load and require that online courses be taught by a state-certified teacher. Doubts about the capacity of MOOCs to improve teaching and learning We came across several skeptics regarding the potential of MOOCs in their current form to improve educational outcomes. As Kolb of RPI pointed out, universities and colleges are “not only delivering content, but trying to create learning environments” while MOOCs are “… for the most part, just taking a swing at the content piece.” Kolb worries that xMOOCs fail to encourage divergent thinking and contribute to higher education’s propensity to “knock creativity out of our students.” For David BenDaniel, Berens Professor of Entrepreneurship at the Johnson School, Cornell University, MOOCs simply cannot accommodate the style of teaching he finds most effective: Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

...my teaching is entirely Socratic: back and forth, give and take with the students. It is something intimate with the class. I walk up and down the aisles. I stop and say “What do you think about this?” I cannot possibly imagine being able to “can” one of my courses where a discussion group can answer all the questions that come up. You cannot duplicate in a MOOC the same rapport that can be created with a limited number of students in a face-to-face class. The use of humor is an example. (BenDaniel, Cornell University). Hersh, formerly at Santa Barbara City College, concurs with this view asserting that “… right now the best trend in learning goes back to Ancient Greece with the Socratic Method.” Hersh observes that while MOOCs may address some of the cognitive aspects of learning, their lack of “human presence” results in a failure to address important affective aspects. He expects that unless MOOC designers devise strategies to foster a greater sense of community and provide additional supports, perhaps in the form of virtual TAs or a “Siri” equivalent, completion rates are likely to remain low. Li Fengliang, of Tsinghua University in China, does not expect MOOCs to have a significant impact on higher education because he doubts that they will be effective educational tools for any but those learners who start with a high degree of interest and ability. Dede, of Harvard, cautions that only a very small number of people are able to learn from instructional materials without any additional form of support and argues that MOOCs will only become successful on a broader scale if peer learning strategies are enlisted. Moreover, Dede feels that the best learning experiences will arise between the cMOOC and xMOOC extremes. Kalbaugh, from Wake Technical Community College, reacting to the recent PCAST letter to President Obama concerning the potential of MOOCs, similarly expresses concerns about which students may benefit from them. From where I sit, I also have grave concerns about new students; underprepared students; older returning, students having the ability to truly learn in such a radically different environment from what they are familiar with. Those are the very students who are finding postsecondary education difficult and who may have the most difficult time with a MOOC environment. For highly capable and prepared students, MOOCs may very well be a great alternative, but these are not the at-risk students that the PCAST letter indicates MOOCs will help. (Kalbaugh, Wake Technical Community College). Many interviewees pointed to the pilot initiative conducted by San José State University with Udacity (see Box 6 and Firmin et al., 2013) as evidence that MOOC-based courses are ineffective for at-risk learners. Dede at Harvard, Thille at Stanford, and others who have spent many years perfecting the process of designing and re-designing online learning experiences, lament the inattention to existing research and knowledge about what does and does not work in online learning: “We are starting without reference to history and making the same mistakes over and over again” (Dede). Thille remarked that much that has been learned about effective design in the past is not only being ignored, but “rediscovered.” Bates at Contact North is, however, hopeful that with universities having prior online experience now joining the MOOCscape, this scenario will change: I’m still seeing speakers from MIT and Stanford deliberately ignoring previous online learning developments as if they were unimportant. This does a great disservice to the many credit-based online learning programs that pioneered good quality course design based on research and Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

evaluation of what works and what doesnâ&#x20AC;&#x2122;t in online teaching. However, on a more positive note, Iâ&#x20AC;&#x2122;m seeing an increasing number of MOOCs moving away from the video lecture format into better designed, more interactive models, as universities with prior experience in online learning have been getting into the MOOC market, so perhaps eventually MOOCs will find their true niche in the market. Stan Silverman at NYIT suggests that MOOCs are failing to leverage technology to best effect in educating people: We are not convinced that a massive approach to anything works. You homogenize it. We're now talking about customizing learning. The word massive suggests you're going in exactly the opposite way. It contradicts the basic philosophy that we think the technology is best used for. (Silverman, NYIT). Guthrie at ITHAKA points out that it requires substantial investments to realize value from the enormous amounts of MOOC platform data in order to improve educational outcomes: I think that even the platform providers are going to be struggling to figure out how to turn that data into analysis that can improve instruction. I haven't heard that people have been able to turn that data into impactful changes yet. (Guthrie, ITHAKA). Bror Saxberg, Chief Learning Officer at Kaplan Inc., is optimistic that MOOCs can help provide certain skills but does not expect that they will help prepare participants to succeed in complex tasks in the workplace: I don't think MOOCs are any kind of answer at all to building expertise that is relevant in the workplace. I think zero-variable-cost solutions are enormously important at the beginning of your trajectory towards becoming highly capable in the workplace. But we know that, on top of technical skills, it takes communication skills; judgment; a whole range of complex, grey areas to be successful at work. That's going to require feedback from professionals who are already doing it in order to hone your skills, if the objective is to get you as ready as possible to do the work at an expert level when you finish your training program. So, how far can zero-variable-cost solutions carry you? I think in some areas like computer science and computer programming, they might carry you pretty far, but not all the way, because you've got to talk to business people. You've got to talk to other engineers. You've got to manage people to do a complex project. All of those things are going to need human coaching and experiences with human beings. (Saxberg, Kaplan Inc.). While interviewees provided many example of how MOOCs have been used to change instruction, for the most part, actual impact on educational outcomes has not been measured in any rigorous fashion, if at all, so it is unclear whether the goal of improving educational outcomes has been achieved. Case 6 and Case 7 highlight two exceptions where instructional changes and their effects on student performance have been carefully documented. While not gold-standard, random assignment experiments, they provide promising evidence of improvement in student performance as a result of integrating MOOCs into flipped on-campus courses, or adopting some of the MOOC strategies such as frequent assessment and automatic feedback. In the section Goal 6: Research on Teaching and Learning we review more formal research that has been conducted on MOOCs.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

Case 6: Course Re-design to a Mastery-Based Model: Michael Cima, MIT Michael Cima, Professor of Materials and Science Engineering at MIT, has been teaching solid-state chemistry on-campus for 28 years and has been instructor in charge for three years. He was invited to offer the course as a MOOC in the Fall of 2012 and, over a period of around six months, spent several hundred hours re-designing the course to conform to the current xMOOC format. Once the MOOC Introduction to Solid-state Chemistry was completed, Cima decided to experiment with some of the MOOC instructional strategies in his face-to-face course to see if he could improve learning outcomes for the on-campus students. He surveyed students who had taken his course in previous years to solicit feedback on potential changes to the course design, ranging from eliminating lectures to no change at all. Not surprisingly, he found that the top performers were inclined towards no change in the course design while those who had struggled were more enthusiastic about changing the assessment format. Cima’s goals for the re-design project were: • • •

To experiment with new forms of assessment that might stimulate more learning and more accurately measure what students know. To ensure that all students kept up with the material from the beginning of the course rather than wait for the customary MIT “fifth-week warning flag” to notify students who were not meeting expectations. To bring up the bottom of the class and raise the pass rate of the course to 100% from an already high rate of around 97%.

The re-designed on-campus course was delivered in the Fall of 2013. It involved three lectures and two recitations per week, as in previous iterations, but a requirement to attend 80% of the sessions was instituted. The course text was replaced by the online content from the MOOC. This content consisted of “learning sequences” which students could use as much or as little as they needed. Each learning sequence is five minutes of me at a chalkboard, then five minutes of a screencast, like a Khan Academy kind of thing. Then there may be a practice problem that you can do. It alternates like that for, on average, about 40-50 minutes of video podcasts. It's like a lecture, but I think it's more interesting than a lecture, because it breaks it down into individual concepts and you are asked to attempt a problem related to the material the lecture's about. I actually like the screencast better than the lecture, because I find my tone is very different. I've been doing this 28 years. Give me a piece of chalk and put me in front of a blackboard, I'm lecturing. If I'm doing a screencast, it's as if a student is standing over my shoulder. The work I do is so completely different. I get very positive responses to my screencasts.(Cima, MIT). Cima found that putting the content online was particularly helpful in addressing the fact that students regularly enter the course with a wide range of background in chemistry, ranging from almost none to three years’ worth of high school courses. In practice, he estimated that around 50% of the students used the online content. Course assignments and assessments were overhauled to replace homework, mid- and end-of-term exams with 14 weekly, online, proctored opportunities for students to check their understanding and demonstrate mastery. The online assessments (a total of 37 spread over the 14 units) were constructed from problems that had been collected in a database over the previous 40 years for use in mid-term and final exams. Each week, students were expected to attempt problems delivered on a platform specially designed for the class by Cima and Isaac Chuang, MIT Professor of Physics and of Electrical Engineering and Computer Science. The platform presented students with problems selected Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

100

at random from the database. In some of the problems, even the numbers used within the problem were randomized to ensure different students received different versions. Assessment problems were not straightforward multiple choice questions that students could guess at or rely on the answer options for hints. According to Cima, 70% of the problems required “floating point answers", that is, a calculation had to be performed and the numerical answer entered online. Students were usually permitted up to three tries for each of these questions. Where multiple choice questions were used, Cima presented “tiered sequences where you have to enter all of them completely correctly to get the assessment done. If you submitted it with one of those things wrong, you'd have to do the entire assessment again.” Multiple choice questions could only be attempted once. Weekly assessments were completed in a proctored assessment room that had been set up for the course with tables, Chromebooks, and Internet drops. The room was staffed with proctors for three hours a day, six days a week, and had seating capacity for 40 students at a time. Assessments were automatically graded, providing instant feedback to students, and after a 24-hour lock-out period each time, could be attempted multiple times during a two-week window. Allowing multiple attempts was an adaptation to compensate for the fact that partial credit could not be earned for solving problems partway without reaching the correct answer. Teaching assistants (TAs) were available for consultation before or after the assessments in a room adjacent to the assessment room, so that students could resolve any difficulties on the spot. Cima noted that the assessment room and TAs were used much more heavily than expected and that twice the seating capacity or twice the number of hours would have been optimal. Cima surmised that, at mid-term, the 361 students were, on average, making two attempts at each of the assessments. Cima was astounded at the increased effort that students made in practicing problems and reattempting assessments in order to satisfy the course requirements. For the past three decades, you could have passed 3.091 [Solid-state Chemistry], never doing a problem completely correctly, as long as you earned enough partial credit. Now they have to do a minimum of 27 [entire assessments] completely correctly, and most students did many more than that completely correctly. I think the average was 33 of the 37. I think the problem for decades has been to get students to actually practice problems. Cima admitted that he cannot be sure that students have actually learned more from the re-designed course, despite apparently better performance on the assessments. He does, however, feel that the new assessments are a better way to determine what students have learned. His biggest concern about the new design is whether students will like it. We know that this activity, this term, was much less stressful than the typical course. Virtually all the freshman advisors are recording that. It's a question of whether you feel, at the end, whether you've learned something. There's this thing at MIT, if it's not hard, it's not worth doing. That's systemic at MIT. I gave this talk, an update in my department, about what we've done this term. Literally, that's the kind of questions I get: "Shouldn't it be hard?" "Wait a minute, it's freshman chemistry, and it's not hard?” That's going to be a big challenge here at MIT. Not just with the faculty, but with students who say that, "Gosh, this is so much easier; maybe I'm not learning anything?" There's this belief that cramming for an exam, you learn something different than if you do it this way, even though these are the same problems. They say, "Well, I can study this for this assessment, so it's not like studying for a final, where I don't know what I'm going to get." Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

101

Cima reported the following outcomes as a result of the course re-design: • • •

• •

The new format of the class appears to have attracted more students. The typical class size of 320-325 students grew to 361. Despite various glitches during the first two weeks, few students switched to the alternative, traditional chemistry class offered on-campus. Part way through the course, Cima surveyed the students to ask if they preferred the weekly computer-based assessments or a written exam. Around 80% of the 290 respondents preferred the weekly assessments. The number of fifth-week warning flags issued dropped from the typical 30-60 (around 15% of the class) to three (less than 1% of the class). Cima believes that getting feedback on performance starting the first week, rather than waiting till Week 5, allowed students to adjust their study habits immediately to meet the course expectations. Students reported spending eight hours per week on the course compared with 8.3 hours per week for the previous year’s cohort. Cima believes the new format helped the weakest and least prepared students the most. All but two students passed the assessment requirements of the re-designed course. “There is no bottom of the class now.” However, eight students did not meet the attendance requirements for lectures and recitations. Some of the stronger students complained that the new assessment regime denied them the opportunity to “shine” because all students were able to do well if they made enough effort. A comparison of student performance in 12 topic areas that were similarly tested in both 2012 and 2013 indicated that average performance for the class in 2013 was significantly higher on every topic. The smallest gain was observed for Conductivity of Semiconductors where around 15% of the 2012 class obtained full credit compared with just over 50% of the 2013 class. The biggest gain was observed for Acids and Bases where only 2% of the class obtained full credit in 2012 compared with over 95% in 2013.

If Cima offers the re-designed course again, he may consider including periodic comprehensive assessments. While confident that the impact on learning outcomes has been positive, he cautions that this process has been labor-intensive and costly, with 33 different people contributing to the course design and delivery, including 11 TAs and a team from the Office of Digital Learning to “help with software infrastructure.” Demand on space and TA time for the assessment room is much higher than for a regular class. He estimates that the re-designed course cost two to three times more than a regular course.

Sources: Professor Michael J. Cima, MIT; Professor Sanjay Sarma, Director of Digital Learning, MIT; Cima, M.J. (2013). My experience teaching 3.091x. MIT Faculty Newsletter, 26(1), 15-17. Retrieved from: http://web.mit.edu/fnl/volume/261/cima.html Cima, M.J. (2014). A mastery-based learning and assessment model applied to 3.091r. (Introduction to Solid-State Chemistry). Internal MIT Report.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

102

Case 7: Re-designing Introduction to Circuit Analysis: Khosrow Ghadiri, San José State University Introduction to Circuit Analysis, a bottleneck electronics and engineering course required of all engineering undergraduates at San José State University (SJSU), traditionally yields a 65% pass rate. The course appeared to be a suitable candidate for a make-over in an effort to improve the passing rate and allow students to progress with their degrees in a timely manner. Using edX course materials, course instructor Khosrow Ghadiri re-designed the 15-week Introduction to Circuit Analysis into a flipped classroom. The re-designed course was offered as EE 098-MIT, one of the three sections of Introduction to Circuit Analysis, in Fall 2012. Eighty-six students were registered for Ghadiri’s section and were given the option to enroll in the new format or in the traditional course offered in another section. None switched out of the re-designed section (Ghadiri, Qayoumi, Junn, Hsu, & Sujitparapitaya, 2013). Ghadiri et al. (2013) describe the highly structured nature of in-class sessions and out-of-class requirements each week. Students in EE 098-MIT were granted full access to the online materials from Anant Agarwal’s MIT Circuits and Electronics MOOC and also to Agarwal’s electronic textbook. They were assigned sections of the MOOC lecture videos, embedded questions, auto-graded problem sets, and labs to complete outside of class, and were expected to show up in the classroom prepared to spend 5-10 minutes answering questions about the assigned material. Ghadiri also asked students to rate concepts presented in the MOOC materials in order of difficulty using feedback forms that he created. Based on these ratings, he spent 20 minutes of class time explaining concepts that students had rated most difficult. Subsequently, the class broke up into groups of three to work collaboratively on problems and to complete a group quiz. This activity was followed by a second quiz to be completed independently. Sixty in-class assessments were completed during the semester, each designed and graded by Ghadiri: a time investment of 80 hours per week (Kolowich, 2013d; K. Ghadiri, personal communication, March 10, 2014). Cheal, Associate Vice President and Senior Academic Technology Officer at SJSU, reported that while students initially balked at the workload (12 hours per week), they were eventually swayed by their own results.

They were probably doing a lot more work because, previously, if they did the problems for homework and they got stuck, they would just stop and wait for class and listen to the lecture and try to understand it that way. Now they were working harder, and they were a little resentful of that at the beginning. But when they saw how well they did on the mid-term they became a lot more enthusiastic. (Cheal, San José State University). Ninety-one percent of the Fall 2012 EE 098-MIT students passed the course, compared with 59% of students in the traditional course offered the previous year. Ghadiri et al. (2013) compared the midterm and final exam results for EE 098-MIT with the results from the previous year’s traditional sections of Introduction to Circuit Analysis and found that scores were 10 to 12 percentage points higher in the re-designed section and that the “lower tail of the curve”(p. 8) had disappeared. Comparing the three Fall 2012 sections, they report a mean score on the final exam for EE 098-MIT of 62% (n=78), compared with 50% (n=50) and 45% (n= 75) in the two concurrent, traditional sections. The finding that lower performers appeared to benefit from the new format is consistent with observations made by Cima in his re-designed Introduction to Solid-state Chemistry course at MIT (see Case 6). Ghadiri et al. reported no significant differences among the three Fall 2012 sections of Introduction to Circuit Analysis in Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

103

students’ starting GPA, suggesting that, on average, students in each section began with similar academic abilities. At the end of EE 098-MIT, Ghadiri solicited feedback from the students about their course experience and asked for suggestions as to how the re-designed course could be further adjusted for Spring 2013. While over half the class indicated that access to resources online was what they liked most about the new class format, 71% felt that the MOOC material did not correspond well to the SJSU class material, and almost half claimed that the online lectures were not helpful, were long, or were difficult. One third of the students suggested replacing the online homework problems with sample problems solved stepby-step. Ghadiri continued to makes changes to the course and, in Fall 2013, one section of EE 098-MIT incorporated hands-on experiments using National Instruments myDAQ hardware and software “…that turned student laptops to measurement instruments. Students were able to actually build circuits of real components and perform experimentations with myDAQ.” (K. Ghadiri, personal communication, March 10, 2014).

Photo courtesy of Khosrow Ghadiri

For Spring 2014, Ghadiri describes further changes to EE 098-MIT: Currently…SJSU is developing online tutorials, quizzes, midterms, and final exams that can be taken on any mobile device such as smart phones, tablets, laptops, and so on. The delivery mechanism custom tailors to each student. Students receive material from the topics that need improvement based on their past performance. Questions are assigned randomly out of question pools with several variables so that the probability that two students have the same question is nearly zero. SJSU is developing a security and student identity verification system to protect the integrity of online classes. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

104

Overall, while the demand on both the instructor’s and the students’ time has been high, SJSU expects that by reducing the number of students who need to repeat Introduction to Circuit Analysis, the financial burden on both students and taxpayers will be lower. As the course has been modified each semester, it has become increasingly mastery-based and the passing rate in one semester was as high as 98%. Ghadiri reports that the re-designed section consistently fills to the 90-seat capacity before other sections are filled and that, one semester, he offered two sections, both of which filled to capacity. The California State University Chancellor’s office has negotiated a license with edX to use MIT’s Circuits and Electronics MOOC across multiple California State University campuses.

Sources: Professor Khosrow Ghadiri, SJSU; Catheryn Cheal, Associate Vice President and Senior Academic Technology Officer, SJSU. Cheal, C. (2012, August 14). Creating MOOCs for College Credit (Research Bulletin). Louisville, CO: EDUCAUSE Center for Applied Research. Retrieved from http://www.educause.edu/ecar Ghadiri, K., Qayoumi, M.H., Junn, E., Hsu, P., & Sujitparapitaya, S. (2013). The transformative potential of blended learning using MIT edX’s 6.002x online MOOC content combined with student team-based learning in class. JUCE (Japanese Universities Association for Computer Education) Journal 2013, No.3. Kolowich, S. (2013d, November 25). A Truce on the Tech Front at San José State. The Chronicle of Higher Education. Retrieved from: http://chronicle.com/article/A-Truce-Over-Technology/143229/

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

105

Goal 5: Innovation in Teaching and Learning Thirty-eight percent of the institutions in our study that were engaging with MOOCs, and nineteen percent of our interviewees overall, presented MOOCs as vehicles for experimenting and innovating with pedagogy and new models of higher education. Innovation as an end goal was mentioned by representatives of seven public universities, three private universities, and one community college. We note, however, that the other five goals we report involve innovating as a means to a particular end. A number of interviewees portrayed experimentation with MOOCs and online learning more generally as preparation for an uncertain future. According to Bacow, President Emeritus of Tufts University, “Some institutions are buying an option on the future. If somebody tells you that they know where this is going, they are wrong.” Harper at the University of Oklahoma exemplifies this position: Goal number two is, as an institution, to be forward thinking and adaptable and flexible in a really quickly changing landscape that nobody knows where it's going to be in two or three years. (Harper, University of Oklahoma). Some hold no doubt about the need for innovation in education and that technology will play a key role. A senior administrator at an East Coast University noted that the University President “holds the belief that residential education is going to change and thinks we can use experiences like MOOCs to change the residential experience.” Lester at FutureLearn asserts that in order to beat disruption, institutions of higher education must get on top of it and can use MOOCs to learn new capabilities and build their capacity for moving in new directions: "A digital generation is coming through the pipeline. Literally from the age of one kids are using and can interact with technology. Universities have got to be ready." Institutions do not want to be left behind in what they perceive could be a game-changing phenomenon for higher education: We like to be cutting edge and we don’t want to see this pass us by and then we are a year behind and have to catch up. So we want to make sure we are in the MOOC game and experimenting and can see where it goes next, like if it does go down the credit road. (Loble, formerly at University of California, Irvine Extension). The other goal is somewhat practical: remaining competitive and remaining viable. We don't want to position ourselves where what we are doing is a product in a delivery system that people do not want anymore. If MOOCs become popular, effective, and a preferred method of delivery, we want to be in the game. (Sechrist, Oklahoma City Community College). Hathaway at the University of Oklahoma observed that, especially for large state universities, change cannot be implemented swiftly so if institutions do not at least experiment with innovative ideas as they arise, they will always lag behind smaller, more nimble competitors. Frankly, it's quite time-consuming to develop dynamic online content. It's not something that you can just say, "Tomorrow, we're going to have it across the broad spectrum that is the University of Oklahoma," or any other university. Other universities are cautioned if they're not getting into this phase that, at some point after this experimentation phase, if it proves to be useful to go entirely online, you won't be able to just decide one day, "Hey, we're going online" …. Particularly for very broad, comprehensive universities like ourselves, seeing and wanting to be a part of the possibilities of the future, you've got to start now. (Hathaway, University of Oklahoma). Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

106

According to Guthrie at ITHAKA, MOOCs have, in the best cases, provided an opportunity for administrators and faculty members to work together to experiment with possibilities. As the interest in MOOCs and online learning has grown across campuses, several universities have established offices of digital learning and have increased their internal capacity for production support. For example, over the last two years Stanford University administrators have been engaging faculty in innovation and experimentation with MOOCs by providing $25,000 seed grants. Faculty members have been encouraged to experiment with a variety of platforms. Similarly, at Cuyahoga Community College (Tri-C), the administration pursues a deliberate college-wide strategy to infuse innovative pedagogies and technology throughout the curriculum in ways that engage not only their students, but also the faculty. According to Miles at Tri-C, the recent creation and delivery of a Tri-C MOOC has contributed successfully to this strategy: The enthusiasm and the new ideas that have emerged as a result of this initial core team working on this MOOC experience has had a multiplier effect in terms of charging other faculty or engaging them in new ways.(Miles, Tri-C). In other cases, individual faculty members may strive to be at the cutting edge of education and routinely experiment with the latest new technologies. Yet others are engaging in MOOCs to help shape an innovation that may significantly affect their careers, in preference to sitting by passively. Some institutions have specific goals for their innovative efforts in the MOOCspace and note that collaboration with partners in this endeavor is key: Many of us seek to use MOOCs as an innovation engine where we can experiment with new kinds of teaching techniques and student support mechanisms. In collaboration with other institutions, the University of Texas System is seeking to develop the learning platform of the 21st century which will differ fundamentally from a traditional LMS. Working with Stanford, edX, and Google, we are helping to drive the development of this new platform. It is less expensive to work collaboratively in this consortium than it would be to do this ourselves. And when you have Stanford, MIT, and Google involved, you probably have the people who can pull this off. (Mintz, University of Texas System). Overall it is part of a big grand experiment because no one knows what MOOCs will look like in 2015-16....The goal is being part of that experiment in a purposeful way and engaging in those discussions. Not just within Cornell, but with institutions all over the world, not bounded by the edX consortium. (Dodds, Cornell University). What I like about MOOCs is it gets us involved with other state systems and people who are really thinking about the future of higher education in ways that I don't think people would otherwise. MOOCs are, from a leadership perspective, a ticket into some really interesting discussions that will help us set direction. (Keyek-Franssen, University of Colorado System). Schneider at Stanford Universityâ&#x20AC;&#x2122;s Lytics Lab described a trajectory of innovation in MOOCs starting with the initial infatuation with scale, proceeding to a search for a peer-mediated solution to the challenge of grading thousands of assignments, to the current recognition that participants may learn best when connected with other participants and finding ways to facilitate this through online chat-rooms and small group discussions: First it was, "Let's make things as big as possible." That was the driver behind many of these courses right from the get-go, and many of the technologies that were being developed to Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

107

support them, particularly around assessment technologies â&#x20AC;&#x201C; both the automatic machine grading tools, and also the advent of peer-review and peer assessments. Turning the grading over to the students, because you're trying to service many people at scale, has meant turning down the authority that lies in the hands of the instructor and the TAs, and turning up the authority that lies in the hands of the other students. That has been somewhat innovative. Chinmay Kulkarni, a doctoral student of Scott Klemmerâ&#x20AC;&#x2122;s, who is the HCI professor, is now working on scaling down again, within this massive setting. Creating little chat rooms or other kinds of smaller group-oriented experiences, where the massiveness has this very useful aspect to it, but can also lead to a sense of anonymity, or feeling lost. If you then zoom in again, and have opportunities for smaller groups to interact with each other, you may be getting strength in those ties, and strengthen the whole network and the course overall. (Schneider, Stanford University). Ironically, while the shift in balance of authority from instructors to students was celebrated as a key innovation by the xMOOC players, the idea encountered resistance when the first cMOOCs were offered in Canada. George Siemens recounted that he and Stephen Downes, as co-instructors of the first MOOC, were initially assailed for diminishing the authority of faculty members in favor of the MOOC participants. MOOCs were also presented as a disruptive innovation within the higher education business model that could help universities become more competitive simply by forcing them to reconsider the status quo and to ask fundamental questions about commonly accepted practices in higher education. Dodds at Cornell University asserts that, at the very least, MOOCs have pushed college personnel to open up to the research about how people learn. Not all our interviewees were as convinced that MOOCs would introduce more productive innovation than existing experiments with online and blended learning. Interviewees with experience in online education and open learning resources over the last two decades pointed out that many MOOCs are reinventing and relearning the missteps and successes of online learning and failing to take advantage of scale, the most characteristic aspect of MOOCs that differentiates them from other online education. We are slightly less persuaded by the notion that there is all sorts of educational innovation to do here because while we strongly believe that innovation is important in teaching, lots of our staff do online elements in their teaching already. So we are not sure what new ideas MOOCs will bring to our students here. (Senior Administrator, Leading UK University). I am very pro-innovation, but I'm also pro-take-a-look-at-the-research-before-you-innovate.... There are a lot of us in the online learning field who have been basically caught unaware that this would catch on so big and, at the same time, they'd be able to completely ignore about 15 years of research in there. (Vignare, University of Maryland University College). It is really just perpetuating the traditional model of teacher-centric learning. We have a quiz after 7 minutes and you just listen to someone else lecture, then we have another quiz and then we go to the discussion board. How is that innovative? We might as well go to the tele-courses of nineties....It is not innovative in any way unless you start applying the analytics, unless you start leveraging technology. There are so many advantages of teaching online and they are not even using any of them. Like the social networking aspect, crowd-sourcing, project-based learning. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

108

You have 20,000 people logged in to the same place. Seems like such a waste. (J. Baker, Foothill College). The value of innovation as an end in itself for educational institutions is hard to measure. As a means to achieving other goals it becomes more tangible and we address the evidence of success in achieving these in other sections of this report. However, it is abundantly clear that many institutions and faculty members have engaged in new educational activities since MOOCs first appeared and that strategies employed online such as more frequent assessments and shorter lectures interspersed with questions are being taken back on-campus. It is less clear what has been gained by these new initiatives. For the most part, institutions are not making any rigorous attempt to assess whether MOOCs are more or less effective than traditional on-campus or online courses in terms of improving learning outcomes, and few have designed metrics to assess the impact of MOOC-related innovations on other objectives.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

109

Case 8: Maximizing the Use of Technology in Learning: University of Oklahoma At the University of Oklahoma (OU), open online courses are being developed in collaboration with NextThought with a goal of maximizing the power of technology and the Internet. OU courses, offered via the Janux Interactive Learning Community, may be taken MOOC-style for free without credit, or for credit with normal tuition fees paid. According to Kyle Harper, Senior Vice Provost, “the headline of this platform is social learning.” Several strategies are being pursued to enhance engagement and learning outcomes which Nick Hathaway, Vice President for Administration and Finance, asserts go beyond “trying to shove the classroom into the Internet:” • • • •

Encouraging social learning through allowing participants to interact with one another by embedding comments and discussions within the actual content, and controlling with whom comments are shared; Game-based learning, applying strategies to improve learning outcomes derived from oncampus research; Frequent use of auto-graded self-assessments, the results of which are only available to the learner, in addition to assessments that count towards a grade; Archiving courses to allow students to indefinitely re-visit online lectures from a completed course, read their own notes or comments, and re-connect with other participants.

Harper, who taught one of the university’s first Janux courses, Law and Justice, was most excited by the increased levels of student engagement and participation that he observed as a result of the social learning features facilitated by the NextThought platform: It's not like we have some really clunky discussion forum under "forums." We do have forums and there are some great debates going on in those, but the social learning is deeply embedded in the content. All of the text that I'm using is totally open access text, so we're able to load the full text into the platform. Like, Aristotle, who's out of copyright, and the Nicomachean Ethics. I can go in there and I can embed, deep in the text, "This is a really important line," or "What does he mean?" I can start a discussion. As with other social network systems, there is an activity stream that can notify a participant, for example, when another participant adds a comment to his or her note, or engages in a different course activity. A student dashboard advertises the current popular discussions within the course and what activities are coming up next. The dashboard can be used to indicate who has contributed the most posts, drawn the most comments, or garnered the most “likes” for their contributions. It is personalized to include each participant’s chosen contacts. Course videos are accompanied by accurate transcripts which are synchronized such that if the participant clicks a spot in the transcript, the video will skip to the same place and vice versa. Comments are embedded within the transcript so that a viewer can see relevant notes and discussions from prior viewers as the video progresses, or enter comments and questions of their own. All artifacts in the course are searchable: video transcript, books, forums, blogs, chats, notes created by the searcher and any shared with him or her. While the default setting on the platform shares each participant’s comments with the whole class, each participant can control the setting for every comment and note he or she makes. For example, it can be Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

110

a private note to self as in a digital notebook, a comment to a study group, a publicly shared contribution to a growing discussion on a particular point, or a direct question to the instructor. The platform can also differentiate the experience for the MOOC participants vs. the fee-paying participants. For example, at the instructor’s behest, the fee-paying students can have access to additional e-book materials, materials offered under "fair use" that require OU student ID's, special discussion forums, and instructor-graded assignments. To further encourage engagement and support interaction, NextThought has developed mobile access to Janux courses. Currently, participants can interact with content on an iPad using a browser or by downloading a Janux iPad application. Going forward, participants will be able to receive notifications and interact with the course content on additional mobile devices including Android tablets and smartphones. Ken Parker, CEO of NextThought, the company that has built the platform used by Janux, espouses an educational philosophy that may well serve to bridge the chasm between xMOOCs and cMOOCs to the benefit of learners: We want to make it easy and effective for people to connect in an online education experience. The Internet transforms by distributing information and connecting people. Current online education has primarily focused on the former. Transformation is more likely when we do both. The trick is to figure out how to make those connections actually achieve learning objectives so that it is not just party time or “here is my latest cat picture.” Along with increased engagement, we want to help develop higher-order skills such as communication, dialogue and teamwork. These are relational activities. It is not enough to simply on-board or import content because all you get is the same old stuff in a slightly nicer interface. We must rethink online education in a connected world.

Sources: Nick Hathaway, Vice President for Administration and Finance, University of Oklahoma; Kyle Harper, Senior Vice Provost, University of Oklahoma; Ken Parker, CEO, NextThought.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

111

Goal 6: Research on Teaching and Learning Research on teaching and learning was stated as a goal for MOOC initiatives by 28% of the 29 institutions offering or using MOOCs and by 18% of our 83 interviewees. This was a stated goal for one community college, three public research universities, and four private research universities. There is considerable overlap between research per se and less formalized efforts to improve teaching and learning which many more interviewees alluded to and we reported under Goal 4: Improving Educational Outcomes. While often a fine line of distinction, we generally include here work that is being conducted by individuals who consider themselves researchers first and foremost, and aim to publish their work, while efforts described under Goal 4 were less about creating generalizable knowledge and more about improving outcomes in a specific situation. In later sections, we provide more detail on Examples of Research being Conducted with MOOCs, Types of Data Available from MOOCs, and on Where are we with Data Mining, Learning Analytics, and MOOCs? It's all about the creation of new knowledge in a different modality. We can speed up the research process because we're going to crowd-source knowledge creation. (Keyek-Franssen, University of Colorado System). Originally we did the MOOC because we wanted to collect data for educational research. I want to have students who aren't saying "Look. I don't want to be a guinea pig. I'm paying $40,000 for this education and you're supposed to know what you're doing and be a good educator." (Pritchard, MIT). We’d like to help create a collaborative research infrastructure for what we are calling the new data science of learning. We can make an important contribution to that because we already have a strong integration of computer science into the learning sciences; the Lytics Lab is a great example of that, we have computer science students working with education students working with communications students all on projects related to this. We are in the process of trying to facilitate a conversation between programmers and researchers but we still need to get a handle on all the data. (Collier, Stanford University). We heard from our interviewees about several types of research being conducted using MOOCs as the vehicle for delivering an intervention, or as a source of data on participant behavior and performance. The researchers themselves are in a range of disciplines which influence the kinds of questions being asked. Computer scientists and engineers have tended to focus either on descriptive studies, summarizing demographics or how much each educational resource (video, e-text, discussion forum etc.) is used, or on design work creating badging systems, discussion opportunities, peer grading, and auto-grading applications. Sociologists and education or “learning science” researchers have more often focused on experimental research, for example conducting A/B tests by e-mailing different messages or materials to different participants selected at random. A great deal of effort is being expended on trying to improve participant engagement and completion of the courses and less effort on determining whether participants actually gain skills or knowledge from a MOOC. And, beyond improving strategies within a MOOC or applying some of these strategies to oncampus and regular online teaching, there has been little research comparing the efficacy of MOOCs to regular online or face-to-face courses with respect to learning outcomes. Guthrie at ITHAKA observed that this question of efficacy is rarely asked with respect to regular college courses, and doubted that there would be any more motivation for such accountability with regard to MOOCs. For now, most Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

112

instructors and researchers appear content with equating the completion of a MOOC with having learned something. But Sarma at MIT asserts that measuring effectiveness will be the next step. “Now we think the car works, so next we’re going to measure the engine efficiency.” How “efficiency” will be defined is yet to be determined. There are various evaluators and research labs in the U.S. and abroad that are assessing the impact of MOOCs or A/B testing is commonly used for market research in online focusing on MOOC data to conduct experiments. Box 6 describes the study environments: viewers of online content are presented conducted by the Research and Planning different versions of the same content at random. For Group for California Community Colleges example, online retailers test varying shades of color or (RP Group) on the San José State different ways of presenting goods to determine which University SJSU+ Augmented Online design optimizes user experience and maximizes desirable Learning Environment Pilot Project. Case transactions: usually the purchase of goods or a service. 9 describes the work of David Pritchard’s RELATE group at MIT. Other groups that have been active in MOOC research include the Lytics Lab at Stanford, and the MOOCLab at University of California, Berkeley. The Lytics Lab is an informal network of interdisciplinary researchers, including faculty members, doctoral, and postdoctoral students who are interested in online learning in general, and in working with MOOC data specifically. Disciplines represented include computer science, education, communications, statistics, and sociology. Box 5: A/B Testing Online

Reflecting the growing interest in using MOOCs as a vehicle for conducting research, several research conferences in 2013 included day-long strands on MOOCs, including the MIT LINC conference in June, and the moocshop preceding the Artificial Intelligence in Education conference in August. A December 2013 conference sponsored by the Bill & Melinda Gates Foundation, the MOOC Research Initiative, featured the work of around 30 groups engaged in MOOC research; the 2014 European MOOCs Stakeholders Summit included presentations by more than 20 other groups; and the first Learning@Scale conference in March 2014 showcased over fifty studies or works in progress. Areas in which interviewees described research explorations and advances included: •

The role of social media networks in teaching and learning

•

Testing pedagogical strategies, for example, o Firmin, et al. (2013) o Champaign et al. (2014)

•

Student engagement and motivation, for example, o Coetzee, Fox, Hearst, and Hartmann (2014) o Kizilcec, Piech and Schneider (2013) o Kizilcec, Schneider, Cohen, and McFarland (2014)

•

Machine learning/modeling research, for example, o Piech et al. (2013) Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

113

o o

Halawa, Greene, and Mitchell (2014) Nguyen, Piech, Huang, and Guibas (2014)

•

Natural language processing

•

Human-computer interaction, for example, o Kulkarni et al. (2013) o Cambre, Kulkarni, Bernstein, and Klemmer (2014)

•

Personalized/adaptive learning, for example, o Buffardi and Edwards (2014)

•

Comparing hybrid with traditional courses, for example, o Ithaka S+R (2013) o Griffiths (2013) o Bruff, Fisher, McEwen, and Smith (2013)

•

Developing data standards and a common platform for data mining, for example, o Veeramachaneni, Dernoncourt, Taylor, Pardos, and O'Reilly (2013) o Dernoncourt et al. (2013)

Each topic area is discussed in greater detail in the Sections Examples of Research being Conducted with MOOCs, and Where are we with Data Mining, Learning Analytics, and MOOCs. Fisher at Vanderbilt University, Parker at NextThought, and others note, however, that the top priority is to deliver high quality content online in a reliable manner and that research considerations must be secondary. Yet others remain skeptical that MOOC-based research can offer insights that have not already been discovered, or that findings from MOOCs can be generalized to other populations and contexts. There is no need to waste time conducting research on the pedagogy of MOOCs. Existing research on online learning has a lot to offer and has already established best practices. Research on MOOCs will provide the same insights. (Daniel). I cannot believe we are going to spend bunch of time and money rediscovering things we already know. Why don’t we start with what we know, or what we think we know, design the environment based on what we think we know and use the environments to refine what we know; let’s integrate research and practice. The environments are not being designed to collect data in the way that will help us make progress on the big questions about either course design or learning. (Thille, Stanford University). The data being gathered are really nothing more than information about these students’ behaviors. They're clicking here, they're clicking there. They're doing this assignment at the eleventh hour…but beyond that, it's not telling us what they're learning. It's back to the old behavioral scientist kind of stuff. What are we able to observe? And beyond that we can't assume anything else. Until we have better assessments on these things, we're not really going to be able to draw many conclusions from these data. (Bishop, University System of Maryland). Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

114

If we were in a consortium with universities that we would not regard as being highly selective, how interested are we in the data that comes from the analytics that involve them and their potential learners? We are not necessarily. How much is that data going to tell us anything that is relevant to what we do? We haven’t been convinced yet that we are going to learn anything about learner behavior that we need to know and don’t know at the moment. We are extremely dubious about the expectation that there will be lots to learn from the big data that will improve pedagogy both online and on-campus. (Senior Administrator, Leading UK University). Jiffy Lube knows more about your car than how much we know about the effect of this technology on actual [on-campus] students. They're not the ones who are taking these courses. (Fox, University of California, Berkeley).

Box 6: San José State University SJSU+ Augmented Online Learning Environment Pilot Project One of the most useful applications to date of MOOC platform data to investigate the impact of online learning in MOOC-like settings is Firmin, Schiorring, Whitmer, Willett, and Sujitparapitaya’s (2013) study of the San José State University’s (2013) Preliminary Summary SJSU+ Augmented Online Learning Environment [AOLE] Pilot Project. This study reviews data from an experiment conducted by San José State University (SJSU) in partnership with Udacity. In Spring semester 2013 SJSU offered three AOLE courses: remedial algebra (Math 6L), introduction to college level algebra (Math 8), and introduction to college level statistics (Stat 95). The courses were offered for a $150 fee to a limited number of students. In addition to the typical content and activities found in MOOCs, students could obtain online help from Udacity Online Support Providers. Participating students were at-risk students, both matriculated and non-matriculated, including youth from a nearby high school and students who had previously failed a face-to-face version of the remedial algebra course. While prior research (e.g., Figlio, Rush, & Yin, 2013; Xu & Jaggars, 2013) indicates that at-risk students are the least likely to thrive in online situations, Catheryn Cheal, Associate Vice President and Senior Academic Technology Officer at SJSU, explained to us that “…these were the students that we wanted to help the most by the lower cost for a MOOC section. It would be a second chance, not normally allowed at SJSU, to pass a previously failed remedial math course.” Both retention and pass rates overall in the AOLE classes were worse than for similar students in equivalent face-to-face courses. However, the study found a number of significant relationships between variables that might be expected to impact success. Student effort, most notably reflected by the number of problem sets submitted by a student, was more important in predicting success in the course than demographics or use of student support services. Firmin et al.’s (2013) evaluation report provides helpful insights as to what can and cannot be done currently with the MOOC platform data and provides a baseline on MOOC data research from which to build towards more complex research designs. It also clearly indicated that online educational experiences for at-risk students require careful structuring and substantial student support.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

115

Obstacles to using MOOCs for Research on Teaching and Learning Dillenbourg, Fox, Kirchner, Mitchell, and Wirsing (Eds., 2014) provide a detailed discussion of the complexities surrounding the use and sharing of the learner data that is available from MOOC platforms. Williams at Stanford University’s Lytics Lab identified a number of challenges to realizing the goal of using MOOCs for conducting research on teaching and learning, highlighted in Box 7 below. A complication expressed by many researchers is that the Institutional Review Boards (IRBs) at different institutions interpret and enforce federal human subject research regulations differently. Similarly, each institution renders its own interpretation of how to apply FERPA to MOOC-related data. This makes it hard to take advantage of the enormous amounts of data potentially available for research purposes. Box 8 summarizes the position of Teachers College IRB with respect to MOOC-derived data. Two recent “data jams” organized by the White House Office of Science and Technology have facilitated discussions at edX and at Stanford on the topic of data standards and sharing, but resolution is not yet in sight. Box 7: Challenges to Conducting Research on Teaching and Learning Using MOOCs Joseph Jay Williams at Stanford’s Lytics Lab identified a number of challenges to realizing the goal of using MOOCs for conducting research on teaching and learning:

“1. The biggest obstacle in all of this is that the researchers don't make the decisions about what gets

taught and how, in a MOOC. Faculty members make the decisions. Udacity owns their MOOC content so maybe they can push faculty into having X or Y. But it's not so common. 2. One big challenge in this kind of research is that, although online environments are fantastically suited to doing randomized, controlled trials, it's really pushing people in terms of the kind of statistics they need to know, and the kind of modeling they have to do, because issues come up with respect to people deciding not to finish your survey, or answering questions randomly. Or you have multiple variables being related. The same variable you're manipulating might be an influence. There is survivorship bias….What this has meant is that a lot of effects are hard to measure. What we really need in these MOOC platforms is more control of the environment. For example, if the surveys are optional, it’s hard to get conclusive results. 3. There's no A/B testing framework for content like videos and exercises, or the way something shows up. There's none of that in the Coursera or edX platforms as yet. Coursera has introduced A/B testing for e-mails. 4. A big issue is that now we've got all these data, but everyone's collecting data and storing them in different ways. Like edX data, Coursera data, Khan Academy data. Everything is in a state where it's hard for people to put things together. 5. Cross-institutional collaborations are extremely complex, and IRBs don't talk well across universities…. We need to think very carefully about the regulatory environment because if the government makes it easy for people to share data and conduct experiments, it's going to really move forward research on MOOCs. If the government puts a lot of things on lockdown and makes it hard to get, for example, cross-institutional IRBs, that's going to put a major damper on things like setting up data stream collaboratives or making anything open source.” (Williams, Stanford University). Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

116

Issues to consider for future MOOC research Several seasoned researchers in the field of educational technology: Barbara Means at the Center for Technology in Learning, SRI International; Jessica Heppen, Managing Researcher at American Institutes for Research (AIR); and Dede at Harvard suggested that, going forward, it would be useful to track course sequences followed by MOOC participants. Both the interest and ability to engage in a higherlevel course on the same topic could, at least for those participants who do not start with expert knowledge, reflect the effectiveness of the initial MOOC. Means at SRI suggested that other longitudinal outcomes to follow might, where relevant, include performance on an industry-wide certification exam. She cautioned, however, that in online courses the students who drop out tend to be the weakest while this is not necessarily the case for face-to-face courses. This creates a problem for comparing the treatment-on-the-treated outcomes of the two modalities as there will be a greater likelihood that the survivors in the online situation are stronger than those in a comparison face-to-face intervention. Heppen noted the importance of only comparing interventions where the performance incentives are similar, suggesting that it would not be best practice to compare outcomes from a free, non-creditbearing course with those from a course that costs several thousand dollars in tuition but contributes towards a degree. Researchers also raised concerns about attaching high stakes (i.e., fees and credits) to experimental interventions with no established record of success. Early in 2013, the Computing Research Association sponsored a workshop on New Technology-Based Models for Postsecondary Learning: Conceptual Frameworks and Research Agendas. A group of 14 experts in educational technology from academia and industry, led by Dede from Harvard University and Eric Grimson from MIT, gathered at MIT to conceptualize the future role of technology in higher education, and to lay out an extensive research agenda. Many of the issues raised in this report as priorities for research apply to MOOCs and should be heeded by aspiring researchers in this field (see Dede, Ed., 2013). A second workshop sponsored by the Computing Research Association at around the same time, Multidisciplinary Research for Online Education, was attended by 94 computer scientists and learning scientists. The resulting report, (Fisher and Fox, 2014), similarly lights many paths forward for future research on MOOCs and other massive open online education, or â&#x20AC;&#x153;MOOE,â&#x20AC;? including its potential impact on workforce development and on enduring interdisciplinary research collaborations. Coming from a different perspective, Reynolds from Technology Crossover Ventures questions what the appropriate measure of success should be for MOOCs and other online experiences, and raises some alternatives to consider: It is a hard thing to prove the quality of an education. What is the measure of success, is it proficiency in the subject? Is it the ability to get a job, or is it the ability to get a job of a certain salary? I think for most people who are attending the online schools they are doing it to drive towards some higher station in life. So is that the success vs. actually learning whatever it is you are learning in terms of the curriculum? (Reynolds, Technology Crossover Ventures). While conducting research on teaching and learning has not been a priority for MOOC initiatives at many institutions, there are plenty of researchers eager to take advantage of the opportunities MOOCs provide to experiment with a variety of embedded interventions and to work with large data samples. While we documented a few examples of such research, it is clear that progress in using MOOCs to improve teaching and learning is being impeded by difficulty using the data, which we elaborate on in the next two sections, and lack of clarity regarding regulations applicable to the participants and their data. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

117

Box 8: IRB Treatment of MOOC Research: What Does and Does Not Require IRB Review? According to Teachers College Institutional Review Board (IRB), course-related activities presented in a MOOC that result in data that may eventually lead to research do not require IRB approval in advance of implementation, or informed consent from the MOOC participants. For example, if all participants are required as part of the course activities to complete the same pre- and post-tests of content knowledge or pre- and post-surveys (which might address issues including demographics, background experience, intentions with respect to course completion, motivation for participation in the course, satisfaction, etc.), IRB approval need not be sought in advance of the course. Any such course activities or assignments must, however, satisfy the requirements of the instructor’s curricular department. Before any analysis of such data is conducted with a purpose that extends beyond an internal assessment of the course for evaluative purposes and is to be presented as “generalizable research,” IRB approval must be sought and is likely to be granted under Exemption 4 provided that participants cannot be identified. Exemption 4: Research involving the collection or study of existing data, documents or records. Sources must either be publicly available or information must be recorded by the investigator in such a manner that subjects cannot be identified, directly or through identifiers linked to subjects. At the other extreme, if some of the MOOC participants will be asked to participate in a research activity that is not directly related to the course, or if a third party wishes to collect or analyze data related to the MOOC for research purposes, it is clear cut that advance IRB approval would be necessary. Where the question of need for advance IRB approval becomes less clear-cut, is when different students are offered different activities. For example, if only a subset of the MOOC participants are offered participation in online chat-rooms or face-to-face study groups to collaborate on course-related issues, or if some, but not all, students are offered extra problem sets or different video and text materials, this may be considered more than a purely curricular decision. While such strategies may in some cases constitute routine pedagogical approaches, if the variations are planned with the intent to conduct research, the Teachers College IRB office advises researchers to check with their own IRB offices whether an advance review is necessary. Source: Teachers College IRB Office.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

118

Case 9: RELATE at MIT David Pritchard’s RELATE (Research in Learning and Assessing and Tutoring Effectively) education group at MIT has been analyzing data from Pritchard’s Mechanics ReView MOOC to capture the impact of this learning experience on educational outcomes and to improve successive iterations of the course, with the aim of building participants’ expertise in problem-solving. The course is designed using a structured pedagogical strategy, “Modeling Applied to Problem Solving,” developed by Pawl, Barrantes, and Pritchard (2009). Mechanics ReView was originally created at MIT in 2009 as a three-week “flipped” review session on Newtonian mechanics for on-campus students. It was subsequently developed into a full, semester-long course with more online preparation. Since 2012, the resulting online course has been offered three times as a MOOC, first through the LON-CAPA platform and more recently via edX. From the beginning, the course was set up as an experiment to investigate how to improve students’ conceptual overview of mechanics, as well as their ability to use “expert-like” approaches to problem-solving. For RELATE, the appeal of offering a MOOC was the promise of large datasets: …we have made an accelerator for physics education research… it's like a particle accelerator in physics. You have a high intensity stream: in this case, students. You can put something in there for them to interact with. Then you have a whole host of data mining and machine learning and psychometric tools available...that's like the detector…to study what comes out of this. …if you want to do good research, you really need 1,000 people. Four hundred is not really enough. Once you split them up, you've only got 200 people per group, then you just don't have enough statistics to detect a fifth of a learning effect size. (Pritchard, MIT). A team of post-doctoral fellows with complementary skills in data mining, psychometrics, and education research has used Item Response Theory (Hambleton, Swaminathan, & Rogers, 1991) to measure the week-by-week skills of Mechanics ReView participants and to study the correlation between time spent on various activities with improvement in problem-solving skills. Champaign et al.’s (2014) paper “Correlating Skill and Improvement in 2 MOOCs with a Student’s Time on Task” presents a detailed description of the data analysis procedures and identifies a number of challenges arising in MOOC data analysis. The researchers also administered pre- and post-tests to measure absolute learning in the MOOC and have compared this with learning in traditional, on-campus, lecture-based courses. Recognizing that the populations online and on-campus are quite different, RELATE also measures the increase in skills for MOOC vs. on-campus participants. Using the participant activity data (“clickstream data”) that is available from edX, RELATE has been able to accurately operationalize concepts such as “time spent reading the book” (after allowing for instances where the student leaves his or her computer without logging out), and “looking at a book page” (which excludes flipping through a number of pages). This has allowed the team to identify patterns in the data such as markedly different allocation of time and use of various resources by MOOC participants when working on homework as opposed to exam problems. This seemingly straightforward analysis requires significant programming skills and investments of time, which few other institutions have been able to muster. Findings that have been used to modify the course over the successive iterations include the observation that frequent checkpoint questions and weekly quizzes increase the likelihood that Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

119

participants will read the course materials and complete the course. Additionally, a flexible schedule with all relevant materials available at least a month ahead of deadlines can boost retention rates. Online support, according to Pritchard, has evolved from none to “wiki 1, wiki 2, then an e-text, then an e-text with homework, then an e-text with homework with different graded levels.” Pritchard observed that extreme diversity of incoming knowledge and skill levels among MOOC participants presents an obstacle in attempts to isolate the impact of the MOOC, or of specific components, on skill improvement. For example, while one might expect that participants who made greater use of the course materials would show a higher skill level, in MIT’s 6.002x Circuits and Electronics MOOC that Pritchard also studied, the highest performers were actually the ones who spent least time on homework and class materials. In Mechanics ReView, 25% of the participants were physics teachers while another 25% had a high school degree or less. Many of the high performing participants started out already proficient in the course concepts and could do the homework and tests quickly and accurately without having to learn the material first. However, more time spent on the checkpoint questions (questions embedded in the e-text that tested conceptual knowledge) in Mechanics ReView did correlate with greater skill improvement (see Champaign et al., 2014). RELATE plans to use A/B testing, which they expect to be available on the edX platform by Summer 2014, in future iterations of Mechanics ReView. Possible tests will include varying the order of problems in assignments or using different instructional materials to continue fine-tuning the delivery of the course in order to maximize student performance. Ultimately, the goal is to build sufficient trajectories of how different types of students learn best to allow creation of a real-time adaptive learning environment in which the content, problems, and support presented to each student vary depending on how (s)he progresses through the course activities.

Sources: Dave Pritchard, Cecil and Ida Greene Professor of Physics, MIT Borkovitz, D. (2013, July 30). Summer MOOCs 2: Mechanics ReView. [Web log post]. Retrieved from: http://debraborkovitz.com/2013/07/summer-moocs-2-physics-review/ Champaign, J., Fredericks, C., Colvin, K., Seaton, D., Liu, A. & Pritchard, D. (2014, March). Correlating skill and improvement in 2 MOOCs with a student’s time on task. Paper presented at Learning@Scale Conference, Atlanta, GA. Retrieved from: http://dx.doi.org/10.1145/2556325.2566250 Carson, S. (2013, May 1). An MIT physics MOOC five years in the making. MIT News. Retrieved from: http://web.mit.edu/newsoffice/2013/mechanics-review-mitx-course.html Scarsonmsm. (2013, May 27). Mechanics ReView MOOC: Learning about digital learning through data [Web log post]. Retrieved from: http://mitopencourseware.wordpress.com/2013/05/17/mechanics-review-mooclearning-about-digital-learning-though-data/

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

120

Examples of Research being Conducted with MOOCs In this section we provide further details regarding each of the types of research being conducted with MOOCs, as listed in the section Goal 6: Research on Teaching and Learning. The role of social media networks in teaching and learning Siemens described a recently-begun 5-year project with Caroline Haythornthwaite of the University of British Columbia and Anatoliy Gruzd of Dalhousie University, funded by the Social Sciences and Humanities Research Council of Canada, to investigate the development of digital literacy and creativity in emerging technologies using participatory approaches. The overarching research question is “How do social media networks influence educational models?” One goal is to devise “theoretical and technical solutions that will help educators and researchers to study learning processes in social media” and another is “to provide educators and learners with tools that will help them to determine whether their particular use of social media is beneficial to their teaching or learning.” (Dalhousie University, June 4, 2013 announcement). Testing pedagogical strategies Several interviewees described MOOC initiatives as an opportunity to explore how to improve current teaching practice. Even if the hype for MOOCs dies out in five years, things would be different. The game has changed because of them. People are rediscovering how people actually learn: what you need to do in order for people to learn. These experiments were happily ignored for years. (Dodds, Cornell University). Research really is a supporting mechanism for improving professional development. Learning more about what works, what doesn't work. Looking into different design models and ways of implementation. (Kleiman, North Carolina State University). Kleiman’s group in particular is less concerned that participants complete the MOOCs they are offering and more concerned that the courses are designed to allow participants to get what they need from the experience, whether that involves completing only two specific units, or the entire sequence. Presurveys elicit details on the participants’ goals, and mid-course surveys inquire as to whether participants’ goals are being met. By tracking resource use and activity, Kleiman is developing an “informal taxonomy of ways different participants get through a course” and re-designing the MOOCs in an effort to meet the participants’ stated needs. Lester at FutureLearn similarly stresses that the point of making better learning experiences should be about meeting the objectives that the learner sets for him or herself, as opposed to those set by an external entity. He suggests that MOOC designers consider applying Argyris’ (1978) concept of single vs. double loop learning, whereby they do not simply try to continuously improve the learning experience as it stands, but question the entire approach and whether they are even teaching what the learner wishes to learn. One particular mechanism that several interviewees expressed an interest in using for the purposes of discovering what works best in teaching is A/B testing, but the technical capability for this is still fairly limited. Coursera allows for randomly selected MOOC participants to be e-mailed different messages, but cannot yet offer alternative versions of resources within a course. Pritchard from MIT reported that A/B testing was possible with an earlier version of the edX platform but that the feature was sidelined in Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

121

later versions. However, he and others are collaborating with edX to re-instate the feature by Spring or Summer 2014. There has been limited use of pre- and post-tests in MOOCs as a means to establish what knowledge and skills participants gain as a result of participation in the courses and how various pedagogical strategies impact these outcomes. Williams at Stanford University reports that Jane Manning, Director of Platforms, Office of the Vice Provost for Online Learning at Stanford, has been working with a few Stanford faculty members to pilot pre- and post-test items within MOOCs. Case 9 describes work conducted by Pritchard’s MIT-based RELATE group that includes the use of pre- and post-tests in a physics MOOC, Mechanics ReView, to assess improvement in problem-solving skills as a result of engagement in the MOOC activities. We look forward to publication of findings. In reality, the most useful data collected so far that can be practically applied towards improving the online learning experience, appears to have come from surveys of the participants during or after the MOOC. Firmin et al.’s (2013) study of the San José State University’s Augmented Online Learning Environment (AOLE) pilot (see Box 6) included three surveys of the students that surfaced substantial feedback, albeit from a relatively small number of students. For example, students suggested providing more up-front information about availability of support services and how to access them, consolidating all course activity on a single website, clarifying course requirements, and organizing forums by course and topic. These suggestions could be swiftly adopted for the next set of courses offered. Fox at University of California, Berkeley suggests that the availability of substantial amounts of data from MOOCs will prompt instructors into “being more scientific in what we do” with regards to improving courses and pedagogical strategies from one semester to the next. For example, it will become possible to apply item response theory to test the quality of the questions presented in assignments and exams. Fox also described an investigation being pursued at University of California, Berkeley with regard to the importance of a specific faculty member’s presence in a MOOC. They’re actually looking to do an experiment to tease apart how much of the magnetism of a MOOC is inherent in the materials versus how much of it relies on the presence of that specific instructor. If I take the exact same materials, but a different person facilitates the course, can I measure the effect that has on retention, and student performance, and so on? (Fox, University of California, Berkeley). Student engagement and motivation A number of experiments have been conducted to investigate how to motivate students in online environments, primarily to promote active participation and persistence. However, as Heppen at AIR points out, it is important, but difficult, to establish incoming or baseline motivation before trying to assess whether it is “malleable.” The MOOCLab at University of California, Berkeley investigated the impact of social reputation features on retention and performance in a software engineering MOOC, Software as a Service. The researchers replaced the standard edX forum with two variants, a basic forum and one in which participants could “upvote and downvote” (Coetzee et al., 2014, p.1178) comments, questions, or answers in the forum, earn a reputation score based on the usefulness of their posts, and eventually gain access to moderation features in the forum. Participants in the MOOC were assigned at random to either one forum or the other on their first visit to the forum, and could not participate in the other forum. Coetzee et al. found Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

122

that the social reputation features encouraged quick response times to forum posts and more responses per post but did not improve grades, retention, or participants’ reported sense of community. Taking advantage of learning analytics techniques, Kizilcec et al. (2013) investigated levels of engagement in MOOCs using participant activity data in three computer science MOOCs. They discover four different categories of learners in MOOCs, reflecting diminishing levels of engagement: completing, auditing, disengaging, and sampling learners. For example, auditing learners stay engaged throughout the majority of the course, but do not complete assessments. These differences in approach to MOOC participation underscore the variety of goals participants have for enrolling in the courses and present an opportunity to personalize the course experience to better meet these varying needs. Kizilcec et al. also show associations between multiple activity types, for example, that participants who are more active in discussion forums are also more likely to complete a MOOC. However, it is not clear that there is any causal relationship. Using the A/B testing feature offered by Coursera, Kizilcec et al. (2014) sent out different versions of emails to MOOC participants to assess the most effective strategies for encouraging engagement in the discussion forums. E-mail messages included neutral reminders to all participants to use the forums and, later in the course, more targeted messages to participants who had not used the forums. Targeted participants received one of three messages: one implying that forum participation yields collective benefits, one implying benefits to the individual, or a purely neutral reminder that the forum existed. The researchers concluded that the encouragements did not have the desired effect, and, to the contrary, the “collectivist” messages were associated with less participation in the forums than if no messages were sent at all. Rene Kizilcec, Daniel Greene, and Rachel Baker have been exploring “implementation intention” by assigning, at random, different versions of a pre-course survey to the participants in a MOOC. The survey includes a question about intentions, for example, “How many videos will you watch per week?” For some participants an additional question asks “What is your plan for making sure you meet this goal.” Impact on completion and performance is being investigated. To control for engagement, other participants are simply asked how many videos they think the average participant watches. Daniel Greene at Stanford’s Lytics Lab is investigating the impact of Carol Dweck’s (2008) concept of “growth mindset” on dropout and completion in a MOOC context by providing some participants with information on the idea that intelligence is not fixed and can be improved through effort, and asking them to reflect on a situation in which they worked hard and improved in what they were doing. Williams, also at Stanford’s Lytics Lab, has been investigating the impact of introducing pop-up messages to MOOC participants while they are engaged in problem-solving. The goal is to simulate a tutor asking questions about the student’s progress and strategy in order to prompt self-monitoring and to suggest study strategies. Miles at Tri-C reports using data from the college’s Pre-algebra CCC MOOC to understand student behaviors such as persistence and time on task, with a view to developing competency-based learning experiences for Tri-C students. Machine learning/modeling research At Stanford’s Lytics Lab, Sherif Halawa, working in conjunction with MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), has been building a model to predict dropping out in MOOCs Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

123

based on video-watching behavior (see Halawa et al., 2014). According to Williams, Halawa intends to add an experimental component in which students who may be at risk of dropping out will be sent an intervention message. Also at Stanford, Jonathan Huang, a post-doctoral fellow in computer science, and Chris Piech, a graduate student in computer science, are engaged in the project Code Webs to facilitate the categorization and automatic grading at scale of students' programming assignments (see Nguyen et al., 2014). Huang and colleagues are also investigating how to use machine-learning to improve peer grading, for example, how to direct assignments to students for grading, and how to weight the peer grade a student gives by using information on how reliable the grader is, or how consistent (s)he is with other graders. Piech et al. (2013) analyzed peer grading activity in two consecutive offerings of Scott Klemmer’s Human Computer Interaction MOOC and found, among other things, that “students are least biased when grading peers with similar score.” (p. 6, para. 8) Natural language processing A number of interviewees commented on the need to find ways to better manage and navigate MOOC discussion forums given how quickly they become difficult to follow due to the sheer volume of commentary. Fisher suggests that this will be an area of research interest for some Vanderbilt faculty members. We have strong natural language processing expertise. How do we use a natural language processor, for example, to grapple with complexities of discussion boards at a global level? Right now the equipment is there. The hardware is sufficient. Through Coursera we have discussion boards that have many thousands of users, many thousands of simultaneous entries. Response time is quite good. But for someone who enters the course three days after it starts, it is already impossible to navigate the discussions. How can we use natural language processing to reorganize those discussions over time? That would be area of research our Vanderbilt faculty would want to get into. (Fisher, Vanderbilt University). Human-computer interaction Interviewees at both Vanderbilt University and Stanford University described the use of MOOCs in current or planned work on human-computer interactions. For example, Chinmay Kulkarni at Stanford University, working with Scott Klemmer (now at University of California, San Diego) created a functionality that allows students to grade each other’s work based on a rubric, and has conducted analyses of what makes for good peer grading (see Kulkarni et al., 2013). Kulkarni recently launched talkabout, an application that plugs into Coursera allowing MOOC participants to engage in video discussion with a small group (five to fifteen) of peers through Google Hangouts (see Cambre et al., 2014). Kulkarni is also investigating what makes discussions work well, for example, the optimal gender composition of a group. Ragan at Penn State World Campus reported that educational psychologists at the university are investigating how to create a “graduated peer assessment mechanism, how to validate those people who are good assessors, and have them move up in status so that you can really tap into the expertise within your community.”

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

124

Personalized/adaptive learning Currently, the Holy Grail in education circles appears to be personalization, at scale, of learning through the adaptiveness of the learning experience. The scale requirement is most likely to be met through online delivery, and the massive amounts of data that are available from courses like MOOCs improve the possibility of creating such adaptive learning experiences. As a professor from an East Coast University points out, “These massive datasets might actually allow machine learning to uncover patterns that no human can actually see, precisely built on the model of Watson.” Dr. John Whitmer, Program Manager, Academic Technology & Analytics Office of the Chancellor, California State University, emphasizes that although the potential exists for platform-driven just-in-time feedback to students, as evident in applications such as Course Signals, Blackboard Analytics for Learn, and Blackboard Retention Center, the necessary algorithms have not yet been incorporated into standard learning management system applications or MOOC platforms. Currently, beyond the large publishers who can afford the development of adaptive learning systems, it appears that academic research and development of adaptive learning is limited to specific applications, primarily in the field of computer science. For example, at University of California, Berkeley’s MOOCLab, Armendariz, MacHardy, and Garcia are developing OCTAL (Online Course Tool for Adaptive Learning), “an adaptive exercise system that customizes the progression of question topics to each student” (see MOOCLab Publications page) for use in computer science courses. At Virginia Tech, Buffardi and Edwards have created CodeWorkout, “an online learning environment that offers drill-and-practice exercises with novel social and adaptive scaffolding” (Buffardi & Edwards, 2014, p. 724) to help computer science students practice programming. In order to provide adaptive learning that is more than superficial presentation of alternative problems or pathways, it must be possible to follow the trajectory of any one student through the course materials. Currently, the platform data from the major MOOC platform providers cannot present individual student trajectories in any easily usable manner. Comparing hybrid with traditional courses Numerous instructors have been experimenting informally with the incorporation of MOOCs or MOOC material into their face-to-face courses and have compared outcomes of the blended/hybrid versions of the course with those reported in the former traditional format. We document several of these instances under Goal 4: Improving Educational Outcomes. Griffiths at Ithaka S+R has been leading a comprehensive research study to investigate the impact of embedding customized MOOCs and other online learning experiences into existing courses at campuses across the University System of Maryland (Ithaka S+R, 2013; Griffiths, 2013). The goal of these initiatives is to improve educational outcomes for students and to lower costs. Of the 22 tests conducted in 2013, five were side-by-side comparisons in which a MOOC was embedded in one section of a face-to-face course. Outcomes are being compared with simultaneously-offered traditional sections. In addition to comparing completion rates in the courses, common assessments were used across the two sections to allow for a comparison of performance. While ideally instructors and students would have been assigned to sections at random, this was not practically feasible, a challenge often encountered by researchers in on-campus higher education settings (e.g., Bowen, Chingos, Lack, & Nygren, 2012; Figlio, Rush, & Yin, 2013). In addition to the side-by-side comparisons, ten case studies were conducted of courses in which the instructor selected one or more MOOCs for blending with their own course materials. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

125

While results of the tests are pending, several implementation challenges are noted in the interim report (see Ithaka S+R, 2013). Faculty members adopting MOOCs created by others usually wish to customize the MOOC to fit optimally within their existing curricula, often requiring that the various components are unbundled. While technically possible, creators of MOOCs are not always receptive to this request and sometimes withhold permission for their courses to be adapted in this fashion. Intellectual property rights issues remain unresolved, resulting in uncertainty regarding ownership of the adapted course and ongoing access to externally developed MOOC materials. Complications arise in attempting to license materials that include copyrighted content. MOOC platforms do not integrate easily into existing learning management systems, requiring instructors to choose one platform or other as the primary interface for their course, and information technology departments to establish links between the systems. At Vanderbilt University, Bruff et al.(2013) investigated students’ perceptions of a blended course in which Stanford University's machine-learning MOOC was “wrapped” with content provided by a Vanderbilt University instructor (Fisher) and delivered on-campus. Given student concerns about the alignment between the MOOC content and in-class components of the course, Bruff et al. suggest incorporating only parts of a MOOC, or parts of several MOOCs, to better integrate the online and inclass elements. Both Ithaka S+R (2013) and Bruff et al. suggest that MOOC creators consider building courses to allow for the possibility of deconstruction by adopters. Note: The item “Developing data standards and a common platform for data mining” is addressed in the section Where are we with Data Mining, Learning Analytics, and MOOCs?

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

126

Types of Data Available from MOOCs This section provides a detailed description of the data that is available from MOOC platforms and the subsequent section on Where are we with Data Mining, Learning Analytics, and MOOCs? addresses the potential for and obstacles constraining use of these data to improve teaching and learning. Interviewees described three main sources of data from MOOCs. One is the “clickstream” logs collected directly from the course platform through server registration of user activity: what Heppen at AIR described as the “data exhaust” produced by any type of learning management system (see Box 9). The raw clickstream data is in the form of log files which record each and every time any platform user accesses a resource from the server, for example, watches a video, stops a video, or enters quiz responses. According to Koh at Coursera, anonymized event logs are given to an appointed data coordinator at each partner institution who is responsible for appropriate Institutional Review Board (IRB) clearances for researchers to use the data: We have certain mechanisms in place to anonymize the data so we strip out the student name and e-mail address and instead use unique numeric identifiers so that researchers can do their research without having to get IRB waivers and don’t know any more than they need to know about the students’ private information. (Koh, Cousera). Programming scripts are required to pull out useful information such as the number of individuals submitting quiz responses, or length of time each video is watched. This is the data that is used for data mining and learning analytics. The report section on Where are we with Data Mining, Learning Analytics, and MOOCs describes in detail the challenges in using these data but, without exception, interviewees who had tried to engage with the platform data asserted that they are difficult to use. Right now, it is so hard to get the data into a way that makes sense so we can do the research that we need to do to look at effectiveness. (Collier, Stanford). Infrastructure and platform decisions are made with the user in mind, but they're not made with research in mind. They can create real complications in understanding, and being able to do the interesting work of tracking how people navigate the course. (Heppen, AIR). Rizvi at Pearson, who is familiar with many learning management systems, notes that these issues are not endemic to MOOCs, but arise with all educational technology products because the “products were led by technology as opposed to pedagogy, and were not designed to ask the right questions.” A second data source is the summary tables made available by the platform providers. At least one of the platform providers, Coursera, provides an administrator dashboard, essentially a bank of tables, summarizing some of the most commonly requested data points. This allows the instructor to see how many participants are active in the course, how many complete each quiz, how many attempts are made, how many answered a question correctly, and so on. These data are reported in the aggregate, not by individual student, and does not show relationships among variables, for example, whether participants who watch more videos perform better on quizzes. To conduct such investigations requires additional programming efforts to parse raw data. According to Koh at Coursera, the demand for the clickstream data is high and growing: in the first four months of 2014 Coursera exported data from 776 sessions. I think we receive an average of one request a day from schools for raw data exports for their own MOOCs, which we provide as part of our agreement with them. They are not usually asking Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

127

for data from other universities, although that is growing, but most people are studying their own classes. But we do have researchers who have been reaching out across institutions and I think that’s actually one of the most promising aspects of this, that if it’s on a common platform and a common data format then it’s easier to access everyone else’s data provided that people opt in and understand that they are sharing the data. We are slowly writing more and more scripts and adding them to the site but, in the meantime, if you want to do anything else with the raw data, and there is a lot of stuff that people want to do, then they can go ahead and do that, and then we encourage them to share the scripts with each other and make them public. Also, you can use programming languages to write your own assessments and then there are ways for the administrators to plug those back into our interface. They can do this without talking to us. (Koh, Coursera). Box 9: What are the Platform Data? “All the course specific data are in one set of databases, which are relational or SQL databases, which have a series of tables. This includes the user behavior within the course like the quiz data, forum posts, and so on. The demographic data are not linked to a course: we collect this on the platform level because one student might take multiple courses but we only need one set of demographic data. This is also a relational SQL database, that is, it has a set of tables. So we export both databases separately and we provide a way to link them up because they use the same common identifiers for each participant. A third database is the clickstream data, which is just basically one long event log or list that shows that, at this time, someone did this, and it just keeps getting longer throughout the course. You have to write scripts to parse that data.” Source: Pang Wei Koh, Coursera.

EdX also provides raw, but not anonymized, data to a data coordinator at each partner institution and offers detailed guidance on how to work with the data files (see edX Data Documentation). According to Petersen, the edX learning analytics team is currently working towards the creation of data dashboards, but the responsibility for writing scripts to parse the data rests with the consortium schools. Petersen notes that, being an open platform, edX has a repository on GitHub and consortium members are encouraged to share their scripts. Led by researchers at Stanford University and MIT, there are promising conversations with edX, Coursera, Udacity, and others towards an interoperability framework, called MOOCdb, for data to facilitate cross-platform analyses (see Veeramachaneni et al., 2013), but, according to Petersen, it will take some “pushing” from members of multiple platforms to catalyze this process.

Thirdly, most institutions offering MOOCs create their own pre- and post-surveys which are administered via the MOOC platform, and are usually optional for participants to complete. Pre-surveys investigate items such as participant demographics, pre-existing familiarity with the course content, other MOOCs completed or in process, how participants learned about the MOOC, reasons and goals for participation, and prior online learning experience. Fisher at Vanderbilt University pointed out that U.S.centric surveys, demographic questions in particular, may not be understood by cultures across the world. Post-surveys typically include questions to elicit feedback on the course design such as accessibility and usefulness of the various types of materials and activities, and how much time was spent on the course. Additionally, participants are often asked whether they achieved their goals, how satisfied they were with the experience, and how they might apply what they gained from the course going forward. Some institutions also include a mid-term survey to obtain feedback on the student Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

128

experience and course design. Sample surveys are available from the American Museum of Natural History and Penn State University in Appendix V. Because these surveys are usually optional, the response rate is typically low (25% - 40% according to a few of our interviewees) and not representative of the entire population. The post-surveys suffer from â&#x20AC;&#x153;survivorshipâ&#x20AC;? bias. One institution that made pre-surveys mandatory, Cuyahoga Community College, found that 59% of the course enrollees were willing to complete the survey in order to access the course materials. The completion rate for this MOOC was higher than for most MOOCs: 18% of actual participants, or 11% of initial enrollees versus the more typical 5% observed. Perhaps the most non-committal students were weeded out by the survey requirement.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

129

Where are we with Data Mining, Learning Analytics, and MOOCs? While there is no doubt that the MOOC platforms collect oceans of data, it is apparent that interpreting these data and querying them to answer useful research questions is a work in progress. As Newfield at the University of California, Santa Barbara succinctly states: “It's a strange field where the promises and the commercially-driven claims are way out in front of the actual data.” More neutrally, Petersen at edX observes: “...as much as we have been talking about learning analytics for years and researching big data, it is still something that is emerging. It is in its forming phase….” For all the volume of data, Petersen cautions that unless analysts start with a good educational research question, there may be little to be gained. Ryan Baker, President of the International Education Data Mining Society (IEDMS), steering committee member of the Society for Learning Analytics Research (SOLAR), and instructor for the MOOC Big Data in Education, similarly recounts that educational data mining and learning analytics is “an incredibly new field: there is no standard text, and everyone has to learn through apprenticeship.” He notes that while computer-assisted instruction has been around since the 1960s, and educational software since the 1980s, “decent log files” have only been available since around 2006. Baker is concerned that many computer scientists are attempting research using MOOC platform data without attending to what has already been learned in education research over decades, and in data mining and learning analytics over the past few years. Whitmer at California State University describes a major gap between promise and reality with respect to research using MOOC data. As a member of the research group studying the SJSU+ AOLE experiment with Udacity, funded by the National Science Foundation, (see Box 6) and also Mt. San Jacinto Community College’s Crafting an Effective Writer MOOC, funded by the Bill & Melinda Gates Foundation and delivered via Coursera, he has grappled with data from two of the major MOOC platform providers. Some of the problems are relatively easy to fix. For example, one platform did not initially store any data on the discussion forums, apparently believing them to be unimportant, while another did not record student views of static content pages, a problem when reading online text is an important component of the course. A more major hurdle Whitmer identifies, common to all online platforms, is that data logging systems were established to record the activity occurring on platform servers. They were not designed specifically to track how learners act, either individually or in aggregate. He alludes to the current data situation as being “almost chaotic when you get under the hood” and suggests that the MOOC platform providers are understating the gap and generally lack educational research expertise. Trying to use the platform data to answer research questions is currently, according to Whitmer, like “trying to build a spaceship at the atomic level.” While it is possible to tease out aggregate data on participant activities in MOOCs, more useful inquiries such as looking for patterns in learner behavior over time and over multiple courses, or relationships between activity and measures of achievement, are not routinely feasible. The MOOC platform providers are saying you can improve pedagogy and learn from research using MOOC data. The data may all be recorded, but getting something meaningful out of them is a very large, resource-intensive job, and most people don't have enough time or resources. The person I listen to the most is Daphne Koller from Coursera say how much hope there is for what we can learn from the data in MOOCs. Every time I hear her say that, I think, "What a great idea and vision. We are really far away from it.” The MOOC platform providers are leading people to believe this is possible, but they're not putting the time and resources in to make it happen. I don't think they're the right people to do it. They don't know the questions to ask. They don't know how we analyze data. They really don't know what rigorous statistical analysis is if you Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

130

look at any of the results of the stuff that they do. Like when they say what their population is, they don't even give a sample size. They don't say what percentage of people respond. I think to do this well, they need to look outside of their own shop and consult or partner with the educational research community, like the whole SOLAR community, Ryan [Baker]'s work, George [Siemens]'s work, Simon Buckingham's work. A lot of us have already done this stuff and understand the domains, both the online teaching and learning domain, as well as the research domain. There's just a lot of expertise that could really be brought to bear on this to make it go a lot further and a lot faster. (Whitmer, California State University). Fox from University of California, Berkeley described the difficulty involved in answering apparently straightforward questions with the data from the edX platform, and Pritchard from MIT provides a graphic account of the challenges in parsing the platform data for a particular MOOC: There's one data store that holds the registry of students. To the extent demographic information is available, it's available in that database, but that's separate from the one where marks are recorded for individual assignments. Then there's a separate database which essentially records what we can think of as the clickstream, the student interaction with the LMS. If you want to ask the question of looking at students from the Americas as a group, “Are they doing better or worse on these homeworks, and does it have any correlation with how much they're interacting with the segments of course material that cover those topics?” That's three different databases that you have to cross-correlate and join across just to do that relatively simple query. (Fox, University of California, Berkeley). We probably invested nine computer science post-doc months in dealing with the data from Anant Agarwal’s Circuits and Electronics MOOC. The data comes off a cloud, or 32 servers that have logs. I think we had 10,000 individual logs. When we put them all together, we had 110 gigabytes of data, which is like a small-town library. It's like 100,000 books.This is for the 7,000 people who got certificates. It's not an amount of data that you can put on your desk. If you try to answer a straightforward question like, "How many pages of the book did the student look at?" the log just says that the person went to page 312 and then, over the last half hour, has clicked up pages eleven times and back pages four times. So you have to figure out that they started on page 312 plus eleven minus four, to figure out what page they're on. Then you have to decide whether they looked at it. We have to look at the plot of how long people looked at each course, and each page, and decide, "Oh, less than ten seconds, we're not going to count that as a valid look. We just think that was navigating from one page to the next." You have to do all that before you have data that are even meaningful, even to answer the question, "What fraction of the pages does the student look at in the book?" We're not using standard procedures to do this. We have to write our own programs. We have to write, not just a little program: we have to figure out how to write it in a generalizable way so that we can use it to analyze the videos as well as the textbook. A significant part of the challenge is to figure out how to avoid double-counting and how not to lose 1,500 students' data. It's not all organized for you. (Pritchard, MIT). Ken Koedinger, Professor of Human-Computer Interaction and Psychology, Carnegie Mellon University, and Director of the Pittsburgh Science of Learning Center (PSLC), drew parallels between the challenges of processing data from traditional instructional data systems and online courses, and MOOC platform data. He asserts that it is easy to underestimate the value of “keystroke” or “clickstream” data from a Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

131

well-designed instructional technology system in understanding the nature of cognitive processes, and easy to overestimate the value of a poorly designed system. He remarked that knowing when a participant has started and stopped a video is of little value unless this information can be combined with information on what students have learned. Currently, the latter is most commonly collected through “structured entry in a graphical interface,” that is, responses to multiple choice or menu-based questions, or entries into a table. Koedinger identified additional current limitations to the use of data from online platforms. For example, when recording activity of users of simulation software, most systems only log the state of the simulation at any one time instead of asking what the learner does to change the simulation. He also observed that there has been significant progress in development of software tools to help code the content of discussion forums, but that privacy issues are more of a challenge for analysis of discussion forum data compared with clickstream data. Whitmer pointed out the need for standardization of the current data formats. He suggested that either the MOOC platform providers take on the task of standardization, or that a combination of researchers and programmers develop a format for everyone to follow. Whitmer also observed that programmers at multiple sites are building data logging routines from scratch, and platform providers are each writing their own data queries, often duplicating effort. A repository for such queries could allow sharing and improved efficiency. Whitmer pointed to the work of the IMS Global Learning Consortium in developing Caliper, a common learning analytics framework that can be used to capture and present learning activity metrics from a variety of systems (see IMS Global Learning Consortium, 2013). Koedinger hypothesized that the current situation arises because of a “not invented here” culture among programmers that steers them towards building their own systems rather than applying or adapting what has already been created. Baker suggested that MOOCs need a “Science of Learning Center” that stores both restricted access data sets from MOOC platforms as well as open access reference datasets for any researcher to use, guides for data logging formats, and analytical tools. The existing Pittsburgh Science of Learning Center (PSLC) already addresses similar data issues for online learning systems through its DataShop, a “data analysis service for the learning science community.” (Baker was formerly the Technical Director of DataShop and is currently a Scientific Advisor). According to Koedinger, PSLC was established with National Science Foundation funding ten years ago to collect data from instructional data systems and online courses, and to use these data to answer research questions. DataShop acts as “a central repository to secure and store research data” and provides “a set of analysis and reporting tools” (https://pslcdatashop.web.cmu.edu/about/), and recommends a standard format for logging data to allow ease of analysis across multiple online learning systems. For example, the data logging and formatting recommendations specify what information is useful to collect about user activity on the platform and how to report data in commonly recognizable spreadsheets. Available tools facilitate data visualization and statistical modeling. Koedinger is currently in discussions with faculty members from MIT, Stanford, and Teachers College regarding the creation of a cross-university collaboration for MOOC data, starting with a discussion of what are the current problems and eventual goals for use of these data. The Anyscale Learning for All (ALFA group), led by Una May O’Reilly and part of MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), appears to be putting a stake in the ground with regards to addressing some of the MOOC data issues with the development of MOOCdb (see Veeramachaneni et Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

132

al., 2013). Dernoncourt et al. (2013), of the ALFA group demonstrate how MOOCdb allows for analysis of course data from both the Coursera and edX platforms. However, it remains to be seen whether other groups will agree to play in this sandbox or will continue to invent their own tools. And until the tools developed by such groups can do more than provide attractive pictures of descriptive data, the frontiers of knowledge about effective pedagogy will not be breached. To realize the full potential of research that can be conducted with the big data from MOOC platforms, Saxberg, Chief Learning Officer at Kaplan Inc., and David Niemi, VP, Measurement and Research, also at Kaplan Inc., suggest that multiple players must engage with each other and strategically plan what data are needed and how the data will be managed. It's kind of a nontrivial thing, actually, to spell out all the data you want in the beginning, and have the system built around that because it's hundreds of potential variables. You're going to have to think about who is every conceivable person that wants to do something with the data, get all of those questions. (Niemi, Kaplan Inc.). You need someone with a deep measurement capability, somebody with deep research capabilities who knows what kind of analysis they're going to want to do at the end, somebody with strong learning science instructional design capabilities because they're going to know what sorts of things they're going to want to try. Then you need your IT people at the table… (Saxberg, Kaplan Inc.). Beyond the planning for production of workable platform data, analysts must be trained in various aspects of handling data, a multi-step process according to Baker. First, the analyst must learn how to store data in accessible formats. Second, the raw data needs to be transformed into meaningful variables through “feature engineering.” Baker emphasized that this process is hard to teach and that there are few courses available on these techniques. A third step involves identifying patterns in the data. Analysts must also master broader theoretical questions about how data mining can be used to understand learning processes, and consider issues such as privacy, societal values, and potential impact of the research. Issues of data privacy, access, and sharing are as yet unresolved in the relatively new legal territory of MOOCs (see Dillenbourg et al., Eds., 2014). For example, to whom do the platform data belong and who can have access without violating FERPA requirements? We found some interviewees more enthusiastic than others regarding the prospect of sharing MOOC data among institutions for research or other purposes, with or without a license fee, but many pointed to significant confusion regarding whether any or all MOOC participants are covered by FERPA or other regulations given that the courses are free and non-credit-bearing, that most students are not enrolled at the MOOC provider’s campus, and that many participants are international. But as Means from SRI pointedly noted, “There's a lot of misinterpretation of FERPA and a lot of times it's actually a really convenient excuse for not providing information to people.” It is apparent that while the potential for MOOCs to contribute significantly to the development of personalized and adaptive learning is high, the reality is far from being achieved. A great deal of coordination and collaboration among researchers, instructional designers, and programmers will be necessary to result in meaningful improvements to teaching and learning.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

133

Resource Requirements and Costs of MOOCs In this section we first review the resources required to produce and deliver, or “consume” MOOCs. Subsequently, we summarize the actual costs of MOOCs from the perspective of the producer (i.e., the college, university, or museum, as opposed to the platform provider or participant) including the results of cost analyses we conducted of MOOC production and delivery at four different institutions. The details of each analysis are available in Case 10, Case 11, Case 12 and Case 13. For MOOCs that are delivered via third party platforms, there are often significant, additional costs to the platform provider which may or may not be passed on to the MOOC producers, depending on whether there is a direct charge for the platform services or a revenue-sharing agreement. The major cost drivers in MOOC production and delivery are: the number of faculty members, administrators, and instructional support personnel participating in the process; the quality of videography; the nature of the delivery platform; technical support for participants; programming for special features such as computer code auto-graders, virtual labs, simulations, or gamification; and analysis of platform data. MOOC production teams that were described to us seldom included fewer than five professionals and, in at least one instance, over 30 people were involved. Course length was not a reliable predictor of costs. MOOC production resource requirements Personnel At most institutions, MOOC development has been a team effort, sometimes including administrators, instructional designers, instructional technologists, programmers, project managers, videographers, and evaluation specialists in addition to the faculty members and teaching assistants (TAs). In contrast to the often solo endeavor of developing a traditional course, MOOCs entail what Schneider at Stanford University described as “a distributed sense of responsibility for the design of the course.” A few institutions have created offices of online learning to facilitate the development of MOOCs and other online and blended courses. A committee may be established to receive and review project requests from faculty members and to award seed grants. For example, at Stanford University an entire unit entitled the Office of the Vice President for Online Learning (VPOL) has been created within the last 18 months. By October 2013, VPOL employed 30 people (four administrators, three researchers plus several doctoral students and post-doctoral fellows, seven in platforms and engineering, five in course design and development, five in course and media production, and one in communications). According to Collier, this team is working on over 100 projects, a quarter of which are MOOCs. Seed grants of $25,000 are provided to faculty members for innovative projects in online and blended learning. Such an office alone would require several million dollars per year to operate, before considering the efforts of faculty members and TAs. Other institutions relied on expanded Centers for Teaching Excellence or Educational Innovation to help design and develop MOOCs. All interviewees who had been involved in the development of a MOOC reported the effort being two to three times greater than creating a traditional course. Instructors typically spent several hundred hours over several months preparing and re-purposing course materials, and practicing lecture delivery prior to video-taping; several days on actual shoots; and one to two days reviewing the finished video. To create one hour’s worth of MOOC video-lecture required three to ten hours of preparation according to several faculty members, the lower end of the range being in instances where the materials were being re-purposed from existing lectures. To create ten minutes of voice-over-PowerPoint video required six Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

134

to eight hours according to an interviewee at a private university. See, for example, Cima’s (2013) account (pp.15-17) of the time and effort required to prepare and deliver the MOOC Introduction to Solid-state Chemistry. Cheal at San José State University, among other interviewees, likened the effort involved in creating a MOOC with writing a textbook in a team. At some institutions faculty members are granted a course release and/or paid stipends ranging from $3,000 - $15,000 per course while developing and delivering a MOOC, but the opportunity costs of the instructor’s time are likely to be higher in many instances. We frequently heard estimates in the order of 400 hours of faculty member time per MOOC developed. While we did not hear of instructors being paid per additional student, Silverman at NYIT points out that the union contract governing faculty member compensation at his institution requires that instructors be paid $50 - $250 per student above a fixed student load. Clearly such clauses would need to be revisited in situations where an instructor offered a MOOC that attracted large enrollments. Courses that have been developed and significantly refined over several years may cost substantially more than those created over a period of a few months. Pritchard, from MIT, estimates that three to four post-doctoral-fellow years have been invested in developing and refining his Mechanics ReView course which has been offered as a MOOC three times. The post-doctoral fellow alone would account for $200,000-$250,000 at a national average salary and benefits rate. In addition, many hours of undergraduate and faculty time were committed to the course development. MOOC platform costs It appears that platform costs for MOOCs offered by Coursera and Udacity are often subsumed into revenue-sharing agreements (see, for example, the Coursera contract with University of Michigan; Young, 2012), in which the platform provider takes responsibility for hosting the courses and drawing in participants, but also keeps most of the revenues earned from those participants who purchase certificates or other services. Coursera’s arrangements with state university systems vary but, in one example, the State of Tennessee was asked to pay $3,000 per course for use of the Coursera platform, plus $25 per student (see Rivard, 2013). One community college paid Udacity a fixed sum of $30,000 for its role in helping to develop and deliver a MOOC on the Udacity platform and providing data on participant activity and performance. edX, a nonprofit organization, offers a “spectrum of full service to self-service” options with respect to MOOC development and delivery with varying arrangements for revenue sharing (see Kolowich, 2013e). Partners may commit to negotiable cash or in-kind contributions (e.g., personnel time, code development, collaboration on research) in order to be supported in their use of the edX platform and for their courses to be hosted on the edX servers. Optional services for assistance in developing and delivering the courses are for additional fee. These services might include instructional design training or video production. A typical contribution might be $250,000 per course and $50,000 for a re-run, plus a revenue-sharing arrangement. One interviewee estimated the costs to a public university for the use of a platform developed by a small, for-profit entity at $20,000 per course. These fees covered the programming costs of creating the platform, “onboarding” courses, hosting, monitoring, managing and maintaining them, data provision, and technical support for course participants. Platforms such as Open edX or Canvas Network are available free but the MOOC provider must have the capacity to create and upload compatible materials, manage the course delivery and attract participants.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

135

Videography Many MOOCs include an hour or more of video per week and we found that videography costs were highly variable. We conducted a detailed estimate of professional videography costs which resulted in a sum of $4,300 per hour of finished video (see Appendix IV). Some institutions use in-house video production teams which cost significantly less (see Case 13), while others pay substantially more than our estimate (see Case 12). We even heard of two cases where a professor established his own recording studio in a home basement. Assessment Assessing students at scale also requires a significant investment of resources, first to develop assignments that can be graded in large numbers and, secondly to devise peer-grading or auto-grading mechanisms or to provide enough TAs to cope with the quantity of assignments to be reviewed. In most cases, multiple choice or short answer tests are devised for MOOCs. All questions and answers must be thoroughly “debugged” before use. In some academic departments, a large bank of questions is collected so that students can be assigned similar questions at random to allow multiple attempts at mastering a concept or problem-solving strategy, and to help avoid cheating. At the University of California, Berkeley, four auto-graders have been developed, requiring an estimated 1,000 hours of engineering time, to allow MOOC participants’ computer code assignments to be automatically assessed. A few institutions were in the process of hiring a Director of Assessment or Director of Learning Outcomes, or had recently done so. Obtaining copyright permissions Additional costs may be incurred for use of copyrighted materials, both for the personnel time involved in obtaining permissions and for fees to cover use of the material. Some institutions worked to substitute copyrighted images and text with open or self-generated artifacts. Refreshing MOOCs to be re-run After the first iteration of a MOOC, almost all institutions have made adjustments to the design and content before re-running. These vary in scope and effort, as do expectations for the need to refresh content on a regular basis. White expected that the computer science MOOCs offered by Georgia Tech would be updated every three years on average, while Li Fengliang of Tsinghua University insisted that MOOC content should be changed every year. MOOC delivery resource requirements While some MOOCs are designed to run without faculty or other ongoing intervention, many online courses involve significant instructor and TA or Graduate Assistant involvement. Several faculty members reported many hours unexpectedly spent on intensive “debugging” and mid-course corrections during the first offering of a MOOC. More routinely, instructors estimated spending at least 20-30 minutes per day participating in the discussion forums. MOOCs often have one or more TAs assigned to monitor these forums and to respond to participant questions. Fox at the University of California, Berkeley estimated 40-50 hours per week of TA time needed to support an engineering MOOC. Additionally, some participants become de facto “community TAs” by being especially helpful in responding to the questions of others participants. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

136

Varying amounts of effort are being made to evaluate the success of MOOCs, with a few institutions assigning internal evaluators to assess their impact or requesting that faculty members conduct their own reviews and submit a formal report to other faculty members on lessons learned. Analyzing the MOOC platform data beyond what is readily available in the participant activity dashboards in order to improve future iterations of MOOCs can be time-consuming and expensive. Other university or college services required to support MOOCs In addition to the direct costs of producing and delivering MOOCs, Cheal (2012) provides insights into a plethora of additional considerations for institutions engaging with MOOCs. For example, MOOCs can only attract massive audiences if they are sufficiently marketed. While the platform providers such as Coursera, edX, and Academic Partnerships fulfill these marketing and communications functions for their partner institutions, those institutions using more “do-it-yourself” platforms must find suitable advertising channels. Computing and Internet services for on-campus students participating in MOOCs may need to be increased or upgraded, for example, retrofitting buildings to provide enough bandwidth capacity for many students to simultaneously download video. Institutional websites and learning management systems are likely to need adjusting to provide an access point to MOOCs. While most major platform providers offer technical support to participants, students who are enrolled at a college or university and are participating in online courses, including MOOCs, often contact their own institution’s help desk. This may require the platform provider to work with help desk staff to prepare them for inquiries, and may require additional staff. A variety of administrative offices are likely to be involved in activities such as establishing contracts between the institution and online platform provider, and between the institution and its faculty members to address intellectual property rights, revenue sharing, faculty compensation, and workload issues. Compliance with disability regulations in MOOCs must be regularly audited and enforced, and accommodations made, for example, extra time on quizzes and exams for students with learning disabilities. For institutions providing credit for MOOCs, the student admissions, registration, billing, authentication, and crediting systems need to be aligned with platform enrollment procedures (see Cheal, 2012). If prerequisites are required for credit-earning participation in a course, a system must be developed to handle large numbers of students. MOOC “consumption” resource requirements While the above considerations pertain to institutions that are producing MOOCs, there are also many resource requirements for institutions that are integrating MOOCs created by others into their own courses. Bacow, President Emeritus of Tufts University, identified the following considerations: • • •

Search cost: time required to find a MOOC that is relevant to the instructors’ existing course. Comparable to putting a course together from books, articles, and other materials. Familiarity with material/adaptation cost: time required to become familiar with the material as it is being delivered so that it can be interpreted and translated for instructor’s own objectives and student population. Licensing fees to use the material. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

137

• • •

Finding material to supplement the MOOC in order for it to fit the existing curriculum and student needs. For example, if you are using a statistics MOOC in an environmental statistics course, you would want different materials than for a course on statistics for engineering. Assessment: creating your own exams and problem sets. Space cost: to bring students together, assuming the MOOC is being used in a blended learning course. If you are a traditional residential institution, it is difficult to explain to a student why you should be paying $50,000 for tuition to just sit in your dorm room and look at it online. So you have to offer some personal contact. Professor's salary for the course. Certification costs: giving credit, recording it, being able to indicate that students completed the course. Cost of technology: if you are reliant on technology to deliver a course, you have to have redundancy. You cannot afford to let anything go wrong. You have to have people to resolve technology problems immediately. (Bacow, President Emeritus, Tufts University).

Costs of MOOC production and delivery Using the ingredients method (Levin & McEwan, 2001) we estimated the actual costs of MOOC production and delivery at four different institutions (see Case 10, Case 11, Case 12, and Case 13). Personnel costs alone ranged between $29,000 and $244,000 per MOOC, depending on the number of people involved in the process, the quality of video production and the amount of time dedicated. Although it would be more accurate to estimate costs of facilities, equipment, and overhead individually, we rely on Levin and McEwan’s (2001) observation that, for educational interventions, personnel costs typically account for 75% of total costs. We estimate total costs per MOOC of $39,000 $325,300, as shown in Figure 3. Table 5 additionally indicates the length of the MOOCs involved and, in two cases where completion data were available, cost per completer. Figure 3: Estimated Costs of MOOC Production and Delivery at Four Institutions (low end estimates)

Large Midwestern University American Museum of Natural History University of Manitoba Teachers College, Columbia University $0

$50,000

$100,000

$150,000

$200,000

$250,000

By comparison, Bates at Contact North currently estimates the costs to develop a regular three-credit online course delivered on a learning management system at $35,000 -$50,000 (A. Bates, personal Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

138

communication, April 29, 2014; Bates & Sangra, 2011, pp. 163-166). Laster, of McGraw-Hill Education, believes that the initial investment of time and resources to develop MOOCs is likely to be three to four times greater than for developing a face-to-face course based on the same content and learning goals. Many interviewees provided their own rough estimates of MOOC production and delivery costs, but acknowledged that these did not rely on rigorous time-use studies or actual cost analyses. The rough estimates reported by interviewees ranged from as low as $5,000 per course at a university where the online course development infrastructure was already in place and existing materials could be repurposed for MOOCs, to as high as $1.2 million at a university that is new to online course delivery and is building sophisticated auto-graders, virtual laboratories and simulations, which may eventually be used in multiple courses (see Box 10). The community colleges in our sample estimated spending in the order of $75,000 per MOOC because they were working from grants of $50,000 and contributing in-kind efforts. The original projected costs for Georgia Tech’s highly publicized Online M.S. in Computer Science (OMSCS) program are publicly available. We provide a review of these costs and the staff/student ratios for the OMSCS courses in Case 5, along with our concerns over the feasibility of these plans. Table 5: Estimated Costs of MOOC Production and Delivery at Four Institutions Institution

Type of MOOC

Length of MOOC (weeks)

Total estimated costs per MOOC

Costs per completer

Teachers College, Columbia University

xMOOC

$38,980

$74

University of Manitoba

cMOOC

$65,800 – 71,800

American Museum of Natural History

xMOOC

$104,620

$272

Large Midwestern University

xMOOC

5-8

$203,770 – $325,330

* Completion data were not available for these MOOCs. See Cases 10, 11, 12, and 13 for sources.

The question for MOOCs over the longer term is whether variable costs can be contained by automating functions and substituting instructional support provided by expensive faculty members with less costly TAs, part-time instructors, or peer-to-peer learning and assessment. In 2012, just as MOOC-mania was beginning to build, Bowen et al. provided an illustration of how instructor costs could be reduced in online and hybrid courses by using part-time instructors and altering course configurations. Some of these ideas could be adopted for MOOC delivery. Many MOOC developers anticipate that the costs of re-running a MOOC will be substantially lower than the costs of initial development. In Case 10 we show that the costs of re-running the first cMOOC, Connectivism and Connected Knowledge, were 38% lower than the costs of the initial offering. However, given the intense level of instructor involvement in cMOOCs, this is unlikely to be a useful predictor for xMOOC re-runs where instructor involvement may be minimal or absent. James at Mt. San Jacinto Community College expected that the re-run costs for the college’s xMOOC, Crafting an Effective Writer, would be minimal, perhaps less than $1,000, compared with her estimate of $75,000 for the initial development and offering. Such assumptions should be rigorously tested through careful cost analyses Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

139

and we recommend that, going forward, MOOC producers attempt to document these re-run costs to help assess the sustainability of MOOC production. Box 10: One Interviewee’s Estimates of MOOC Production and Delivery Costs [If developing a course] for a traditional class for 30 (or 300) students costs $100,000 then, on average, the first time you [turn a class into a MOOC] it might be $250,000 to [produce it] and another $75,000 $150,000 to deliver the first time. Subsequent delivery would be about half that - call it $50,000 - $75,000. Refreshing could be anywhere from $25,000 to $75,000 depending on the nature of the class and nature of the changes. However, …there are other complexities. The first Biology course in a track might require a protein modeler which costs $75,000-$100,000 to build. Once built, we can use it for other Biology courses. So those subsequent courses in the same sequence should be substantially less expensive to create and deliver. Some courses could cost as much as $750,000+ to $1.2 million to produce - these "monster" classes would spawn a number of shorter courses.

While our cost analyses help shed light on the costs of MOOC production and delivery, several important questions remain unanswered, in addition to whether repeat offerings will cost less than the first delivery. While the cost numbers seem high and many instructors perceived greater expenditure of effort to create a MOOC as compared to developing a face-to-face or regular online course, we have no documentation to prove this. If we wish to answer the question of whether MOOCs are a cost-effective means to deliver education, we must be able to compare the costs of MOOCs to the costs of the alternative delivery mechanisms, as well as the effectiveness of each alternative with respect to a common outcome of interest, such as increasing participants’ level of knowledge or skill in a specific subject area.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

140

Case 10: Cost Analysis for Development and Delivery of Connectivism and Connected Knowledge (a cMOOC) Connectivism and Connected Knowledge (CCK08), the first course to be dubbed a “MOOC,” was developed and delivered in 2008 by George Siemens and Stephen Downes. It was offered at the University of Manitoba to enrolled students for fee and for credit and also as a free, non-credit-bearing course to any other interested participants. Twenty-five students formally enrolled in the 12-week course and another 2,300 participated. The course has been re-run three times since. Siemens estimated the time burden for CCK08 development and delivery as follows: • •

Course design and development: 100-150 hours over two months; Course delivery: 70 hours per week for the first 2-3 weeks (interacting with students and posting on discussion forums or writing blog posts to summarize discussion and activities), tapering down to 30 hours per week in the twelfth week.

At the lower end of Siemens’ estimates, the total number of hours would be around 715. At the high end, the time commitment could have been as much as 770 hours. We estimate costs at each end of the range. Downes estimated his total time commitment for CCK08 at 88-108 hours: • • •

20-40 hours in programming time to make adjustments to the gRSShopper course aggregation software that he had developed over many years; 20 hours setting up the course website; 4 hours per week during course delivery to maintain the site and prepare audio archives.

Using U.S. national average salary and benefits rates for public postsecondary faculty members and public sector research scientists, the costs of personnel time to replicate CCK08 ranges from $49,400 to $53,800. No technology support, learning designers, or teaching assistants (TAs) were utilized in the development and delivery of CCK08. We do not estimate costs individually for facilities, other equipment and overhead but assuming personnel costs amount to 75% of total costs (see Levin & McEwan, 2001), the total cost estimate to replicate the cMOOC CCK08 would be $65,800 to $71,800 as shown in Table 6 below. Re-runs of CCK08 required less design and development time. Additionally, with better course management software, weekly delivery time for CCK12 fell to 30 to 40 hours for the first two to three weeks. Some repeat students self-selected as TAs and reduced the instructors’ time burden by helping manage the forums, responding to inquiries and providing guidance to new students. (Downes estimates that around 100 participants from CCK08 returned for CCK09). Set-up time for the course website dropped from 20 hours to 4 hours. Twitter and Diigo were incorporated as new tools.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

141

For Siemens, we estimate the total time commitment for a CCK08 re-run at 284 hours: • • •

20 hours to “refresh” the course design and resources before a new launch; 28 hours per week in delivery for the first 3 weeks; 20 hours per week in delivery for the remaining 9 weeks.

For Downes, we estimate the total time commitment for a CCK08 re-run at 72 hours: • • •

4 hours website set-up; 20 hours adjusting gRSShopper to accommodate new tools; 4 hours per week maintaining the course site. Table 6: Estimated Replication Costs for the First Run of CCK08 and Re-run Ingredient

First run low estimate

First run high estimate

Re-run estimate

Instructor 1

$44,756

$48,199

$17,777

Instructor 2

$4,597

$5,642

$3,761

$9,015

Facilities, equipment, overhead

$16,451

$17,947

$10,184

Total

$65,804

$71,788

S40,737

TAs

The possible range of time committed by the self-selected TAs could be very wide and we do not have reliable data to use. However, given the substantive role played by these participants, the opportunity cost of their time should be considered when replicating a MOOC. We use an estimate of 350 hours total, using an assumption that the TAs collectively replace the reduced hours in Siemens’s delivery time. We use the lower end of Siemens’ delivery time estimates to be conservative. Total estimated costs for the re-run are $40,740, 38% lower than the low estimate for the first run. Siemens indicates that planning time for a cMOOC is greater than for a face-to-face course because there is less room for correcting errors “on the fly.” Delivery time is far greater: with a face-to-face course, there are fewer learners resulting in less writing to be read, and less time required to synthesize contributions. With over 2,000 students in a MOOC, Siemens observes that the instructor “can be active in a huge range of conversations.”

Sources: George Siemens, formerly at Athabasca University, and Stephen Downes, National Research Council.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

142

Box 11: What is gRSShopper Anyway?

â&#x20AC;&#x153;It is a Swiss-Army-Knife-of-an-application that is optimized to support distributed network resources.

It's a blogging tool, it's an RSS reader, it's an e-mail mailing list. Websites such as blogging sites or discussion sites can create a data representation of their content called "feeds." A feed is a machinereadable version of the content of the site. When the site is updated, the feed is updated. An RSS reader accesses a specific type of feed called an RSS feed. It retrieves that feed and displays the content in a centralized location for the user. People use an RSS reader to read a number of different websites all in a single location. gRSShopper retrieves feeds from students participating in our MOOCs. gRSShopper is more than just an RSS reader because it can publish the aggregated feeds as a web page or as an e-mail newsletter. Our MOOC participants write in their own blogs which might be on Blogger, WordPress, Tumblr, Posterous, and so on. gRSShopper aggregates these feeds, organizes them, puts them in a searchable database and automatically sends out a newsletter of all the posts that participants made in the previous 24 hours. To participate in one of our courses, people just read an e-mail, click on any link shown and go directly to some other person's website to look at what they posted on their website, and perhaps comment on that website. They don't even have to come to our course website, they can simply interact with other participants in the course. Most online courses are site-based, centralized, and participants need to visit the site to access resources. gRSShopper allows you to stay where you are. It will come get you.â&#x20AC;? Source: Stephen Downes, National Research Council.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

143

Case 11: Costs of MOOC Production at a Large Midwestern University Before entering the MOOCspace, this university already had an established infrastructure and personnel for the development of online courses. Once the decision was made to offer MOOCs, a small number of faculty members with prior â&#x20AC;&#x153;media experienceâ&#x20AC;? were invited to develop and deliver five- to eight-week MOOCs, primarily to showcase the university and engage new audiences. Each faculty member was assigned a design and support team of five to six people to help in the design and production of the MOOC. Each team included a project manager, instructional designers, instructional technologists, and a liaison to the video production team. Additional personnel supervised the design and support teams, and provided programming capacity, overall project management, evaluation, and administrative services, including campus events to increase knowledge about MOOCs. As a routine part of the project management function at this university, detailed time logs are kept by each design team member so that costs for these personnel can be tracked accurately. We used the cost estimates provided by the university for these personnel in our analysis because we did not obtain enough detail regarding the personnel ingredients (e.g., specific role, level of experience, highest degree of education) to allow us to assign prices ourselves. Faculty member and teaching assistant (TA) time were not logged but estimates were obtained either during or after MOOC production. We estimated costs for faculty members and TAs ourselves, using national average salary and benefits rates for postsecondary public institutions. For the first three MOOCs created and delivered, the time burdens and resulting cost estimates are shown in Tables 7 and 8 respectively. Table 7: Range of Hours Spent per MOOC on Design, Production, and Delivery at a Large Midwestern University Low

High

Design team

200

500

Video production team

700

900

Platform/technical issues

150

155

Faculty member

220

650

1,140

2,245

Total

Table 8: Range of Estimated Personnel Costs per MOOC for Design, Production, and Delivery at a Large Midwestern University

Design and support team Computer programming unit Management (avg. across 3 MOOCs) Faculty member TA Total personnel costs

Low

High

$70,000

$125,000

$15,000

$77,200

$5,630

$13,770

$13,029

$ 152,830

$244,000

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

144

The faculty time burden was relatively low because the dedicated design and support team took on much of the task of course design and development. Design team time varied depending on the complexity of the learning activities. For example, in one MOOC, participants were required to upload artifacts and report data which were automatically analyzed. Overall results were shared out to all the MOOC participants. The amount of time required for platform and technical issues such as uploading content and â&#x20AC;&#x153;getting things to work on Courseraâ&#x20AC;? dropped as the team built experience with the platform. Note that the costs of facilities, equipment and overhead are not estimated here but, if we assume that personnel costs account for 75% of total costs (Levin & McEwan, 2001), the range of total costs per MOOC would be $203,770 â&#x20AC;&#x201C; $325,330. We note that salary levels at this geographical location may be lower than national averages so that costs for the non-teaching personnel could be higher on a national average basis (in the order of a few thousand dollars).

Source: Evaluator at a large Midwestern university.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

145

Case 12: American Museum of Natural History MOOC Initiative: Resource Requirements and Expected Benefits Between September and December 2013, the American Museum of Natural History (AMNH), as an invited partner in the Coursera Teacher Professional Development program, serially delivered three four-week long MOOCs targeted at science educators. Planning efforts began in Spring 2013 and involved a team of museum professionals who had significant previous experience in developing and delivering online education, including Seminars on Science, an online education program for science teachers. The core MOOC production team comprised a project director, a project manager, an in-house video producer, an educational technologist, and a senior administrator who also served as one of the MOOC instructors. While the museum had already previously developed many digital resources including science-content videos and educational essays on science topics, MOOCs presented a new challenge to develop lecturebased videos with â&#x20AC;&#x153;talking headsâ&#x20AC;? or voice-over PowerPoint presentations, multiple-choice quizzes, peer-graded assessments, and pre- and post-course surveys. In addition, large amounts of MOOC participant data became available for analysis and interpretation. The personnel effort associated with the production and delivery of the three MOOCs, as well as processing the course-related data are summarized in Table 9, based on estimates of time use provided by the AMNH team. The project manager and project director spent the equivalent of 25 and 11 entire workweeks respectively on the project, while the instructors spent, on average, about six workweeks each, shooting videos and developing, adapting, or reviewing course content. The core team met once or twice a week for one to two hours to plan, design, execute, and review the MOOC production and delivery. The teaching assistant (TA) spent a total of 400 hours managing the discussion forums, processing survey responses, and reviewing the platform data. Using national average salaries and benefits rates for personnel, wherever possible at similar positions in postsecondary institutions to allow comparability with the other MOOC costs we present, we estimate the personnel costs to develop the three MOOCs created by AMNH at $235,000, or $78,470 per MOOC. If we assume that personnel costs account for 75% of total costs once facilities, equipment, and overhead are included (Levin and McEwan, 2001, p.53), we estimate total costs at $104,620 per MOOC. Of the total 39,685 participants who initially enrolled in the three MOOCs, 1,155 earned a certificate of completion (scoring 72%-84% of possible points) or a certificate of distinction (scoring 85% of possible points). Costs per completer for the MOOCs amount to $272. Because we are mostly using postsecondary personnel prices, we assume a 1,560-hour year, a rate of 28.5% for benefits, and that all employees are in-house. We expect that actual costs for AMNH would be significantly higher given that New York City costs are above the national average, benefits rates at the museum are 40.5% for full-time staff, and several of the personnel were hired as consultants at hourly rates. For example, the HTML writer cost almost three times more than the national average estimate, and costs of video shooting and editing amounted to 5.5 times more. The costs of the platform, captioning, content hosting, and analysis of user data to populate the data dashboard were assumed by Coursera. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

146

Table 9: Personnel Hours Spent by AMNH to Develop Three MOOCs Ingredient

Hours

Senior management

125

Project director

454

Project manager

980

Instructors

910

Educational technologist

174

400

Evaluation expert

Graphic designer

350

Video producer

293

Video shooter

Video editor

210

HTML writer

CSS Writer

Legal personnel

Marketing personnel

Business manager

Total for 3 MOOCs

4, 045

Per MOOC

1,348

AMNH will be offering the three MOOCs again throughout 2014 and while adjustments are being made to each course, initial costs of production can be amortized over the multiple repeats, allowing a â&#x20AC;&#x153;run rateâ&#x20AC;? to be established for the ongoing delivery and maintenance of the courses. In addition to assessing success of the MOOC initiative in terms of course completions, other indicators of impact are being monitored. For example, survey responses and follow-up interviews with consenting participants will indicate whether teachers report learning new content and using museum materials and teaching strategies in their own classrooms. The number of people who visit the museum online or on-site, the number of new international and domestic users of AMNH content, (e.g., the science media videos or Science Bulletins that are recommended in the MOOCs), and the number of people who enroll in Seminars on Science are being tracked and compared to pre-MOOC activity. Sources: Dr. Ro Kinzler, Senior Director, Science Education, American Museum of Natural History, NY; Dr. Robert Steiner, Director, Online Teacher Education Programs, American Museum of Natural History, NY; Maria Janelli, Senior Manager, Online Teacher Education Programs, American Museum of Natural History, NY.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

147

Case 13: Time-by-Task and Cost Analysis for Big Data in Education Development and Delivery Big Data in Education was an eight-week MOOC delivered on the Coursera platform between late October 2013 and mid-December 2013. Ryan Baker, a faculty member at Teachers College, Columbia University, developed the course, adapting it from a 16-week on-campus version usually taught to classes ranging in size from eight to fifteen students. While Baker’s initial target enrollment for this specialized course on educational data mining and learning analytics was 200 participants, 48,058 individuals registered for the course, 2,650 watched the final lecture, and 526 completed the last assignment. Planning and preparation for the course began in mid-March 2013. Big Data in Education was free, open to any participant, and non-credit-bearing. Participants who completed the course earned a Statement of Accomplishment. Baker kept track of time and tasks related to the MOOC in an Excel spreadsheet from June (when our study began) to the end of December 2013. Hours spent on activities prior to that date were estimated. Total time logged plus time estimated was 176 hours, with the heaviest burden falling during the first three months of planning and preparation of materials, the month prior to launch, and the first few weeks of course delivery (see Figure 4). Time spent on various tasks is shown in Figure 5 and included: • • • • • •

Creating course materials such as slides, assignments, and quizzes (58 hours); Set-up and video-recording using ScreenFlow software (46 hours yielding 6 1/2 hours of finished video used in the MOOC); Planning, bureaucracy, and coordination with Coursera, the teaching assistant (TA), and the course production team (37 hours); Participating in the forums and responding to participant e-mails (26 hours); “Debugging” slides, assignments, and quiz questions during the course (7 hours); Open office hours (3 hours).

Hours per month

Figure 4: Faculty Member's Hours by Month for Development and Delivery of Big Data in Education 60

50 40 30 20 10

18.3

50.3 3

2.7

10.6

38.8

13.4

2013 Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

148

Over the 41 weeks of preparation and delivery, Baker spent an average of 4 1/2 hours per week on the course with a high of 22 1/2 hours in one week and a low of zero hours in another. In addition to Baker, several other personnel worked on the MOOC: •

•

• • •

A teaching assistant spent approximately 15 hours per week over 16 weeks for a total of 240 hours, with the heaviest burden falling during the three weeks prior to course launch and during the first three weeks of course delivery. Tasks included: coordinating among faculty member, video team, and Coursera’s course coordinator; checking that the uploaded videos were working; posting assignments and “inline” quiz questions (which are embedded in the videos); and participating in the discussion forum. Seven individuals from the Educational Data Mining Laboratory at Teachers College read and participated in the discussion forums. Two were M.A. students, three were Ph.D. students and two were post-doctoral fellows. We estimate 2 hours per person per week over the 8 week period for a total of 112 hours. A senior administrator coordinated the production activities one hour per week for eight weeks. Two video-specialists from Columbia Center for New Media Teaching and Learning (CCNMTL) edited the video, linked files, requested captioning, and uploaded video, for three to five hours per week totaling around 32 hours. A senior educational technologist from CCNMTL served as the day-to-day project manager for the MOOC production and delivery, spending eight hours per week for four weeks prior to launch, five to six hours per week over the first few weeks of course delivery, two to three hours per week over the last few weeks of course delivery, and ten hours after the MOOC finished, for a total of around 75 hours. This included monitoring the online discussion forum for technical questions, averaging three hours per week over eight weeks for a total of 24 hours.

Figure 5: Faculty Member's Hours by Task for Development and Delivery of Big Data in Education

Open office hours Debugging

Forums, e-mail

Coordination/planning/bureaucracy

Set-up/recording

Creating course material

30 40 Hours

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

149

Cost analysis of Ryan Bakerâ&#x20AC;&#x2122;s MOOC: Big Data in Education Based on the above information, we estimated the personnel costs required to replicate the development and delivery of Big Data in Education using national average salaries and benefit rates for postsecondary personnel at private universities. We obtain an estimate for personnel costs of $29,240 (see Table 10). We have not estimated the costs of facilities, computers and other equipment, or overhead, but if we assume that personnel costs account for 75% of the total costs for the course (Levin & McEwan, 2011), the total estimate would be $38,980. The costs of the platform, captioning, content hosting, and analysis of user data to populate the data dashboard were assumed by Coursera. Table 10: Personnel Costs for Big Data in Education Development and Delivery Personnel Ingredients

Cost

Faculty member

$12,354

Teaching assistant

$4,950

Forum monitors

$5,377

Education administrator - educational support services

$584

Educational technologist: project manager

$4,725

Video Editor

$1,248

Total Personnel Costs

$29,238

In a sensitivity analysis, we substituted the national average faculty member salary (which averages across assistant, associate, and full professors) for the average full professorâ&#x20AC;&#x2122;s salary at a research institution. This yielded a total cost estimate of $45,010, compared with our baseline estimate of $38,980. In a second sensitivity analysis, we used the average salary for a full professor at Columbia University (http://chronicle.com/article/2013-AAUP-Faculty-Salary/138291?cid=megamenu) and adjusted prices to reflect a New York metropolitan area. Resulting costs totaled $69,510. In a final sensitivity analysis, we substituted the estimated costs of externally contracted video production ($4,300 per hour of finished video as shown in Appendix IV) for the video editors. This substantially increases total costs to $76,250 if we use the national average cost of a faculty member, and to $110,950 if we use the average salary and benefits of a full Columbia University professor and New York prices. A summary of these sensitivity analyses is shown in Table 11. If Big Data in Education were run multiple times using the same materials, the costs of time invested in producing the course, which we estimate to be 70-80% of the total, could be spread over the multiple runs. With 526 students completing Big Data in Education, estimated costs per completer using the baseline cost analysis would be $74. If the course were developed and delivered by a full Columbia University professor and an external video production team, with prices adjusted for a New York metropolitan area, costs would rise to $211 per completer.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

150

Table 11: Estimated Replication Costs of Big Data in Education under Various Assumptions Faculty and benefits assumptions

Geographical adjustment

Video Production

National average private university faculty

N/A

Internal

$38,980

$74

National average private university full professor

N/A

Internal

$45,010

$86

NY, metropolitan

Internal

$69,510

$132

External

$76,250

$145

External

$110,950

$211

Average full Columbia University professor National average private university faculty

N/A

Average full Columbia University professor

NY, metropolitan

Total estimated costs

Estimated costs per completer

Non-quantified consequences of MOOC production As a result of spending time on the development and delivery of this MOOC, Baker documented significantly lower productivity compared to the prior year in terms of submitted conference proposals, journal articles, and book chapters (see Table 12). It is not obvious how to quantify these changes in economic terms but there is clearly a trade-off to be considered in terms of the professional status and institutional visibility to be gained by delivering a MOOC to thousands of diverse participants vs. disseminating research findings to an audience of conference attendees and journal or book readers. While the number of grant proposals that Baker submitted remained unchanged, it is possible that acceptance could be greater or lower depending on the funders’ perceptions of the value of one activity over another. Possible benefits to both the instructor and the institution could accrue going forward if grant proposals were more likely to be accepted due to a MOOC being seen as a viable vehicle for the large-scale dissemination of research findings. Additionally, delivery of the MOOC could result in a higher number of applicants to Teachers College programs in general and to the M.A. program in Cognitive Studies in particular. To track this it would be necessary to include a survey item in the admissions package asking what factors had influenced the applicant’s decision to apply, with the MOOC being offered as one of a number of possible options. Table 12: Professional Productivity Changes as a Result of Time Devoted to Big Data in Education Items submitted

Sep ’12 - Feb ‘13

Sep ’13 - Feb ‘14

Year-to-year change

Research grant proposals

None

Journal articles/revisions

- 46%

- 67%

- 33%

Book chapters Conference paper proposals

Sources: Ryan Baker, Teachers College, Columbia University, NY; Yuan “Elle” Wang, Teachers College, Columbia University, NY; Michael Cennamo, CCNMTL, Columbia University, NY. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

151

Where are MOOCs Going over the Next Five Years? The final two open-ended questions of our interview protocol invited interviewees to predict how the MOOC phenomenon might play out over the next five years: • •

Where do you see you see things going with MOOCs over the next five years? (Generally, and at your institution). How do you see MOOCs helping your institution meet its goals over the next five years?

Most interviewees were willing to speculate about how MOOCs might evolve over the next few years and how they might fit into the changing landscape of higher education. We begin this section by summarizing expectations about more general changes in higher education and subsequently present specific ideas raised regarding MOOCs of the future. Our goal was to surface a variety of ideas as opposed to developing a rigorous frequency report on a list of possibilities from a stratified sample of participants. Accordingly, we provide some idea of the “popularity” of each idea by loosely indicating how many interviewees mentioned it, using terms such as “many, some, several, a few, a couple, one.” Given the open-ended nature of the questions and variety of interviewee roles, we caution readers against focusing too much on this quantification. The future of higher education Many interviewees envisaged a future of more “unbundling” of educational services, more choice for students in how their education is delivered, greater price competition among providers, and greater use of technology in education. As BenDaniel from Cornell University observed “University education is reaching a point of diminishing returns as the cost of university staffing is growing unsustainably…. something will change.” Several questioned whether two- or four-year degrees are the most useful way to acquire an education and suggested that while colleges and universities may continue to provide foundational skills and knowledge, the flexibility of online education will allow for more “just-in-time” learning experiences throughout an individual’s career. Several interviewees subscribed to the notion, often attributed to President Rafael Reif at MIT, that four-year residential education might someday be replaced with three years of residential courses and one year of online learning. Overall, the need for flexibility of educational options was often repeated, as typified by Royal at Inver Hills Community College: “We should have a series of different educational environments that suit the diversity of our student base.” Most interviewees believed that higher education will be increasingly pressured to shift towards competency-based models of credentialing. Several foresaw the possibility that the emphasis on earning a degree will dwindle in favor of an emphasis on demonstrating capabilities needed by employers. As Kirschner at Macaulay Honors College asserted, the burden of showing what a student can do will “shift away from the degree as a proxy” because for some employers, higher education has “failed to make the degree stand for something.” Concomitantly, Cator of Digital Promise favors a shift in federal funding for postsecondary education away from simply paying for access and towards paying for the demonstration of competency and attainment of credentials: “the incentives then would be all around performing, producing, learning, rather than around getting into a class, going to college, or not going.” Smith, of Straighterline, carries this vision even further to a future in which federal funding for college is replaced by life-long education accounts that individuals can use to select their own educational Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

152

experiences, provided each one meets minimum standards of accreditation (see Smith, 2013). Minghua Li, Professor, School of Public Administration at East China Normal University, believes that the major online platform providers should be collaborating to create a course accreditation system and could subsequently pressure the federal government into allowing the application of financial aid to pay for online courses offered outside of college degrees, including MOOCs. A few interviewees conjectured that, as more education is offered online, some institutions of higher education could shrink and disappear while virtual universities, or virtual extensions of established institutions, expand. However, many interviewees doubted that MOOCs would dramatically affect brickand-mortar colleges and universities. Most interviewees expected to see changes in pedagogical strategies, for example, more flipped classrooms and greater emphasis on peer learning. Increased use of both gamification and game-based learning strategies is expected in some content areas to “hook” students through constant feedback, challenge, and creation of a social community. For some interviewees, the social community aspect is one of the most powerful that can be developed in online environments. Parker of NextThought envisions “tech-enabled learning communities that will persist beyond the university experience providing individuals with a life-long network of peers across time and space.” One concern raised by Newfield at the University of California, Santa Barbara, among others, is that “the genie of free, quality education is out of the bottle”, that is, the expectation that high quality educational experiences can be delivered at negligible cost has caught on, despite the lack of evidence to show whether MOOCs and similar experiences offered at scale can substitute some part of face-to-face education. Newfield worries that well-intentioned politicians and business leaders will continue to press for degrees costing $10,000 or less, and will fail to invest enough in public education. Silverman at NYIT fears that to combat escalating costs, institutions will ever more aggressively replace tenured professors with talented teachers to deliver the curriculum, but these instructors will not participate in the continuous research necessary to constantly push academic disciplines forward. MOOCs and the future We categorized the assortment of ideas offered by interviewees regarding MOOCs in the future of education into three general topic areas: 1) those that describe how the courses themselves might evolve from current MOOCs; 2) ideas that capture how MOOCs may affect aspects of the current economic model of higher education; and 3) ideas about how MOOCs can facilitate relationships among institutions of higher education, and between these institutions and other parties. We summarize the ideas in Table 13 and loosely indicate the frequency with which each idea arose. The remainder of this section provides additional detail on each idea.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

153

Table 13: Ideas Raised by Interviewees about MOOCs in the Future of Education Ideas Raised by Interviewees about MOOCs in the Future of Education

Mentions among 83 interviewees

How MOOCs will evolve as courses MOOCs will serve as educational resources rather than as stand-alone courses

Many

MOOCs will be targeted at specific audiences

Many

MOOCs will offer revenue-generating services such as tutoring, face-to-face interactions, and study groups

Many

MOOCs will offer credentials of economic value such as high school or college credits, or badges that are recognized by employers

Some

Learning analytics data will catalyze the advent of personalized, adaptive, and masterybased learning

A few

Convergence of cMOOC and xMOOC pedagogy

A few

Continuous A/B testing will allow for iterative improvements in materials and activities

A few

MOOCs will incorporate computer grading of open-ended assignments

A few

MOOCs will be available on mobile devices

A couple

How MOOCs could affect the economic model of higher education Students will cobble together their own certifications

Many

Meeting demand for higher education in rapidly developing countries

A few

Faculty members as free agents

A couple

MOOCs-Inside platform and course markets

A couple

Replacing the course catalog and course â&#x20AC;&#x153;shoppingâ&#x20AC;?

One

How MOOCs can facilitate relationships among institutions Cross-institutional collaborations

Many

Life-long connections to alumni

A few

Community outreach

A couple

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

154

I. How MOOCs will Evolve as Courses Starting with Koh at Coursera, all interviewees expected that MOOCs and MOOC platforms, assuming they still exist, will look vastly different and more sophisticated in five years’ time. While not everyone was certain that MOOCs would persist, many expected them to endure either as stand-alone learning experiences or as the online portion of the increasingly popular flipped classroom model. MOOCs were expected to evolve in a number of ways and more than one cynical interviewee observed that this evolution could lead MOOCs to more closely resemble regular online courses of today. MOOCs will serve as educational resources rather than as stand-alone courses Many interviewees expressed a sentiment similar to that of Dodds at Cornell University that MOOCs will “evolve into high-end, multi-media textbook replacements.” Lester at FutureLearn envisions the possibility of a tagging system within MOOC materials to allow students and faculty members to search for relevant items. We could tag things within courses. That would probably take quite a bit of work, but if it was planned up front so it would just build up, then it could be done. In which case, you open up the fact that MOOCs are being used as resources, much as a textbook would, rather than as a course. (Lester, FutureLearn). MOOCs will be targeted at specific audiences Many interviewees perceived that a weakness of current MOOCs is their need to present content and activities that are accessible to participants with a wide range of incoming knowledge and skills. These interviewees generally felt that MOOC pedagogy and completion rates could be improved by targeting specific audiences: what Silverman at NYIT terms “narrowcasting” in contrast to the “broadcasting” he associates with most MOOCs. While this may reduce the number of participants, it is more likely that those who enroll will be active in the course. Examples of such targeting included MOOCs for K-12 students, MOOCs for teacher professional development, and MOOCs for corporate training. MOOCs in K-12: Kinzler from AMNH believes that “There is a lot of potential for digital content to be the direct instruction component for K-12,” thereby allowing teachers to “…be more facilitators of meaningmaking than deliverers of content.” A few participants suggested that MOOCs might help high school students who do not have access to teachers in less common subjects such as chemistry, physics, or a foreign language. However, as noted earlier in this report, the prospect of MOOCs in K-12 raises a number of policy challenges. For example, public funds are currently apportioned to schools on a perstudent basis and do not follow a student taking courses offered by other institutions. To avoid some of the issues endemic to the K-12 regulatory environment, Paska at Harrison Central School District suggested that MOOCs could provide credit towards an Advanced Placement (AP) degree or towards college. He speculated that Boards of Cooperative Educational Services (BOCES), or similar organizations in other states, might be able to facilitate the introduction of MOOCs by using existing Cooperative Service agreements (CoSers) to share courses across district lines, and by soliciting entrepreneurial aid to support MOOC development. MOOCs for teacher professional development: Kleiman at North Carolina State University reports a high degree of interest from state-level officials regarding the possibility of using MOOCs, perhaps in a blended model of learning, to meet the professional development needs of teachers and administrators. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

155

This audience could well be expected to fit the mold of well-educated, motivated learners who might fare well in MOOCs, particularly if completion of a course is a job requirement and professional time is dedicated to it. MOOCs for corporate training: A number of interviewees observed that corporations have taken a fair amount of interest in MOOCs. Rizvi at Pearson alluded to a McKinsey survey-finding that many European employers find university graduates ill-equipped with the skills required for entry-level jobs and he sees a role for MOOCs in closing this gap. He suggested that traditional universities are not necessarily the best source of training on certain topics, for example, Google could provide MOOCs on programming and L’Oreal on marketing. Already, Rizvi noted, “the big industrialist families” in Pakistan and the Philippines that employ hundreds of thousands of people are establishing their own universities to help prepare graduates with the skills needed by these employers. Lester at FutureLearn similarly pointed out that MOOCs may be used to address some of the tensions that currently exist between business and universities: Universities have an interest in quality, governance, and reputation. It takes a very long time to produce new content while business and technology move very fast. Businesses want immediate responses; turnaround of content that is relevant and useful. Universities traditionally haven’t been very good at serving those markets. There is a market failure there. (Lester, FutureLearn). Lester noted that Google and Apple have already built their own course platforms and he pointed to the MOOCs created by Enterasys Networks as a potential model for technical training in the future. The possibility that MOOCs will be used for corporate training purposes was also raised by other interviewees. Many pointed to the announcement in late 2013 by Udacity of a shift away from working with institutions of higher education in favor of corporations. Three of our interviewees from platform providers reported ongoing discussions with large corporations and, unusually, we encountered representatives of several corporations at the MOOC Research Initiative in December who attended to learn about MOOCs and explore their applicability for internal use, or for collaborative initiatives with institutions of higher education. Saxberg and Niemi at Kaplan Inc. described plans to develop a MOOC on cognitive task analysis. Beyond our own study sample, Radford et al. (2014) report that while only 7% of employers they surveyed in North Carolina had already used MOOCs for employee professional development, another 76% had either considered doing so or would be open to the idea. MOOCs will offer revenue-generating services such as tutoring, face-to-face interactions, and study groups There was consensus among our interviewees that MOOCs will not be indefinitely sustainable without some form of revenue or demonstrable impact on reducing the costs of education. Interviewees expected revenue-generating services to be offered to MOOC participants, such as online tutoring, opportunities for face-to-face interactions with a local instructor and a local cohort of participants, or placement in an online study group with a few carefully selected peers. Many interviewees believed that maximum benefit will be derived from MOOCs only if opportunities are provided for participants to work simultaneously face-to-face with an instructor. While localized meetups have been organized by the platform providers and institutions creating MOOCs, these interactions are only likely to be effective at improving the educational impact of MOOCs if students can follow up with some sustained educational activity, for example, establishing small, collaborative study groups. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

156

If I had to launch a MOOC today I would look for more ways to enhance the level of engagement of the course and look to create some distributed but localized gatherings like student meet-ups so the students could have a connection to rely on, encourage, and motivate them. There are very subtle ways in which group dynamics contribute to you feeling a sense of desire to continue and get things done because you are pacing with the rest of the class. (Royal, Inver Hills Community College). MOOCs will offer credentials of economic value such as high school or college credits, or badges that are recognized by employers Some interviewees expressed the view that the MOOC movement would soon fade out unless participants are able to earn meaningful credentials carrying economic value. Miles at Tri-C anticipates that, over the next few years, students will ask for credits for MOOCs completed and the college will have to figure out how to award credit based on competency. MOOCs may offer traditionally accepted credentials such as high school or college credits, and professional certifications, or newer types of recognition such as virtual badges. While a small number of MOOC participants are already paying for verified certificates of completion, employers are likely to want confirmation that job candidates can actually apply skills and knowledge in practice, perhaps even using secured online environments to test these during the job interview process. The concept of badging … provides an opportunity to link business and industry. I think both business and educational institutions have concluded that there is a need for better connections between employer expectations and student preparation, and, to me, badging could be another linkage to help reduce the mismatch of expectations and needs. So if an employer can see who has badges in specific competencies that the employer is seeking, that would be helpful. (Royal, Inver Hills Community College). The whole issue of credentialing is going to be critical in this…The real test of whatever system we are going to have for credentialing is what the employers who hire the individuals with the certification take as markers for people that are adequately prepared to contribute value to their businesses. That is going to be the measure. Who knows, it may be that large corporations say that students trained under the new models are equally as productive as the traditional models, then I can imagine that we are looking at real, substantive transformation of higher education. (Professor, East Coast University). Radford et al. (2014) report that while only 31% of 103 employers surveyed in North Carolina had even heard of MOOCs and only one had used a MOOC provider’s recommendation for recruitment purposes, 73% were favorably inclined towards considering completion of MOOCs in hiring decisions: “…while traditional degrees and credentials were still necessary qualifications for jobs, respondents tended to view MOOC course taking as an indication of greater motivation” (p.2). This finding bodes well for the future of MOOCs. Learning analytics data will catalyze the advent of personalized, adaptive, and masterybased learning A few interviewees expect that truly adaptive learning will indeed materialize over the next few years so that the MOOC experience can be tailored to each individual participant. Laster at McGraw-Hill Education observes that adaptive learning solutions are not simple or inexpensive to develop as each Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

157

course must be developed with the adaptive learning strategies built in. This may require working with a third-party provider of adaptive learning solutions or developing the capacity in-house. In either event, the process requires an investment of time from subject-matter experts, instructional designers, instructional technologists, data analysts, and computer programmers. As these constituencies are already being represented in many MOOC production teams, it is likely that some of these teams will progress towards tackling the adaptive learning challenge over the next few years. To provide learning on an adaptive and personalized level, the capacity for MOOC platforms to track and report individual level data must first be increased. As Whitmer suggested “the next frontier will be looking for patterns in learner behavior over time and over multiple courses, and looking for relationships between activity and measures of achievement.” Renick, who is well versed with the use of student data and predictive analytics at Georgia State University, believes that MOOCs could be helpful in the university’s ongoing quest to improve course completion and graduation rates “if they can provide really granular understanding of how students are performing long before final grades are submitted.” Convergence of cMOOC and xMOOC pedagogy From a pedagogical perspective, MOOCs of the future are expected to converge between the current extremes of connectivist MOOCs, which encourage creation of artifacts and networked learning, and xMOOCs which are more about transmission of knowledge and what educators might term “drill and practice.” Several interviewees anticipated that course designers will adopt some combination of the social aspects of cMOOCs and the structured presentation of content and activities typical of xMOOCs. A key aspect of this shift will involve platform design that facilitates peer learning and enables more efficient collaborative work. As reported under Goal 6: Conducting Research on Teaching and Learning, Kulkarni and others at Stanford are already investigating the optimal composition of study groups. A small number of interviewees envisioned the capability for MOOC platforms to identify student characteristics and to recommend productive combinations of students for online study groups. Dede at Harvard anticipates delivering a “massive learning experience” that incorporates “an automated pattern-matching tool that creates teams that have complementary knowledge and skill, based on the surveys students take before the course.” The objective of the course “on peer learning and by peer learning” will be to help participants learn how to build social capital and develop other non-cognitive skills such as collaboration, metacognition, and tenacity. Ideally, the experience will render participants more effective at peer learning so that they can maximize the benefits from future participation in learning experiences such as MOOCs. Insights obtained from this larger audience may be adapted to improve peer-learning on-campus. Parker similarly expects future versions of the NextThought platform to allow formation of small groups guided by student attributes. Experience and research will help identify the best predictors of individual and group success. Continuous A/B testing will allow for iterative improvements in materials and activities While the Coursera platform currently facilitates A/B testing (see Box 5) by allowing delivery of different e-mails to participants at random, several researchers we interviewed did not expect it would be long before A/B testing will be possible directly within the Coursera platform and others. This would allow different materials or activities to be presented to different students, and an investigation of whether these variations affect student behavior and performance. If it appears that a particular variation has led to an improvement in participant persistence or performance, future iterations of the course could Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

158

incorporate the relevant feature. This presumes that there is both funding available and institutional will to keep perfecting a MOOC once it has been developed. Lester of FutureLearn also envisions the use of A/B testing to fine-tune instructional materials, but he acknowledges that unless it can be demonstrated that learning outcomes have been improved, such efforts would be in vain. We will do A/B testing and figure out, "This video, or videos with a human face on it, actually do far better than the videos without a human face," and so on. We will constantly learn and find out what the students want to do, and what they engage best with. (Lester, FutureLearn). MOOCs will incorporate computer grading of open-ended assignments While much of the current focus with respect to grading of MOOC assignments is on improving peergrading strategies, a few interviewees commented on expected developments to improve the capacity for computer-based grading of open-ended assignments such as essays. Depending on the instructor’s subject area, this development was more welcome or less so. Some strongly argued that computers are incapable of capturing the nuances an experienced instructor would perceive, while others, including Baker at Teachers College, were more sanguine. MOOCs will be available on mobile devices Dillenbourg et al., (Eds., 2014) describe ways in which engagement in MOOCs via mobile devices can improve “information access, production and collection of data by learners, and communication.” Mobile access to online course content is already available to some extent from some providers such as 2U (see Press Release) and Coursera. NextThought allows for access to MOOC course content via iPad tablets (see Case 8) and offers a mobile notification strategy such that if a participant initiates a thread or asks a question, he or she immediately receives an e-mail or instant message when a response is posted. A single click takes the participant back to the online dialogue. CEO Parker is confident that mobile devices will increasingly supplant paper and even laptops, and he expects that a fully interactive MOOC experience will soon be supported at least on Android tablets and smartphones. It is likely that other platform providers will follow suit. Mobile allows for omnipresence and immediacy. Dialogues fade out when too much time elapses between the interactions, that’s why the social networking sites have invested hugely in mobile strategies. Mobile devices will play a key role in connecting people. (Parker, NextThought).

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

159

II. How MOOCs could Affect the Economic Model of Higher Education While interviewees considered it highly unlikely that MOOCs will provide a complete cure for higher education’s “cost disease” (see Bowen, 2012), many identified ways in which MOOCs may affect the current economic model of higher education, both in the U.S. and abroad. Several interviewees speculated that campus-level versions of commonly offered introductory courses such as Statistics 101 or Economics 101 could be each replaced by a single “master” course, in some cases a MOOC, that could be supported locally by non-tenured instructors or teaching assistants. Kalbaugh at Wake Technical Community College believes that MOOCs may more cost-effectively serve the role previously played by community colleges of providing low-cost, low-risk, and life-long learning opportunities to non-degree seeking adults. But Kalbaugh, and several other community college administrators, do not see MOOCs serving their core, degree-seeking student body more effectively than current course offerings. Students will cobble together their own certifications Perhaps the most common theme we heard regarding the future of MOOCs is the role they would play as one of many choices available to students who will increasingly be able to pick and choose their own educational experiences, which may or may not include a traditional college degree. Fisher at Vanderbilt University observed that it is already possible to take all the necessary courses for a computer science degree through MOOCs (see Fisher & Fox, 2014, Figure 1, p.4), with only the lack of credits preventing anyone from actually earning a degree this way. He foresees the possibility that students will be able to design their own majors in MOOCspace, provided there is some means of accreditation. Kirschner at Macaulay Honors College predicts that the burden of responsibility for demonstrating a person’s capabilities will shift away from educating institutions to the students themselves: …this is part of a larger shift towards self-credentialing where the student is in control of saying “This is what I know how to do.” For example, “I can do statistics or analyze your data and here is my evidence.” The student is going to have to aggregate learning experiences of certain types. Some of these will be certified, or the students will assemble an e-portfolio that stands as evidence of what they can do. I don’t think the one-size-fits-all B.A. credential will stand as the dominant form of credentialing. Where the revolution is going to come first is with the smartest, best-prepared kids, not the least-prepared students. There’s always going to be 10,000 applicants lined up to go to Harvard, but will they continue to be the smartest 10,000 applicants? More and more of these students may say: “I had a really good high school education and I’d like the partying, hanging out, and so on, but my family can’t afford it. So do I need a four-year degree at enormous cost, or should I take one year at college, one year studying abroad, and fill in with MOOCs and internships?” (Kirschner, CUNY). To facilitate such a change, Smith at Straighterline believes that students should be able to choose from a menu of options and apply public funding to pay for their choices. I would look at all the subsidies that higher ed enjoys from the federal government, directly in terms of loan subsidies, or tax credits, or direct state support, or nonprofit status, all these different ways that higher ed is subsidized. They all add up. There are half a dozen subsidies, so it's a pretty enormous number. Take that, and set some minimum baseline standards, and allocate the rest in some form of educational, or training, or life-time educational account that people can access once they turn, say, 18. You can adjust it for different socioeconomic Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

160

categories. Let people make their own training decisions with those dollars. That’s the kind of model I'd like to see…something more like vouchers or food stamps, as opposed to this kind of patchwork subsidy structure that we currently have. (Smith, Straighterline). Straighterline already offers what may become a more popular model of higher education in the future. Students can sign up for $99 per month to take as many or as few courses as they wish. Services include online tutoring provided by a Pearson-owned company, Smart Thinking; student advising including technical support; proctoring services from Proctor U; and free transcripting. All Straighterline courses carry ACE credit recommendations. In addition, Straighterline has negotiated formal articulation agreements with 65 institutions to guarantee credit for Straighterline courses. Smith notes that an equity issue may arise because the most motivated students require the least amount of support, and are therefore the cheapest to serve, while less motivated students need the most support services. If stronger students do not pay for services they are not using, there is no subsidy for the weaker students who do need these services. The weaker students could end up paying more to earn the same credential, providing yet another disincentive to education. Downes at the National Research Council is working towards the development of advanced personal learning environments (PLEs) through the Learning and Performance Support System project. The goal is to allow each individual learner to set up his or her own network of resources independently of any one provider. Each learner’s PLE would link to a variety of learning resources offered by multiple providers, including MOOC platform providers, corporate trainers, commercial education companies, free-lance instructors, and traditional institutions of higher education. Not everyone was as enamored of this idea, and some expressed doubt that the majority of students could be self-directed enough to take advantage of such freedom of choice. For example, Paska noted: One problem I see with the possibility of earning a MOOC degree is how do I know what I don’t know? Students don’t always know what they need and often they need a sequence of instruction to obtain the requisite knowledge and skills. (Paska, Harrison Central School District). This, of course, leaves the door open to online advisory services that can help guide students through a maze of online learning opportunities to construct a coherent set of credentials. Kirschner at Macaulay Honors College believes that there is much to be done to help learners navigate the rapidly growing choices: Who will be the Amazon of online learning? If students are going to be cobbling a little bit from here and from there, who is going to rank, for instance, the available stats courses? If I want a job at a pharmaceutical company and am interested in stats, I should be able to go to somewhere and see the training and courses needed for this job and, of all online stats courses, some designation that these are the ones that most people in those jobs say are the best. There are companies already looking into this, like Balloon, SkilledUp, and Lynda.com. That’s the lens into education that I think is so revolutionary. (Kirschner, Macaulay Honors College).

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

161

Meeting demand for higher education in rapidly developing countries A few interviewees expect that MOOCs could help address the shortage of universities and faculty members in developing countries. Peercy at the University of Wisconsin-Madison observed a drastic shortfall in qualified faculty members abroad to meet the rising demand for engineering degrees and believes that MOOCs could play a significant role in addressing this concern. Similarly, Mintz at the University of Texas System envisions specially developed MOOCs to fulfill the unmet need for higher education in developing countries. In the developing countries they cannot create universities fast enough to meet the need. They face all kinds of challenges. In India, currently ten percent of college-aged students enter college. Clearly that needs to rise, but how can they do that in an affordable manner? The answer seems to me new kinds of MOOCs, or some kind of hybrid delivery using MOOCs. I'm hoping that edX will move into that area. Not targeting high-end courses to autodidacts and degree holders, but standard undergraduate classes. They're going to have to develop a model to incentivize campuses to create those courses for free. Probably what will have to happen is some foundation will have to pay for those courses. They would have to be developed in some collaborative fashion with the country that would consume them. It would also have to be in the indigenous language. It may not be done by the most famous faculty members, and it may be done through the Education College or the Teachers College. What I envision is edX identifying courses that need to be produced, running a competition in which everybody would be invited to participate, and then choosing the best development team that it could find that's most likely to produce what we really want. (Mintz, University of Texas System) One of the most interesting proposals we heard for using MOOCs to provide higher education in developing countries was University of Colorado Anschutz Medical Campusâ&#x20AC;&#x2122;s idea to offer a MOOC for global health responders that, for students who complete the MOOC successfully, would be supplemented with a two-day, face-to-face, hands-on certification course, designed by University of Colorado faculty members and delivered at locations around the globe. Faculty members as free agents A professor at an East Coast university described a future in which the massive data from MOOCs could be used to identify particularly effective teachers who might then act more like free agents: Let me just describe a possible path of evolution. I think the final end point is faculty free agency. As I think about what the technology might be able to do, the model I tell people is to think about baseball and free agency. The reason it is relevant is there is a reality that some teachers are just better teachers than others. Because there is little in the way of data by which one can compare, there is little commonly agreed upon science around this. [With MOOC data] you might be able to identify first a set of individuals who are far more effective at getting learners to a high level of competency in a given time, the equivalent of sports superstars. Once you identify those individuals, we can work on what is it that they are doing in their classrooms that makes them so effective. Then we can train other people to do it. It is completely possible that at some point these faculty members will not work for a university. May be some people are so good at this that a commercial entity might say that Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

162

instead of working for the university, come work for us…Using the power of the market to identify extraordinarily successful individuals is exciting. (Professor, East Coast University). Parts of this vision are already appearing at Straighterline, where qualified instructors from any institution can apply to provide supplementary and support services to students enrolled in Straighterline’s online courses. According to Straighterline’s CEO, Smith, these faculty members can earn $50-$100 per student for providing these extra services. While there may be limits to how many students any one instructor can effectively support, if faculty members could earn royalties based on the number of students they attract, there may well be outsize earnings accruing to master teachers in the future. MOOCs-Inside platform and course markets Minghua Li of East China Normal University envisions a future which encompasses a number of ideas both about how MOOCs will change and how higher education will change. He expects that, in the future, mainstream higher education courses will be hybrids of MOOCs with local face-to-face interactions in a classroom. He proposes the idea of a “MOOCs-Inside platform” that will facilitate the development of an accredited independent course market operating in parallel to the current “degree market.” Instructors will not need to be affiliated with a university, but only with one or more platforms through which he or she will offer courses and be connected to students. Minghua Li foresees that some institutions will primarily produce MOOCs while other will “consume” them. Some institutions will attract fewer residential students and can rent out underutilized facilities to local instructors hired by the MOOC-producing institutions, or to free-lance instructors associated with the platform providers. Reynolds from Technology Crossover Ventures similarly predicts a future of course markets rather than degree markets, comparing the shift with that of music from CDs to iTunes. Replacing the course catalog and course “shopping” Sreenivasan, formerly Chief Digital Officer and Professor at Columbia University, noted the positive impact of Coursera’s requirement that MOOC materials be ready at least two weeks in advance of delivery. He believes that the customary, but fraught, shopping period at the start of each semester can be replaced by a course catalog composed of MOOC-style introductory videos to help students preview and select courses. My goal is to have 3,200 one-hour previews or trailers of each of our courses. The students live in a preview culture: before they go to a movie they see a preview, before they buy a book they read a chapter. Why is this only thing in their lives where they don't have a preview? (Sreenivasan, formerly at Columbia University).

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

163

III. How MOOCs can Facilitate Relationships among Institutions MOOCs appear to have presented numerous opportunities for colleges and universities to engage with outside entities, including other institutions of higher education, corporations, funding agencies, local and distant communities, and alumni. Many interviewees expected more cross-institutional collaboration to share resources and expertise, and to spread costs that could not be borne by a single institution. Cross-institutional collaborations A number of cross-institutional collaborations have already been formed to offer online courses, including MOOCs, and several interviewees expect to see this trend grow. Griffiths at Ithaka S+R pointed to the Semester Online initiative facilitated by 2U and suggested that similar consortia could soon offer MOOCs that would be accepted for credit by partner institutions. However, the ephemeral life of Semester Online demonstrated that such collaborations among institutions of higher education are not always easy to negotiate and sustain (Straumsheim, 2014). According to another interviewee, some of the fifteen Committee on Institutional Cooperation (CIC) state universities are considering extending their “coopetition” arrangement that originated with sharing computer hardware and other resources, to pool funds for the development of MOOCs. Nominations would be submitted to the fund for MOOC ideas and some of these MOOCs could be jointly developed across a few universities in the consortium. The goal would be to attract applicants to full courses or programs offered by the universities. Our interviewee suggested that these MOOCs could be integrated with admissions procedures, for example, by encouraging top performers to apply to programs offered by CIC universities. He also suggested that reviewing participant persistence in one or more MOOCs could help in making financial aid decisions to help avoid the losses incurred when aid is awarded to students who subsequently drop out. Mintz at the University of Texas System highlighted the example of an initiative organized by HASTAC which involved Cathy Davidson’s MOOC History and Future of (Mostly) Higher Education being offered concurrently with on-campus courses led by Davidson at Duke University, Palumbo-Liu at Stanford and Newfield at the University of California, Santa Barbara. You have a face-to-face intimate encounter taking place within a context of global connectivity and featuring the biggest names from around the world who've thought about the future of higher ed. It's going to be through Coursera, and HASTAC is the organizer. To me this is tremendously exciting, and a potential model for really interesting next-generation crossinstitutional collaborations. (Mintz, University of Texas System). Fisher at Vanderbilt University is excited about the possibilities for what he calls “crowd-sourced curriculum” whereby one institution develops a MOOC and another follows it with a complementary, sequenced offering: We are going to see more collaborative agreements between MOOC providers and more interactions between MOOC courses. For example, we are currently working on a sequence. University of Maryland will offer a MOOC and we will follow that up with another MOOC. It is not quite a prerequisite, but if you want to take two sequenced MOOCs, they are coordinated Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

164

that way. We are going to see more of that. We are going to see various kinds of interactions between MOOCs. (Fisher, Vanderbilt University). In late 2013, the State of California announced a $57 million Online Education Initiative to develop online courses that could be used across all 112 community colleges in the state (see Raths, 2014). The goal of the initiative is to increase the number of students who earn a two-year college degree and subsequently transfer to four-year colleges. While it is not clear how closely any of these courses will resemble MOOCs, it is likely that shared courses will need to serve very large numbers of students. One of our interviewees described the potential for this initiative: …we can imagine that all 112 community colleges in California work together to orchestrate a consistent course management system and course schedule portal where any community college student can enroll in any of the available courses developed by different community colleges, but they would have a consistent learning experience. The coordinated effort would allow provision of 24/7 technology support and online tutors and advisors, which none of us could afford individually, and these courses would all be open access and available for credit. (Online learning coordinator, Large California Community College). Royal, at Inver Hills Community College, cautions that collaborations are never easy and are often born of necessity. …partnerships take a lot of work, there are differing beliefs about what the outcomes should be for the course and therefore what the lessons should be, and faculty don’t always like to give up their autonomy. It is hard to negotiate expectations. But I think more institutions will be forced to partner as financial resources become scarcer. For example, if someone wants to do a MOOC but doesn’t have enough money, either they partner or they cannot be in the game. When people don’t have the money to do things, that makes them creative. Money can be a driver into partnerships. (Royal, Inver Hills Community College). Collaborations are also likely to form around efforts to develop online platforms in addition to existing consortia such as edX. Collier at Stanford University anticipates that the need for third-party platform providers to generate revenues will result in greater conflict between these platform providers and faculty members. With many faculty members wanting to share their content for free with other universities, their goals will increasingly diverge from those of the platform providers, driving them to seek out other like-minded entities with whom to partner. Peercy at the University of Wisconsin-Madison is a strong proponent of globalized standards for engineering education and cross-border collaboration to deal with global problems of water, energy, pollution and resource distribution. With their international audiences, Peercy anticipates that MOOCs may allow for de facto standardization and could offer opportunities for multi-national efforts to address such global issues. Alumni Relations Several interviewees suggested that MOOCs can help alumni reconnect with their alma maters and with one another. A goal of the Social Media cMOOC created by the Department of Communication at the University of West Virginia was to provide continuing education for alumni. Sreenivasan, formerly at Columbia University, also mentioned “retraining opportunities for alumni” as a goal for Columbia’s Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

165

MOOCs. Fisher at Vanderbilt University perceives an opportunity for MOOCs to strengthen connections with alumni: They are not paying just for four years but for a lifetime of learning possibilities that Vanderbilt can offer them. Maybe we charge other people for MOOCs but not our alumni, or we provide them other affordances like on-campus experiences as part of the MOOC. MOOCs could change this relationship and our definition of alumni: they would never stop being our students. (Fisher, Vanderbilt University). An evaluator at a large Midwestern university noted that using MOOCs to connect with alumni could help address sustainability of MOOC costs if ongoing participation in these courses encourages the alumni to donate more, either directly to the MOOC initiatives, or more generally to the institution. Community Outreach Several interviewees mentioned opportunities for MOOCs to be used in the future to strengthen connections with surrounding communities. For example, Fisher at Vanderbilt University noted that many of Vanderbiltâ&#x20AC;&#x2122;s faculty members from the school of education already work with teachers in surrounding K-12 classrooms. Fisher envisages that these connections could be leveraged to implement a closed instance of a MOOC to improve the level of computer science education for the K-12 students. Vanderbilt University undergraduates could visit the classrooms on a weekly basis as â&#x20AC;&#x153;journey instructorsâ&#x20AC;? to supplement the online learning.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

166

Conclusions and Recommendations It is widely felt that higher education is in the midst of a significant change (e.g., Bowen, 2013; Barber, Donnelly, & Rizvi, 2013) and must accommodate budget shrinkages and the changing needs of learners and employers. High expectations abound for the role MOOCs can play in this transformation. This paper identifies six major goals that institutions hope to achieve through MOOC initiatives. The first goal is to extend reach of the institution to a wider audience and improve access to education. Data from MOOC platforms indicate that this goal has been met to some extent in terms of geographical spread, but less so in terms of reaching individuals with fewer educational opportunities. Participants are mostly limited to highly motivated learners who have access to high bandwidth Internet connections and, perhaps not surprisingly, the majority has already earned at least a bachelorâ&#x20AC;&#x2122;s or a masterâ&#x20AC;&#x2122;s degree, or even a doctorate. These may be new audiences for many existing college and university courses and programs, but it appears that MOOCs are mostly educating the educated and are therefore increasing the divide between those who have access to education and those who do not. It may be more accurate to state that they are extending access to life-long education rather than making education accessible to a broad group of people. On the other hand, this educated audience is also more likely to be able to pay for certificates and other services, allowing the production and delivery of MOOCs to be financially sustainable. While many institutions express the desire to improve access to education, and this is indubitably a laudable goal, there is little real incentive or financial justification for institutions to do so, except for those who can pay for it. Going forward, if institutions genuinely wish to broaden access to less educated audiences, they must identify multiple channels of communication to reach potential recruits. For example, they may need to use social media networks and advertise through high schools, employment agencies, or community organizations in the U.S. and abroad. Additionally, courses must be designed to serve less self-directed learners by incorporating motivational features and optional instructional scaffolding to address differences in participant preparation levels. However, as higher education institutions can ill afford to offer free and unsubsidized educational opportunities to the world, the costs of developing MOOCs will need to be covered by some fraction of participants who have adequate resources. The second goal is to build and maintain brand. If success in achieving this goal was measured purely by media publicity surrounding MOOCs, many institutions have indeed become more visible publicly as a result of their MOOC activities. Mostly this has been positive, but not always. Although several institutions suggested anecdotally that this goal has been at least partially accomplished, measuring success more tangibly could be achieved through comparing pre- and post-MOOC metrics on student recruitment and enrollment statistics, success in faculty recruitment and retention, breadth and generosity of donors, and quantity and size of grants. In addition, the costs of MOOC production and delivery should be compared to other strategies known to be effective at building and maintaining brand in order to ascertain which alternatives are most cost-effective. Typical of any early-stage, technology-based innovation, achievement of the third goal, to improve the economics of higher education, is still elusive for the most part. We found scant evidence that MOOCs have increased revenues substantially, and substantial evidence that they have increased costs significantly. Gradually materializing revenues could help shift the balance over the next few years but, unless MOOC producers and platform providers continue to expand ways to confer economic value on MOOC completion, for example, in the form of employer-recognized credentials, the market and Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

167

associated revenue streams will remain limited. One way or another, MOOCs will need to prove they can solve a problem or provide a service more efficiently than existing alternatives. We found that costs of developing MOOCs are very high and the process demands a great deal of personnel time and effort. Course design and delivery has shifted from a solo endeavor to team efforts including administrators in offices of digital technology, instructional designers, instructional technologists, videographers, and project managers. Faculty members themselves are, in most cases, being vastly underpaid for the opportunity costs of their time to develop MOOC content. While their primary intentions may be to improve opportunities for the educationally underserved, they are perhaps most effectively subsidizing the salaries of other employees at their own institutions or at the MOOC platform providers. Currently, there is little documentation to help ascertain whether costs of reoffering a MOOC diminish substantially as compared with costs of the initial offering. MOOCs could potentially reduce the costs of higher education if they can be used by academic institutions to eliminate the reproduction of similar courses across many campuses. If the licensing issues surrounding the sharing of materials across multiple campuses can be resolved, and if MOOCs can be re-run several times without significant adjustments at each offering, the high costs of initial MOOC development could be amortized over all the instances of use. But given that, in education, costs are mostly personnel-related, overall costs can only fall if personnel numbers or salaries fall first. Reduction of costs could be realized if academic institutions are willing to shift more of the responsibility for instructional support away from tenured faculty members and onto non-tenured instructors, teaching assistants, or outsourced personnel. Tenured faculty positions could not be easily eliminated in the short term, but they could be reduced over time, contributing to the current long-term trend towards â&#x20AC;&#x153;adjunctizationâ&#x20AC;? of higher education. This trend does not, however, bode well for the capacity of academic institutions to further various academic disciplines through research, nor is it likely to be welcomed by faculty members and their unions. For now, the picture being painted to keep the peace is one in which repetitive activities such as delivering the same lecture every year or grading uniform assignments are turned over to the MOOC, while faculty members spend time in the classroom with students on problem-solving, discussion, and debate. This approach may prove to be more educationally effective than current practice, but it will also be more expensive and therefore more financially unsustainable than an already unsustainable system. From the perspective of learners, MOOCs will only favorably impact the economics of obtaining an education if they remain free or charge minimal fees, but can be used to substitute costlier on-campus or online courses to obtain college credits or other credentials that will be recognized by employers. While this scenario may be beneficial to the students and to the taxpayer, it may not be in the interests of the academic institutions themselves as this will imply a loss of revenue per student, which may or may not be possible to make up by increasing enrollments. State and federal education policymakers could adjust regulations to create pathways for MOOCs to be accepted for credit in high schools or to satisfy government-mandated continuing education for professionals. In higher education it is less clear how accreditation agencies could be persuaded to move in this direction. Establishment of an accrediting organization for MOOCs and other non-traditional educational experiences would allow learners to accumulate a portfolio of credentials that serve as a viable supplement or alternative to a college degree. Furthermore, such accreditation of individual courses or other educational experiences should confer the ability to use public funds towards the costs of these credentials. Fain (2014) reports on two potential efforts in this direction and highlights some of the surrounding regulatory issues. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

168

Ironically, MOOCs have introduced possibilities that, should they materialize, could significantly compete with the very institutions that are racing to develop them. Many interviewees would readily assert that the fourth goal of improving educational outcomes has been achieved through MOOCs, at least for a limited group of learners. Primarily these positive reports arise in situations where MOOCs have been integrated with on-campus courses or where on-campus courses have been re-designed to incorporate MOOC-like components. Documentation of a small number of these experiments has shown early signs of improving student performance as measured by assessments and course completion rates. Typical strategies include flipping the classroom, implementing frequent assessments, and spending class time on problem-solving activities, often in small groups, rather than on lecture. Little effort has been made to ascertain whether participants in stand-alone MOOCs gain useful skills and knowledge that can be applied in productive, real-world contexts. To determine whether the goal of improving educational outcomes can indeed be achieved through MOOCs alone, will require rigorous evaluation employing pre- and post-assessments of knowledge and skills, and comparisons with the outcomes of face-to-face or other online courses. Educators must get beyond the notion that course completion per se equates with learning and consider how to allow learners to demonstrate valuable competencies, both cognitive and non-cognitive, that may be acquired through MOOC participation. Longitudinal studies tracking post-MOOC outcomes such as sequences of courses taken, professional certifications obtained, or job opportunities received would help assess the longer term value of participating in these courses. More effort is also required to assess whether MOOCs as stand-alone experiences can be designed effectively to increase knowledge and cultivate skills for any but the most motivated learners. Several interviewees noted that there is much to be learned from the existing research on best practices in online learning, and that MOOC producers have for the most part failed to take advantage of accumulated expertise in this area. A useful exercise would involve reviewing with these experts of online learning what pedagogical practices can be usefully migrated from regular online courses to MOOCs. The goal of innovation in teaching and learning has certainly been met at institutions where online learning did not previously exist, as evidenced by the sheer number of new courses produced and course re-design efforts undertaken. Metrics on the frequency of these efforts could be compared with pre-MOOC activity to provide an assessment of how well the goal of innovation has been achieved, but it is not clear that innovation per se is a sufficient aspiration. In many cases, institutions are innovating with a further goal in mind, such as improving educational outcomes or increasing revenues. Assessing impact of MOOC initiatives on these distal goals would be a more worthwhile endeavor in judging their contribution to higher education. However, some institutions are unclear as to why they are embarking on MOOC initiatives and until they can agree internally on suitable and realistic goals, they will struggle to justify the expense and effort. For institutions that have been offering online and hybrid courses for many years, MOOCs represent more of an incremental step along a pre-existing trajectory than a major innovation. Some even consider MOOCs to be regressive because they fail to incorporate best practices in distance learning and repeat many of the mistakes of earlier attempts to educate at scale. Finally, the goal of conducting research on teaching and learning has met with many obstacles, but a few well-funded institutions with strong programming capacity have not been deterred from efforts to use the â&#x20AC;&#x153;big dataâ&#x20AC;? from the course delivery platforms to document online learner behavior and to experiment with variations in pedagogical strategies. Regrettably, MOOCs are often serving as more of a Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

169

playground populated by captive participants within which researchers tinker with the artifacts rather than as an object of research to determine whether MOOCs are an effective and cost-effective means of educating a broad range of learners compared with existing models of education. A potentially useful application of the big data from MOOCs is to provide learner analytics. These can be used to iteratively improve courses or to catalyze the advent of high quality adaptive learning and personalized educational experiences which meet the needs of a variety of learners. It is apparent, however, that the useful application of data mining techniques to data from MOOC platforms could be facilitated by standardization of data formats across the various platforms. It is also clear that the high demand on resources to develop sophisticated adaptive learning mechanisms will require the establishment of working partnerships between educators, instructional designers, and programmers. Federal approval of cross-institutional IRB agreements could facilitate projects in which expertise and resources can be pooled. Additionally, clarifications regarding applicability of FERPA to MOOCs would alleviate current confusion over how to handle participant data. Given the considerable investment that MOOCs represent, we strongly recommend that prior to embarking on MOOC-related initiatives, institutions carefully consider their goals and whether MOOCs present a realistic and financially justifiable means to achieve them. Administrators at academic institutions should work with representative faculty members and, where relevant, their unions to establish a strategy for engagement and a process for production or adoption of MOOCs. Data collection protocols should be devised up front to measure relevant indicators both prior to and post MOOC engagement in order to permit an objective assessment of whether stated goals are being met. By tracking personnel time commitments with respect to MOOC-related efforts and accounting for displacement of other productive activities, such as teaching or grant writing, costs of such initiatives can be estimated and compared with the evidence of goal achievement to judge whether MOOCs are a worthwhile investment of limited educational resources. Based on what we heard from our interviewees regarding MOOCs of the future, we expect that MOOCs will evolve to more closely resemble regular online courses in that they will be targeted at specific audiences, will offer some form of credential, and will combine elements of both synchronous and asynchronous participation. While participation at some basic level may be offered free, it is likely that revenue-generating services will proliferate including verified certificates, employer recommendation services, online tutoring, peer study group selection and facilitation, face-to-face components with a local instructor, and more complex assessments that require human grading or personalized interaction. We expect convergence between the extremes of cMOOC and xMOOC pedagogy to combine the strengths of networked learning and “creation of artifacts” with structured progression through a series of activities designed to improve specific skills and knowledge. Much of the hype surrounding MOOCs may be subsiding but it is clear that the infrastructure and effort that has been poured into such initiatives are not likely to evaporate overnight. Whether MOOCs as they currently stand persist into the future is certainly debatable, but there is no doubt that online and hybrid learning are here to stay and that MOOCs have ignited many valuable inter-disciplinary and crossinstitutional discussions about how best to improve intellectual capital. There remains much work to be done to document how well MOOCs and related interventions serve their intended purposes and whether they do so cost-effectively.

 Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

170

References Acemoglu, D., Laibson, D., & List, J. A. (2014). Equalizing superstars: The Internet and the democratization of education. NBER Working Paper Series (19851). Retrieved from: http://economics.mit.edu/files/9452 AHEAD. (2014, April). Whatâ&#x20AC;&#x2122;s AHEAD key trends in education Poll #1: Massive Open Online Courses (MOOCs). Alliance for Higher Education and Democracy (AHEAD) at the University of Pennsylvania. Retrieved from: http://www.gse.upenn.edu/pdf/ahead/whats_ahead/01_moocs.pdf Allen, E., & Seaman, J. (2013). Changing course: Ten years of tracking online education in the United States. Babson Survey Research Group Report. Retrieved from: http://sloanconsortium.org/publications/survey/changing_course_2012 Allen, E., & Seaman, J. (2014). Grade change: Tracking online education in the United States. Babson Survey Research Group Report. Retrieved from: http://sloanconsortium.org/publications/survey/grade-change-2013 Anders., G. (2014, January 21). Coursera flirts with diplomas: online 'Specialization' is $250. Forbes Tech [Web log post]. Retrieved from: http://www.forbes.com/sites/georgeanders/2014/01/21/coursera-flirts-with-diplomas-onlinespecialization-is-250/ Argyris, C., & SchĂśn, D. (1978). Organizational learning: a theory of action perspective. Reading, MA: Addison-Wesley. Bacow, L.S., Bowen, W.G., Guthrie, K.M., Lack, K.A., & Long, M.P. (2012). Barriers to adoption of online learning systems in U.S. higher education. Ithaka S+R. Retrieved from: http://www.sr.ithaka.org/research-publications/barriers-adoption-online-learning-systems-ushigher-education Barber, M., Donnelly, K., & Rizvi, S. (2013). An avalanche is coming. Institute for Public Policy Research, London, UK. Retrieved from: http://www.ippr.org/publication/55/10432/an-avalanche-iscoming-higher-education-and-the-revolution-ahead Bates, A., & Sangra, A. (2011). Managing technology in higher education: strategies for transforming teaching and learning. San Francisco, CA: Jossey-Bass. BIS (2013). The maturing of the MOOC: Literature review of massive open online courses and other forms of online distance learning. BIS Research Paper, Vol. 130. Retrieved from: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/240193/131173-maturing-of-the-mooc.pdf Boddy, C., Detellier, C., Duarte, S., Dulpaa, E., Erdmer, A., Levasseur, D., McKay, M., & Ufholz, L. (2013). Report of the e-learning working group. University of Ottawa, Canada. Retrieved from: http://www.uottawa.ca/vr-etudes-academic/en/documents/e-learning-working-groupreport.pdf Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

171

Borkovitz, D. (2013, July 30). Summer MOOCs 2: Mechanics ReView. [Web log post]. Retrieved from: http://debraborkovitz.com/2013/07/summer-moocs-2-physics-review/ Bowen, W.G. (2012). The ‘Cost Disease’ in higher education: Is technology the answer? The Tanner Lectures at Stanford University. Retrieved from: http://www.ithaka.org/sites/default/files/files/ITHAKA-TheCostDiseaseinHigherEducation.pdf Bowen, W.G. (2013). Higher education in the digital age. Princeton, NJ: Princeton University Press. Bowen, W.G., Chingos, M.M., Lack, K.A., & Nygren, T.I. (2012). Interactive learning online at public universities: evidence from randomized trials. Ithaka S+R. Retrieved from: http://webcache.googleusercontent.com/search?q=cache:http://www.sr.ithaka.org/sites/defau lt/files/reports/sr-ithaka-interactive-learning-online-at-public-universities.pdf Breslow, L., Pritchard, D. E., DeBoer, J., Stump, G. S., Ho, A.D., & Seaton, D.T. (2013). Studying learning in the worldwide classroom: research into edX’s first MOOC. Research & Practice in Assessment 8, 13-25. Retrieved from: http://www.rpajournal.com/studying-learning-in-the-worldwideclassroom-research-into-edxs-first-mooc/ Bruff, D.O., Fisher, D.H., McEwen, K.E., & Smith, B.E. (2013). Wrapping a MOOC: Student perceptions of an experiment in blended learning. Journal of Online Teaching and Learning, 9 (2). Retrieved from: https://my.vanderbilt.edu/douglasfisher/files/2013/06/JOLTPaperFinal6-9-2013.pdf Buffardi, K., & Edwards, S.H. (2014). Introducing CodeWorkout: an adaptive and social learning environment. Proceedings of the 45th ACM Technical Symposium on Computer Science Education, 724. Abstract retrieved from: http://dl.acm.org/citation.cfm?doid=2538862.2544317 Cambre, J., Kulkarni, C., Bernstein, M.S., & Klemmer, S.R. (2014). Talkabout: small-group discussions in massive global classes. Retrieved from: https://hci.stanford.edu/publications/2014/PeerStudio/LAS2014-CambreTalkabout.pdf Carey, K. (2010, May 27). The better mouse-trap problem. The Chronicle of Higher Education. Retrieved from: http://chronicle.com/blogs/brainstorm/the-better-mousetrap-problem/24353 Carson, S. (2013, May 1). An MIT physics MOOC five years in the making. MIT News. Retrieved from: http://web.mit.edu/newsoffice/2013/mechanics-review-mitx-course.html Chafkin, M. (2013, November 14). Udacity’s Sebastian Thrun, godfather of free online education, changes course. Fast Company. Retrieved from: http://www.fastcompany.com/3021473/udacity-sebastian-thrun-uphill-climb Champaign, J., Fredericks, C., Colvin, K., Seaton, D., Liu, A. & Pritchard, D. (2014, March). Correlating skill and improvement in 2 MOOCs with a student’s time on task. Paper presented at Learning@Scale Conference, Atlanta, GA. Retrieved from: http://dx.doi.org/10.1145/2556325.2566250 Cheal, C. (2012, August 14). Creating MOOCs for College Credit (Research Bulletin). Louisville, CO: EDUCAUSE Center for Applied Research. Retrieved from http://www.educause.edu/ecar

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

172

Christensen, G., Steinmetz, A., Alcorn, B., Bennett A., Woods, D., & Emanuel , E.J. (2013). The MOOC phenomenon: Who takes massive open online courses and why? Working Paper. University of Pennsylvania. Retrieved from: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2350964 Cima, M.J. (2013). My experience teaching 3.091x. MIT Faculty Newsletter, 26(1), 15-17. Retrieved from: http://web.mit.edu/fnl/volume/261/cima.html Cima, M.J. (2014). A mastery-based learning and assessment model applied to 3.091r. (Introduction to Solid-State Chemistry). Internal MIT Report. Coetzee, D., Fox, A., Hearst, M.A., & Hartmann, B. (2014, February). Should your MOOC forum use a reputation system? Proceedings of the 17th ACM Conference on Computer-Supported Collaborative Work, Baltimore, MD. Retrieved from: http://dl.acm.org/citation.cfm?doid=2531602.2531657 Cormier, D., & Siemens, G. (2010). Through the open door: open courses as research, learning, and engagement. EDUCAUSE Review, 45(4), 30-39. Retrieved from: http://www.educause.edu/ero/article/through-open-door-open-courses-research-learning-andengagement Curley, N. (2014, January 27). Online education platform Coursera blocks students in Syria and Iran. Wamda [Web log Post]. Retrieved from: http://www.wamda.com/2014/01/coursera-blockssyria-and-iran-moocs-online-courses Daniel, J. (2012). Making sense of MOOCs: Musings in a maze of myth, paradox and possibility. Journal of Interactive Media in Education, 3. Retrieved from http://wwwjime.open.ac.uk/jime/article/viewArticle/2012-18/html DeBoer, J., Ho, A., Stump, G., & Breslow, L. (2014). Changing “course:” reconceptualizing educational variables for massive open online courses. Educational Researcher, 43(2), 74-84. doi: 10.3102/0013189X14523038. Dede, C. (Ed.). (2013). Connecting the dots: new technology-based models for postsecondary learning. EDUCAUSE Review, September/October 2013. Dernoncourt, F., Taylor, C., O’Reilly, U., Veeramachaneni, K., Wu, S., Do, C., & Halawa, S. (2013, December). MoocViz: a large scale, open access, collaborative, data analytics platform for MOOCs. Paper presented at NIPS Workshop on Data-Driven Education, Lake Tahoe, Nevada. Retrieved from: http://groups.csail.mit.edu/EVODesignOpt/groupWebSite/uploads/Site/MoocViz.pdf deWaard, I., Abajian, S., Gallagher, M. S., Hogue, R., Keskin, N., Koutropoulos, A., & Rodriguez, O. C. (2011). Using mLearning and MOOCs to understand chaos, emergence, and complexity in education. The International Review of Research in Open and Distance Learning, 12(7), 94-115. Retrieved from: http://www.irrodl.org/index.php/irrodl/article/view/1046/2043

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

173

Dillenbourg, P., Fox, A., Kirchner, C., Mitchell, J., & Wirsing, M. (Eds). (2014). Massive open online courses: current state and perspectives. Dagstuhl Manifestos. Schloss Dagstuhl - LeibnizZentrum fĂźr Informatik. Forthcoming at: http://www.dagstuhl.de/en/program/calendar/semhp/?semnr=14112 Downes, S. (2008). Places to go: connectivism & connective knowledge. Innovate, 5(1). Retrieved from: http://www.innovateonline.info/index.php?view=article&id=668 Dweck, C. (2008). Mindset: The new psychology of success. New York: Random House. Fain, P. (2013, January 1). Paying for proof. Inside Higher Ed. Retrieved from: http://www.insidehighered.com/news/2013/01/09/courseras-fee-based-course-option Fain, P. (2014, May 9). Ideas take shape for new accreditors aimed at emerging online providers. Inside Higher Ed. Retrieved from: http://www.insidehighered.com/news/2014/05/09/ideas-takeshape-new-accreditors-aimed-emerging-online-providers#sthash.TrrbNDAP.dpbs Figlio, D., Rush, M., & Yin, L. (2013). Experimental estimates of the effects of online instruction on student learning. Journal of Labor Economics, 31(4), 763-784. Firmin, R., Schiorring, E., Whitmer, J., Willett, T., & Sujitparapitaya, S. (2013). Preliminary summary SJSU+ Augmented Online Learning Environment pilot project. Retrieved from: http://www.sjsu.edu/chemistry/People/Faculty/Collins_Research_Page/AOLE Report September 10 2013 final.pdf Fisher, D. (in press). Leveraging AI teaching in the cloud for AI teaching on campus. AI Magazine. Vol. 35. Fisher, D., & Fox, A. (2014). Report on the CCC-CRA Workshop on Multidisciplinary Research for Online Education. Computing Research Association. Retrieved from: http://www.cra.org/ccc/files/docs/CCC-MROE-Report.pdf Fox, A. (2013). From MOOCs to SPOCs. Communications of the ACM. 56(12), 38-40. Retrieved from: http://cacm.acm.org/magazines/2013/12/169931-from-moocs-to-spocs/fulltext Ghadiri, K., Qayoumi, M.H., Junn, E., Hsu, P., & Sujitparapitaya, S. (2013). The transformative potential of blended learning using MIT edXâ&#x20AC;&#x2122;s 6.002x online MOOC content combined with student teambased learning in class. JUCE (Japanese Universities Association for Computer Education) Journal 2013, No.3. Griffiths, R. (2013). MOOCs in the classroom? Ithaka S+R. Retrieved from: http://www.sr.ithaka.org/sites/default/files/files/S-R_BriefingPaper_Moocs_20131028.pdf Grossman, S. (2013, June 26). Survey finds only limited public awareness of MOOCs. The Chronicle of Higher Education. Retrieved from: http://chronicle.com/blogs/wiredcampus/survey-finds-onlylimited-public-awareness-of-moocs/44549

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

174

Grover, S., Franz, P., Schneider, E., & Pea, R. (2013). The MOOC as distributed intelligence: dimensions of a framework & evaluation of MOOCs. Paper presented at the 10th International Conference on Computer Supported Collaborative Learning, Madison, U.S.A. Retrieved from: http://lytics.stanford.edu/wordpress/wp-content/uploads/2013/04/Framework-for-DesignEvaluation-of-MOOCs-Grover-Franz-Schneider-Pea_final.pdf Halawa, S., Greene, D., & Mitchell, J. (2014). Dropout prediction in MOOCs using learner activity features. Proceedings of the European MOOC Stakeholder Summit (EMOOCS 2014), Lausanne, Switzerland. Retrieved from: http://www.stanford.edu/~halawa/cgi-bin/files/emoocs2014.pdf Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage Publications. Ho, A. D., Reich, J., Nesterko, S. O., Seaton, D. T., Mullaney, T., Waldo, J., & Chuang, I. (2014). HarvardX and MITx: The first year of open online courses. HarvardX and MITx Working Paper No. 1. Retrieved from: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2381263 Hoxby, C. M. (2014). The economics of online postsecondary education: MOOCs, nonselective education, and highly selective education. NBER Working Paper 19816. Retrieved from: http://www.nber.org/papers/w19816 IMS Global Learning Consortium. (2013). Learning measurement for analytics whitepaper. Retrieved from: http://www.imsglobal.org/IMSLearningAnalyticsWP.pdf Ithaka S+R. (2013). Interim report: A collaborative effort to test MOOCs and other online learning platforms on campuses of the University System of Maryland. Retrieved from: http://www.sr.ithaka.org/sites/default/files/reports/S-R_Moocs_InterimReport_20131024.pdf Jaschik, S. (2013, August 19). Feminist anti-MOOC. Inside Higher Ed. Retrieved from: http://www.insidehighered.com/news/2013/08/19/feminist-professors-create-alternativemoocs Kelly, A.P., & Carey, K. (Eds.). (2013). Stretching the higher education dollar: how innovation can improve access, equity, and affordability. Cambridge, MA: Harvard Education Press. Kizilcec, R., Piech, C., & Schneider, E. (2013). Deconstructing disengagement: analyzing learner subpopulations in massive open online courses. Proceedings of the Third International Conference on Learning Analytics and Knowledge, pp.170â&#x20AC;&#x201C;179. Retrieved from: http://www.stanford.edu/~cpiech/bio/papers/deconstructingDisengagement.pdf Kizilcec, R. F., Schneider, E., Cohen, G. L., & McFarland, D. A. (2014, March). Encouraging forum participation in online courses with collectivist, individualist, and neutral motivational framings. eLearning Papers, 37, 13-22. ISSN: 1887-1542. Retrieved from: http://rene.kizilcec.com/wpcontent/uploads/2013/02/kizilcec2014encouraging2014elearning.pdf Kolowich, S. (2013a, April 29). Why some colleges are saying no to MOOC deals, at least for now. The Chronicle of Higher Education. Retrieved from: http://chronicle.com/article/Why-SomeColleges-Are-Saying/138863/ Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

175

Kolowich, S. (2013b, May 2). Why professors at San José State won't use a Harvard professor's MOOC. The Chronicle of Higher Education. Retrieved from: http://chronicle.com/article/WhyProfessors-at-San-José/138941/ Kolowich, S. (2013c, February 7). American Council on Education recommends 5 MOOCs for credit. The Chronicle of Higher Education. Retrieved from: https://chronicle.com/article/American-Councilon-Education/137155/ Kolowich, S. (2013d, November 25). A truce on the tech front at San José State. The Chronicle of Higher Education. Retrieved from: http://chronicle.com/article/A-Truce-Over-Technology/143229/ Kolowich, S. (2013e, February 21). How edX plans to earn, and share, revenue from its free online courses. The Chronicle of Higher Education. Retrieved from: http://chronicle.com/article/HowEdX-Plans-to-Earn-and/137433/ Kolowich. S. (2014, May 2). Coursera will offer certificates for sequences of MOOCs. The Chronicle of Higher Education. Retrieved from: http://chronicle.com/blogs/wiredcampus/coursera-will-offercertificates-for-sequences-of-moocs/49581 Kulkarni, C., Koh, P. W., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S.R. (2013). Peer and self-assessment in massive online classes. ACM Transactions on Computer-Human Interactions 9(4) Article 39, 31 pages. Retrieved from: http://www.cogsci.ucsd.edu/~mboyle/COGS1/readings/Klemmer-COGS1Peer%20and%20self%20assesment%20in%20massive%20online%20classes.pdf Lack, K. (2013). Current status of research on online learning in postsecondary education. Ithaka S+R. Retrieved from: http://www.sr.ithaka.org/sites/default/files/reports/ithaka-sr-online-learningpostsecondary-education-may2012.pdf Levin, H. M., & McEwan, P. J. (2001). Cost-effectiveness analysis: methods and applications (2nd ed.). Thousand Oaks, CA: Sage Publications. Lewin, T. (2013, December 10). After setbacks, online courses are rethought. New York Times. Retrieved from: http://www.nytimes.com/2013/12/11/us/after-setbacks-online-courses-arerethought.html?_r=0 Li, M. (2011). Nong Min Gong Gao Deng Jiao Yu Xu Qiu Gong Ji He Ren Zheng Zhi Du Yan Jiu [Research on Demand, Supply and Accreditation System of Higher Education for Migrant Workers in China]. Beijing, China: Yan Shi Press. Liyanagunawardena, T. R., Adams, A. A., & Williams, S. A. (2013). MOOCs: a systematic study of the published literature 2008-2012. The International Review of Research in Open and Distance Learning 14 (3), 202-227. Retrieved from: http://www.irrodl.org/index.php/irrodl/article/view/1455 Markoff, J, (2011, August 15). Virtual and artificial, but 58,000 want course. The New York Times. Retrieved from: http://www.nytimes.com/2011/08/16/science/16stanford.html

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

176

Masters, K. (2011). A brief guide to understanding MOOCs. The Internet Journal of Medical Education, 1(2). Retrieved from: http://archive.ispub.com/journal/the-internet-journal-of-medicaleducation/volume-1-number-2/a-brief-guide-to-understanding-moocs.html sthash.ankCZvLf.dpbs McGrory, K. (2013, November 7). More high school students in Florida are taking advantage of massive open online courses, or MOOCs. Miami Herald. Retrieved from: http://www.miamiherald.com/2013/11/07/3738068/more-high-school-students-in-florida.html Means, B., Bakia, M., & Murphy, R. (2014). Learning online: what research tells us about whether, when and how. New York, NY: Routledge. Miller, B. (2010). The course of innovation: using technology to transform higher education. Education Sector Report. Retrieved from: http://www.educationsector.org/usr_doc/NCATReport_RELEASE.pdf Moore, G. A. (2014). Crossing the chasm, 3rd Edition: marketing and selling disruptive products to mainstream customers. HarperCollins, NY. Nguyen, A., Piech, C., Huang, J., & Guibas, L. (2014). Codewebs: scalable homework search for massive open online programming courses. Proceedings of the 23rd International World Wide Web Conference, Seoul, Korea. Retrieved from: http://www.stanford.edu/~jhuang11/research/pubs/www14/nphg-www14.pdf Norton, A., Sonneman, J., & McGannon, C. (2013). The online evolution: when technology meets tradition in higher education. Grattan Institute, Australia. Retrieved from: http://grattan.edu.au/publications/reports/post/the-online-evolution-when-technology-meetstradition-in-higher-education/ Pawl, A., Barrantes, A., & Pritchard, D. (2009, July). Modeling applied to problem solving. Paper presented at Physics Education Research Conference, Ann Arbor, Michigan. Retrieved from http://www.compadre.org/Repository/document/ServeFile.cfm?ID=9899&DocID=1531 Piech, C., Huang, J., Chen, Z., Do, C., Ng, A., & Koller, D. (2013). Tuned models of peer assessment in MOOCs. Proceedings of The 6th International Conference on Educational Data Mining, Memphis, TN. Retrieved from: http://www.stanford.edu/~jhuang11/research/pubs/edm13/edm13.pdf Poster E. C., Mancini M., Ganji, D. (2013, August 5). MOOCs and more: Expanding online access for nurses. EDUCAUSE Review Online. Retrieved from: http://www.educause.edu/ero/article/moocs-and-more-expanding-online-access-nurses Radford, A. W., Robles, J., Cataylo, S., Horn, L., Thornton, J., Whitfield, K. (2014).The employer potential of MOOCs: a survey of human resource professionalsâ&#x20AC;&#x2122; thinking on MOOCs. RTI International. Retrieved from: http://www.rti.org/pubs/duke_handbook-final-03252014.pdf Raths, D. (2014, February 26). California community colleges joining forces for online success. Campus Technology. Retrieved from: http://campustechnology.com/articles/2014/02/26/californiacommunity-colleges-joining-forces-for-online-success.aspx Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

177

Rivard, R. (2013, May 5). State systems go MOOC. Inside Higher Ed. Retrieved from: http://www.insidehighered.com/news/2013/05/30/state-systems-and-universities-nine-statesstart-experimenting-coursera#sthash.NepRMKmy.dpbs Rizvi, S., Donnelly, K., & Barber, M. (2012). Oceans of innovation: the Atlantic, the Pacific, global leadership and the future of education. Institute for Public Policy Research, London, UK. Retrieved from: http://www.ippr.org/publication/55/9543/oceans-of-innovation-the-atlanticthe-pacific-global-leadership-and-the-future-of-education Ruth, S. (2013). Can MOOCs help reduce college tuition? George Mason University School of Public Policy. Research Paper No. 2014-06. Retrieved from: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2367425 Sandeen, C. (2013, October 1). The emerging world of alternative credentials. Higher Education Today: A Blog by ACE. [Web log post]. Retrieved from: http://higheredtoday.org/2013/10/01/theemerging-world-of-alternative-credentials/ Sandeen, C. & Jarratt, D. (2013). To MOOC or not to MOOC: strategic lessons from the pioneers, an analysis of administrator and faculty motivations. Survey conducted by ACE and Inside Track. Retrieved from: http://www.insidetrack.com/wp-content/uploads/2013/09/aceinsidetrack_mooc_strategy_motivations_study.pdf Scarsonmsm. (2013, May 27). Mechanics ReView MOOC: learning about digital learning through data [Web log post]. Retrieved from: http://mitopencourseware.wordpress.com/2013/05/17/mechanics-review-mooc-learningabout-digital-learning-though-data/ Shea, P. & Bidjerano, T. (2014). Does online learning impede degree completion? A national study of community college students. Computers & Education 75(2014), 103-11. Smith, B. (2013). The â&#x20AC;&#x153;Perfect Marketâ&#x20AC;? challenge to the subsidy structure of higher education. EDUCAUSE Review, 48(5), 80-87. Straumsheim, C. (2014, April 3). 2U Ends Semester Online. Inside Higher Ed. Retrieved from: http://www.insidehighered.com/news/2014/04/03/online-education-provider-2u-disbandsemester-online-consortium Swan, K., Bogle, L., Day, S., & Matthews, D. (2014). Assessing Massive Open Online Course pedagogy (AMP). Paper presented at the 2014 Annual Conference of the American Educational Research Association, Philadelphia, PA. Veeramachaneni, K., Dernoncourt, F., Taylor, C., Pardos, Z., & O'Reilly, U. (2013). MOOCdb: developing data standards for MOOC data science. Proceedings of the 1st Workshop on Massive Open Online Courses at the 16th Annual Conference on Artificial Intelligence in Education, Memphis, TN. Retrieved from: http://edf.stanford.edu/sites/default/files/Verramachaneni%20et%20al.%202013.pdf Waldrop, M.M. (2013). Online learning: campus 2.0. Nature, 495, 160-163. Retrieved from: http://www.nature.com/news/online-learning-campus-2-0-1.12590 Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

178

Xu, D. & Jaggars, S.S. (2013). Adaptability to online learning: differences across types of students and academic subject areas. CCRC Working Paper No. 54. Retrieved from: http://ccrc.tc.columbia.edu/media/k2/attachments/adaptability-to-online-learning.pdf Young, J. (2012, July 19). Inside the Coursera Contract: How an Upstart Company Might Profit From Free Courses. Chronicle of Higher Education. Retrieved from: https://chronicle.com/article/How-anUpstart-Company-Might/133065/

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

179

Appendices Appendix I: Institutional Affiliations and Roles of Interviewees Table A.1 Institutional Affiliations and Roles of Interviewees Number of institutions represented*

Administrators/ executives

Faculty members

Administrators/ faculty members

Researchers

Other roles**

Total interviewees

Public universities

Private universities

Community colleges

Platform providers

Research organizations

Other for-profit education companies

Other institutions***

Total

20 1

26 10

* Note that while only one person was interviewed at the majority of institutions, at a few institutions several individuals were interviewed, for example, to include one or two administrators, one or two faculty members, and one or two researchers. **Other roles: 1 educational technologist; 1 foundation program officer ***Other institutions: 1 museum (2 interviewees), 1 K-12 school district; 1 educational technology advocacy group; 1 higher education association; 1 venture capital firm; 1 foundation; 1 independent consultant.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

180

Appendix II: MOOCs: Expectations and Reality Interview Protocol Interviewer Name(s): Mode of communication: (phone, Skype, face-to-face meeting and location): Interviewee Name:

Job title:

Institution:

Date:

Start time:

End time:

1. What is your institution doing currently with respect to MOOCs? 2. What are the primary goals of your institution in pursuing MOOCs? 3. What is your role in this work and how did this role develop? 4. How do you and your institution define a MOOC? 5. Are there different types of MOOC? What are their characteristics? (e.g., structure, purpose, enrollment level, whether there is any fee associated, whether any kind of credential is offered). If a credential is offered, how does this differ from regular course credentials? Why did you choose specific types of MOOCs for your purposes? 6. For specific MOOCs: What are the educational objectives of the MOOC(s) you are offering? What educational outcomes are being measured? What other data are you collecting pre-, post- and during the course? e.g., enrollment, demographics of participants, reasons for taking course, participation, test scores, completion, post-course applications (networking, pursuit of further study, employer acceptance of any credentials). How is data collected from MOOCs being used to improve pedagogy either online or on-campus? 7. What amount of effort, personnel and other resources are required for the development and delivery of MOOCs? 8. What are the most important cost drivers? 9. How do the costs related to development and delivery of MOOCs compare with those for face-to-face or traditional online courses addressing the same topics? 10. Where do you see you see things going with MOOCs over the next 5 years? (generally and at your institution). 11. How do you see MOOCs helping your institution meet its goals over the next five years?

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

181

Appendix III: What Institutions are Doing in the MOOCspace This appendix provides a brief summary of the MOOC-related activity of each institution included in our study, provided the interviewee agreed to be identified. Names and titles of each interviewee are listed as written by the interviewees on their Informed Consent forms. Six interviewees requested total or partial anonymity. The information provided, titles, and roles of interviewees were current at the time of the interview (June 2013 – February 2014) unless otherwise mentioned. We list institutions alphabetically within the following categories: private universities, public universities, community colleges, platform providers, other for-profit education companies, research organizations, and other institutions/individuals. Private Universities Athabasca University, Canada Athabasca University administered the grants made available under the MOOC Research Initiative funded by the Bill & Melinda Gates Foundation. The grants were provided to examine the effectiveness of different MOOC models for a variety of audiences in different contexts. • George Siemens, Professor, Centre for Distance Education, Athabasca University, Canada. Siemens created the first cMOOC, Connectivism and Connected Knowledge, with Stephen Downes in 2008, which was offered at the University of Manitoba, Canada (see page 31 Origins of the cMOOC) and has re-run this twice since. He also offers a MOOC on learning analytics every Spring. In December 2013, Siemens moved to the University of Texas at Austin to become Director of the Learning Innovation and Networked Knowledge Research Lab. Columbia University, NY Columbia signed an agreement with Coursera in Fall 2012. Three MOOCs were offered in the first wave: Natural Language Processing, MOS Transistors, and Financial Engineering and Risk Management. Five more courses were launched or planned for Fall 2013 and Spring 2014 in diverse areas including economics, virology, history, and earth sciences. In March 2014, the university announced a partnership with edX. • Michael Cennamo, Educational Technologist, Columbia Center for New Media Teaching and Learning, Columbia University. Cennamo has worked on the design and development of four of Columbia’s Coursera MOOCs and is currently working on three edX MOOCs. •

Vincent Racaniello, Professor, Columbia University. Racaniello offered a series of two MOOCs, Virology 1: How Viruses Work and How Viruses Cause Disease, based on his on-campus course. (See Case 1).

•

Sree Sreenivasan, Former Chief Digital Officer and Professor, Columbia University. Sreenivasan coordinated online initiatives on the Columbia University campus. The role included cataloging online initiatives; sharing what was and was not working; finding ways to encourage and help faculty, departments and schools learn more about online learning and social and digital media. In August 2013, Sreenivasan became Chief Digital Officer at the New York City Metropolitan Museum of Art. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

182

Cornell University, NY Cornell joined the edX consortium in May 2013. From the 12-14 proposals submitted by faculty members through an internal Request for Proposals process, a committee chaired by the Provost selected four courses for the first round that would represent the Cornell brand without competing with on-campus courses. The second round of MOOCs will be launched in Fall 2014. • David BenDaniel, Berens Professor of Entrepreneurship, Johnson School, Cornell University. BenDaniel was invited to teach a MOOC on entrepreneurship during a sabbatical visit to MIT but declined because MOOCs cannot accommodate his Socratic style of teaching. •

Ted Dodds, Chief Information Officer and Vice President for Information Technologies, Cornell University. Dodds is responsible for the entire IT@Cornell community. He is tasked with rebalancing allocation of IT resources to maximize the return on investment.

East China Normal University, China The University is not currently offering MOOCs. • Minghua Li, Professor, School of Public Administration, East China Normal University. Li, in his 2011 book Research on Demand, Supply and Accreditation System of Higher Education for Migrant Workers in China, predicted a shift from degree markets to course markets. He is advocating for a complex platform that supports his vision for “MOOCs-Inside” courses, which are a form of blended learning. Harvard University, MA Harvard is one of the founding partners of edX. As of December 2013, HarvardX had offered 21 MOOCs on subjects including history, literature, medicine, statistics, and computer science. • Chris Dede, Wirth Professor in Learning Technology, Harvard University. Dede conducts research on online immersive simulations and is planning to conduct a massive, online experience on the subject of peer learning to model evidence-based online pedagogy at scale that includes effective use of peer learning. Massachusetts Institute of Technology (MIT), MA MIT is one of the founding members of edX and has offered 18 MITx MOOCs as of December 2013. XSeries certificates are available for sequences of MITx courses within a specific domain. • Professor Michael J. Cima, MIT. Prof. Cima offered the MOOC Introduction to Solid-state Chemistry and has integrated elements of the MOOC into his on-campus course. (See Case 6 ) •

Dave Pritchard, Cecil and Ida Greene Professor of Physics, MIT. Prichard offered the MOOC Mechanics Online twice using the LON-CAPA platform and Mechanics ReView once via edX. He conducts research related to MOOCs through the group Research in Learning and Assessing and Tutoring Effectively (RELATE) (See Case 9).

•

Professor Sanjay Sarma, Director of Digital Learning, MIT. Sarma works with faculty members to encourage them to offer their courses online and to support them through the design and development of the courses on the edX platform. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

183

New York Institute of Technology (NYIT), NY NYIT offered a series of webinar-like “mini-MOOCs” for the New York State Teacher Centers, providing technology-oriented professional development to K-12 teachers. Each section in the series consisted of a 30-minute synchronous session using Blackboard Collaborate and a three-week asynchronous component using Moodle. • Stan Silverman, Professor and Director, Technology Based Learning, New York Institute of Technology. Silverman created and facilitated the mini-MOOCs. Rensselaer Polytechnic Institute (RPI), NY RPI as an institution has not committed to creating MOOCs. RPI is interested in MOOCs as an experiment, for example in flipped classrooms, but is not convinced that they affect its mainstream business at present. Some instructors are, of their own accord, using MOOC content in flipped classroom activities. RPI is developing a MOOC-like bridging course in Calculus for incoming students. • John E. Kolb, VP of Information Technology and Services and CIO, RPI, NY. Kolb provides leadership for acquisition, use, and growth of campus-wide information resources, services, and technology to support education, research, and administrative activities. Stanford University, CA xMOOCs originated at the Stanford campus in 2011 with courses offered by Andrew Ng, Jennifer Widom, Sebastian Thrun and Peter Norvig (see page 34 Origins of the Modern MOOC (xMOOC). The MOOC platform providers Udacity, Coursera, and NovoEd were each founded by Stanford University professors. • Amy Collier, Director of Digital Learning Initiatives, Stanford University. Collier heads the pedagogy team that works with faculty members to design instruction for online and blended learning environments. •

Emily Schneider, Student Director, Lytics Lab, Stanford University. Schneider helped establish the Lytics Lab to bring together researchers to collaborate on research related to online learning or MOOCs. Schneider was a lead organizer of the moocshop in 2013, one of the first MOOC research conferences.

•

Candace Thille, Assistant Professor, Director of Open Learning Initiative, Senior Research Fellow, Office of the Vice Provost for Online Learning, Stanford University. Thille directed the Open Learning Initiative (OLI) at Carnegie Mellon University before moving to Stanford in 2013. She is currently building the functionality of the OLI platform into the Open edX platform to support the OLI open courses.

•

Joseph Jay Williams, Research Fellow, Lytics Lab, Graduate School of Education, Office of the Vice Provost of Online Learning, Stanford University. Williams works in the Lytics Lab conducting research on MOOCs and on Khan Academy and supporting the research of other graduate students in the Lab.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

184

Teachers College (TC), Columbia University, NY TC offered its first MOOC in Fall 2013, Big Data in Education, hosted on the Coursera platform. • Ryan Baker, Associate Professor, Teachers College, Columbia University, NY. Baker created and taught Big Data in Education. •

Yuan "Elle" Wang, MOOC Researcher, Teachers College, Columbia University. Wang previously taught a MOOC via GLR, an online learning company based in China. She was the teaching assistant for Big Data in Education and is conducting research using data collected during the MOOC.

Tsinghua University, China Tsinghua University signed a contract with edX to offer MOOCs in Summer 2013 and has offered two courses: Principles of Electric Circuits and History of Chinese Architecture. The content videos are presented in Mandarin with English subtitles. Five more MOOCs are planned for Spring 2014. • Dr. Fengliang Li, Associate Professor, Institute of Education, Tsinghua University, China. Li is interested in MOOC research. Tufts University, MA Tufts University has not offered a MOOC. • Lawrence S. Bacow, President Emeritus, Tufts University. Bacow is currently collaborating with Bill Bowen, former President of Princeton University and the Mellon Foundation; Mike McPherson, President of the Spencer Foundation; and Kevin Guthrie, President of ITHAKA, on a series of papers exploring issues of governance related to MOOCs, and more generally on how technology may influence colleges and universities in the future. Vanderbilt University, TN Vanderbilt University is both a producer of MOOCs and a consumer in that it integrates MOOCs created by other institutions into its own courses. The University partnered with Coursera in August 2012 and initially offered five MOOCs. As of August 2013, a total of 12 MOOCs had been planned or offered. •

Doug Fisher, Director of the Vanderbilt Institute for Digital Learning, Vanderbilt University. Fisher leads the Institute’s work on research, innovation, and support for digital learning. The Institute funds the exploration of digital learning by faculty members and fellows, and oversees the production of MOOCs. Fisher integrated a “closed instance” of the MOOC Artificial Intelligence, created at University of California, Berkeley and offered via edX, into his own course on the topic. Public Universities

City University of New York (CUNY), NY A number of CUNY colleges have experimented with various MOOC derivatives. Macaulay Honors College joined as a node in a “distributed online collaborative course” (DOCC) on feminism and technology. JustPublics@365, a partnership between the Graduate Center of the City University of New York and the Ford Foundation offered a participatory open online course. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

185

•

Ann Kirschner, University Dean, Macaulay Honors College. Kirschner is an author and speaker on education policy and strategy, and digital media.

•

Joseph Ugoretz, Associate Dean of Teaching, Learning and Technology, Macaulay Honors College (CUNY). Ugoretz offered a mini-MOOC: Alternate Worlds, Imagining the Future of Education. The course was structured as a MOOC and was open but not massive.

Georgia Institute of Technology (Georgia Tech), GA Georgia Tech’s Center for 21st Century Universities and Georgia Tech Professional Education (GTPE) are creating about 20 MOOCs hosted by Coursera, mostly funded by the Bill & Melinda Gates Foundation. In 2013, Georgia Tech signed a 3-year contract with Udacity to offer the Online Master of Science in Computer Science program (OMSCS) which began enrolling students in January 2014. The initiative received a $2 million gift from AT&T to cover start-up costs. • David White, Assistant Dean for Academic Programs and Executive Director of the Online M.S. in Computer Science, College of Computing, Georgia Tech. White works with faculty, administration, and Udacity to implement the OMSCS program. Georgia State University, GA The University System has signed contracts with Coursera and Desire2Learn. However, it is not presently developing MOOCs as an institution. Individual professors have created their own MOOCs or used parts of MOOCs created by others. In January 2013, the Georgia State University Senate passed a resolution that students claiming competencies based on their MOOC experiences can apply to take a test of these competencies for potential granting of credit. • Timothy Renick, Vice Provost and Chief Enrollment Officer, Georgia State University. Renick directs Georgia State's admissions, enrollment, scholarship aid, and student success programs. North Carolina State University, NC Two centers at the university experimented with MOOCs. Distance Education Learning Technology Applications (DELTA) offered an open online course (OOC), Digital ASIC Design, with Engineering Online, and the Electrical and Computer Engineering Department. Separately, the Friday Institute has offered multiple MOOC-Eds (MOOCs for educators). • Dr. Glenn Kleiman, Executive Director, Friday Institute for Educational Innovation, North Carolina State University. The Friday Institute is focused on digital learning and professional development in the K-12 sector. The institute has offered several MOOC-Eds: Planning for Digital Transition; Coaching Digital Learning: Cultivating a Culture of Change; and World Class Teaching. Additional MOOC-Eds under development will focus on disciplinary literacy and several topics in teaching mathematics. Further information is available as www.mooc-ed.org. Pennsylvania State University, PA A MOOC strategy group was formed in 2012 and a first wave of five MOOCs was offered on the Coursera platform in Fall 2013 and Spring 2014: Introduction to Art; Maps and the Geospatial Revolution; Change, Innovation and Creativity; The Epidemics: The Dynamics of Infectious Diseases; and Energy, the Environment, and Our Future. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

186

•

Lawrence C. Ragan, Director, The Center for Online Innovation in Learning (COIL), Penn State World Campus. Ragan has contributed to the establishment and management of Penn State World campus. He directs COIL's work of research, scholarship, technology innovation, and leadership development programming. COIL spearheads the research related to the MOOCs created by Penn State University.

San José State University (SJSU), CA SJSU, in partnership with Udacity, offered Remedial Algebra, Introduction to College Level Algebra, and Introduction to College Level Statistics in Spring 2013 and Introduction to Psychology and Introduction to Java Programming in Summer 2013. The University also offered Circuits Analysis in a flipped classroom using materials from the MITx MOOC Circuits and Electronics, which was hosted on the edX platform. • Catheryn Cheal, Ph.D., Associate Vice President and Senior Academic Technology Officer, San José State University, CA. Cheal was closely involved in faculty development and management of the SJSU partnerships with Udacity and edX. University of California (UC), Berkeley, CA UC Berkeley is one of the contributing members of edX. MOOCLab, a three-year research initiative within the Berkeley Resource Center for Online Education (BRCOE), funds and develops MOOCs as vehicles for pedagogical research in online education. • Armando Fox, Professor, Electrical Engineering and Computer Science Department, Faculty Advisor to the MOOCLab, UC Berkeley, CA. Fox launched the first software engineering MOOC on Coursera, Software as a Service, in January 2012. The course has been offered five times and was moved to the edX platform in Fall 2012. University of California, Irvine, CA UC Irvine was the first UC campus to become involved in open learning and one of the first universities invited to create MOOCs by Coursera. UC Irvine launched six MOOCs in Spring 2013 on various subjects including personal financial planning, math, physics, and public health. Four of the courses were offered again in Fall 2013, plus two new courses in economics. Some UC Irvine MOOCs offer the Signature Track option and two math MOOCs have earned ACE credit recommendations. In addition to the Coursera MOOCs, UC Irvine also offered the MOOC Society, Science, Survival: Lessons from AMC’s “The Walking Dead,” based on the television series “The Walking Dead,” on the Instructure Canvas Network platform. • Melissa Loble, formerly Associate Dean for Distance Learning, University of California, Irvine Extension. Loble provided leadership in curriculum development, instructional design, and the selection and utilization of underlying platforms and technologies. Irvine Extension is the continuing education arm of the University that offers online as well as on-site courses. Loble recently moved to Canvas Network where she is Senior Director, responsible for strategic planning and day-to-day operations.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

187

University of California, Santa Barbara, CA UC Santa Barbara has not offered a MOOC. • Christopher Newfield, Professor of Literature and American Studies, University of California, Santa Barbara. Newfield taught one of the three courses offered in tandem with the Coursera MOOC History and Future of (mostly) Higher Education. He received a Bill & Melinda Gates Foundation grant to conduct research on MOOCs offered at San José State University in partnership with Udacity. University of Colorado System, CO The University of Colorado’s Boulder campus became a Coursera partner in May 2013, with a System contract following later that year. The Boulder campus offered four physics MOOCs that ran parallel to the on-campus courses; two graduate level engineering courses, Parallel Electronics and Linear Programming; and one undergraduate level course in the humanities, Comic Books and Graphic Novels. The Colorado Springs campus offered the course Intro to Game Programming using C#. The Downtown Denver campus (School of Business) offered the course Global Energy Management. Two more Coursera courses are planned for Summer and Fall 2014 release: Deciphering Secrets: Unlocking the Manuscripts of Medieval Spain will be taught from the Colorado Springs campus, and Global Health Responder from the Anschutz Medical Campus. In addition to the Coursera courses, the university is planning to convert a popular outreach course entitled Mini Med School into a MOOC to be offered on the Canvas learning management system. • Deborah Keyek-Franssen, (Ph.D.), Associate Vice President for Digital Education and Engagement, University of Colorado System. Keyek-Franssen helps campuses in the development and implementation of strategic plans for digital education including teaching and learning with technology, online courses and programs, and MOOCs. University System of Maryland (USM), MD USM is participating in a study led by Ithaka S+R involving 20 tests to integrate online components from Coursera, OLI at Carnegie Mellon, and various Pearson Products into their courses across a variety of disciplines. USM is considering the development of a “USM-OOC,” a system-level open online course, produced and hosted by one USM campus and made available to all system campuses. • MJ Bishop, Director, Center for Innovation and Excellence in Learning and Teaching (CIELT), University System of Maryland. CIELT was established in 2013 to promote academic innovation. The center conducts research in best practices, disseminates results, and supports the USM campuses in implementing academic practices. University of Maryland University College (UMUC), MD Being an adult learning college, UMUC prefers to be a consumer of MOOCs rather than a producer. For students completing MOOCs that have been recommended for credit by ACE, UMUC will award transfer credits provided the students show adequate evidence of competency. • Karen Vignare, Associate Provost, Center for Innovation in Learning, University of Maryland University College (UMUC).

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

188

Vignare leads the Center for Innovation in Learning which was established in 2013 as an incubator for new approaches, practices, and learning models to understand and improve online learning. University of Oklahoma, OK University of Oklahoma has partnered with NextThought to provide courses that are open to the public for free as well as available for credit to matriculated students who receive additional student support and services. In Fall 2013, the first six courses were launched in subjects including Philosophy, International Studies, Meteorology, Chemistry, and Statistics. Fourteen additional courses were offered in Spring 2014. A total of 20 MOOCs have been developed and offered in one year. • Kyle Harper, Senior Vice Provost, University of Oklahoma. Harper assists the Provost on major new academic initiatives and acts as a liaison to the new working group on digital technology. •

Nick Hathaway, Vice President for Administration and Finance, University of Oklahoma. Hathaway serves as the Chief of Staff for the President's office and partners with Harper to move the MOOC initiative forward.

University of Texas System, TX University of Texas System is a member of the edX consortium and contributed $10 million towards building several dozen courses over several years. It has offered eight MOOCs so far in subjects including mathematics, music, medicine, and globalization. University of Texas at Arlington College of Nursing has created a MOOC as part of the MOOC2Degree initiative (see Case 4). In Fall 2014, University of Texas at Austin is planning to experiment with a “freemium” model whereby a free version of a course will be offered via edX along with a paid, gated course that includes more assessments, student support, and course credit. • Steven Mintz, Executive Director, Institute for Transformational Learning, University of Texas System. The Institute is charged with expanding access to educational programs that will improve learning and reduce costs, and promote a culture of educational innovation throughout the University of Texas System. University of Wisconsin-Madison, WI The University offered two MOOCs via Coursera in Fall 2013, Video Games and Learning and Markets with Frictions. • Dr. Paul S. Peercy, Dean Emeritus, College of Engineering, University of Wisconsin-Madison. Peercy is a part of the Global Engineering Deans' Council and a proponent of technology in STEM education globally for which demand exceeds supply of quality instruction. Over a period of around 10 years, Peercy oversaw his department’s shift to the use of flipped classrooms. West Virginia University (WVU), WV WVU signed a contract with Coursera in Summer 2013 and a committee was formed to invite and review faculty member proposals for courses to be launched in 2014.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

189

•

Matthew Martin, Professor of Communication Studies, West Virginia University. In February 2013, the Department of Communication Studies offered a four-week cMOOC on social media facilitated by four faculty members including Martin. The department is planning to offer another cMOOC on the topics of influence and persuasion. Community Colleges

Academic Senate for California Community Colleges, CA This is a nonprofit organization created for the promotion and advancement of public community college education in California. • Beth Smith, President, Academic Senate for California Community Colleges. Smith represents 50,000 faculty members at community colleges in California. Smith expects that many community college faculty members will to try to adapt MOOC content to test how it works within their own classes but she cautions that there are a number of “unresolved issues” with respect to offering MOOCs as stand-alone online courses. Smith is also a Professor of Mathematics at Grossmont Community College, CA, which has not engaged in MOOC activity. Cuyahoga Community College (Tri-C), OH Tri-C offered a four-week developmental math MOOC, Believe Island, hosted on Blackboard CourseSites and funded by the Bill & Melinda Gates Foundation. The MOOC was offered four times in Summer 2013 and remains open without facilitation (see Case 2). • Belinda Miles, Provost and Executive Vice President of Academic and Student Affairs, Cuyahoga Community College, OH. Miles actively supported and led the Tri-C MOOC initiative. •

A. Sasha Thackaberry, District Director, eLearning Technologies, Cuyahoga Community College, OH. Thackaberry was the project lead and created the first module of the developmental math MOOC as a prototype. She supports innovative eLearning initiatives across the college to provide solutions to institutional challenges.

Foothill College, CA Foothill College is not producing or using a MOOC at present. • Judy Baker, Ph.D., Dean, Foothill College, CA. Baker has been active in the Open Education Resources (OER) movement since its inception and has offered workshops over the years to community college personnel on OER and online learning, including MOOCs. Inver Hills Community College, MN Inver Hills Community College is not currently producing or using MOOCs. • Christina Royal, Chief Academic Officer/Provost, Inver Hills Community College. In her previous role as Associate Vice President for eLearning and Innovation at Cuyahoga Community College (Tri-C), Royal participated actively in Tri-C’s initiative to create a developmental math MOOC. Currently Royal is developing a strategic plan for online and adult learning at Inver Hills Community College. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

190

Mt. San Jacinto Community College, CA In 2013, Mt. San Jacinto Community College offered a developmental English MOOC, Crafting an Effective Writer, on the Coursera platform. The project was funded by the Bill & Melinda Gates Foundation. • Patricia James, Dean of Distance Education, Mt. San Jacinto Community College, CA. James is an administrator for instructional technology, and distance education programs. She was actively involved in the design and development of Crafting an Effective Writer. Oklahoma City Community College, OK The college is evaluating MOOCs and other variations including SMOCs but is unwilling to commit scarce resources to an initiative with uncertain outcomes for its students. • Paul Sechrist, President, Oklahoma City Community College. Santa Barbara City College, CA The college is not producing or using a MOOC at present but provides many online courses for its students. •

Dr. Douglas Hersh, Dean, Educational Programs, Santa Barbara City College, CA. Hersh conducts research on human presence design for online learning and has facilitated a move towards a unified learning management system for the college. He is a proponent of the establishment of a state-wide system to allow community college students to take online courses from any of the state’s 112 community colleges. Since the time of our interview, Hersh moved to Grossmont College where he is Dean, Learning and Technology Resources.

Wake Technical Community College, NC The college partnered with Udacity to create the MOOC Introductory Algebra Review, supported by the Bill & Melinda Gates Foundation, to help students prepare for the North Carolina community college placement test. The MOOC was opened in May 2013 and will remain open for 5 years. • Laura Kalbaugh, Dean, Wake Technical Community College, Raleigh, NC. Kalbaugh led the project to create Introductory Algebra Review.

Platform Providers Coursera, CA Coursera is the largest for-profit MOOC platform provider, founded by computer science professors Andrew Ng and Daphne Koller from Stanford University. • Pang Wei Koh, Director of Course Operations, Coursera. Koh oversees the design, implementation, and support of all online courses and works with faculty and staff from partner institutions. edX, MA edX is a nonprofit entity founded by MIT and Harvard University and managed as a separate entity from the universities. The edX consortium includes 26 contributing member institutions. MOOCs constitute one part of a broad strategy for reinventing online and campus education. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

191

•

Rebecca Petersen, Director, Research and Educational Initiatives, edX. Petersen leads experimental initiatives and pilots the use of MOOCs in various academic settings, and promotes blended learning models among K-12 and higher education institutions.

•

Howard Lurie, formerly Vice President for External Affairs, edX; currently Managing Director, Education Partnerships, Acrobatiq - A Carnegie Mellon University Venture.

FutureLearn, United Kingdom (UK) FutureLearn is a company wholly owned by The Open University in the UK, and provides platform services for MOOC production and delivery to 23 university partners. Additionally, three content partners - the British Library, British Museum, and British Council - work with some university partners to create courses that are offered in tandem with their exhibitions and other events. • Mark Lester, Head of UK Partnerships, FutureLearn. Lester, previously at The Open University, is responsible for recruiting partners and supporting them in the use of the FutureLearn platform. NextThought, OK NextThought has built a platform that encourages and facilitates connection among participants in an online learning community and can be used to deliver MOOCs. NextThought has helped develop and deliver 20 online courses for the University of Oklahoma. • Ken Parker, CEO, NextThought. Parker is the co-founder of NextThought. Straighterline, MD Straighterline does not offer MOOCs but offers online courses, all with ACE credit recommendations, on a subscription basis. The organization licenses materials from content providers including McGraw-Hill, ThinkWell, and Saylor Foundation, and provides supplementary services such as academic support, student advising, proctoring, pathways to credit, and transcripting. Articulation agreements with academic institutions guarantee acceptance of some courses for credit. Students can try the first two lessons of any Straighterline course for free. • Burck Smith, CEO, Straighterline. Smith is the founder of Straighterline and writes about education policy.

Other For-profit Education Companies Academic Partnerships (AP), TX AP launched the MOOC2Degree initiative in January 2013 which converts the initial core course of a partner university’s online degree program into a MOOC. Completers who are subsequently accepted into the partner university’s full program can receive credit for the MOOC based upon criteria established by the university. AP helps to adapt the courses for delivery on any platform of the partner university’s choice, and markets the courses to prospective students. • Dr. Justyna Dymerska, Executive Vice President, Academic Partnerships. Dymerska directs AP's international operations and strategy.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

192

Kaplan Inc., USA Kaplan University is working on “large, open, zero-variable-cost-delivery pilots” and is planning to create a MOOC on the application of learning science at scale to instructional environments. It will be based on Kaplan's internal training program for instructional designers. • David Niemi, VP, Measurement and Research, Kaplan Inc. Niemi leads efforts to improve the quality of measurement, evaluate the effectiveness of curricula and instruction, and study the impact of innovative products and strategies. •

Bror Saxberg, Chief Learning Officer, Kaplan Inc. Saxberg leads the Kaplan Innovation group that will design the MOOC.

McGraw-Hill Education, NY The organization is not developing any platform that is targeted specifically to the delivery of MOOCs, but is working with institutions developing MOOCs and with MOOC platform providers to help them apply appropriate pedagogical strategies and to design learning interactions. • Stephen Laster, Chief Digital Officer, McGraw-Hill Education. Laster is responsible for the organization's e-learning and educational technology strategy. Pearson, NY Pearson is not directly creating MOOCs, but some Pearson business units provide infrastructure support to other institutions. For example, Embanet Compass helps many institutions provide their online courses. Pearson has also invested significantly in Coursera via its investment in Learn Capital. •

Saad Rizvi, Senior Vice President, Efficacy, Pearson. Rizvi is a co-author of the publications Oceans of innovation: The Atlantic, the Pacific, Global Leadership and the Future of Education (2012), and An Avalanche is Coming: Higher Education and the Revolution Ahead (2013). Research Organizations

American Institutes for Research (AIR), DC AIR is a behavioral and social science research and evaluation organization. •

Jessica Heppen, Managing Researcher, American Institutes for Research. Heppen conducts research on online learning, mostly in the K-12 sector, and has engaged in conversations with Cousera about possible research collaborations.

California State University (CSU), CA • Dr. John Whitmer, Program Manager, Academic Technology & Analytics Office of the Chancellor, California State University. Whitmer was a co-author of the report on San José State University’s SJSU+ Augmented Online Learning Environment [AOLE] Pilot Project and is conducting research on Mt. San Jacinto’s MOOC Crafting an Effective Writer. Note: We list CSU here rather than under public universities above because we interviewed Whitmer as a researcher of MOOCs, not as a representative of CSU. We are aware, however, that CSU created a Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

193

MOOC Joint Task Force to examine current and potential legislation regarding online courses and to ensure effective faculty voice regarding the role of MOOCs in universities. Community College Research Center (CCRC), Teachers College, Columbia University, NY CCRC has published some of the best-known studies on online learning in higher education. The center is not actively conducting research on MOOCs but some of the community colleges it is working with are experimenting with integrating MOOC materials into their face-to-face or hybrid courses. •

Shanna Jaggars, Assistant Director, Community College Research Center, Teachers College, Columbia University. Jaggars manages research studies focusing on community college student success and improving student motivation.

Note: While we list CCRC here separately from Teachers College above, we do not count it as a separate institution for the purposes of our frequency analysis of institutional goals for pursuing MOOCs. Contact North, Ontario, Canada Contact North is Ontario's distance education and training network. •

Dr. Tony Bates, Research Associate, Contact North, Ontario. Bates advises institutions of higher education on online learning strategy. He has published books on technology and e-learning, including analyses of the costs of online programs. He served as a participant instructor in the cMOOC CHANGE offered by Stephen Downes, focusing on management in higher education. Bates maintains a blog on online learning and distance education resources.

ITHAKA/Ithaka S+R, NY In partnership with the University System of Maryland (USM), Ithaka S+R is conducting a study involving 20 experiments to integrate online components from Coursera, OLI at Carnegie Mellon, and various Pearson Products into USM courses across a variety of disciplines. The study includes side-by-side comparisons of traditional and hybrid versions of large introductory level courses, as well as case studies of smaller, mostly seminar-style courses. •

Kevin Guthrie, President, ITHAKA. Guthrie is a founder of ITHAKA.

•

Rebecca Griffiths, Program Director for Online Learning, Ithaka S+R. Griffiths oversees the Bill & Melinda Gates Foundation-funded study being conducted in partnership with USM.

National Research Council (NRC), Canada NRC is the Government of Canada's research and technology institution. • Stephen Downes, National Research Council. Downes offered the first cMOOC, Connectivism and Connected Knowledge, with George Siemens in 2008. (See page 33 Origins of the cMOOC). He has offered other cMOOCs since then including Personal Learning Environments, Networks, and Knowledge (PLENK) with George Siemens; Critical Literacies with Rita Cobb; CHANGE, a 30-week-long MOOC with multiple facilitators, Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

194

collaborators, and visiting experts; and The Future of Higher Education in partnership with Desire2Learn. Downes maintains a widely-read, online newsletter about online learning: OLDaily. Pittsburgh Science of Learning Center (PSLC), PA PSLC, based at Carnegie Mellon University, is one of the National Science Foundation Science of Learning Centers. It “leverages cognitive theory and computational modeling to identify the instructional conditions that cause robust student learning” (see LearnLab at PSLC). Learning researchers at PSLC conduct research on learning theories, for example, by embedding features into Open Learning Initiative (OLI) courses. PSLC also established DataShop, “a central repository to secure and store research data” and “a set of analysis and reporting tools” for the learning science community (see PSLC DataShop). • Ken Koedinger, Professor of Human-Computer Interaction and Psychology, Carnegie Mellon University. Koedinger has two decades of experience in educational research in educational technology and intelligent tutoring systems. He is the Director of PSLC. SRI International, CA SRI is an independent, nonprofit research institute. •

Barbara Means, Center for Technology in Learning, SRI International. Means will be conducting a review of projects funded by the Bill & Melinda Gates Foundation under the Post-Secondary Success Group's technology-based initiative. Some of these include MOOCs. Means participated in reviewing proposals for the MOOC Research Initiative. Other Institutions/Individuals

American Council on Education (ACE), DC ACE provides credit recommendations for the military, government agencies, and corporate entities, evaluating occupational training and experiences to discover embedded competencies that can be aligned with credit-bearing degree programs. Several MOOCs have been reviewed by ACE and recommended for credit. •

Cathy Sandeen, Vice President, Education Attainment and Innovation, American Council on Education. Sandeen oversees ACE’s national agenda and directs the Center for Education Attainment and Innovation.

American Museum of Natural History (AMNH), NY AMNH offered three 4-week courses aimed at educators via the Coursera Teacher Professional Development program: Genetics and Society: A Course for Educators; The Dynamic Earth: A Course for Educators; and Evolution: a Course for Educators. • Dr. Ro Kinzler, Senior Director, Science Education, American Museum of Natural History, NY. • Dr. Robert Steiner, Director, Online Teacher Education Programs, American Museum of Natural History, NY. Steiner and Kinzler were heavily involved with directing the overall MOOC initiative, course design, and creation of materials. Kinzler served as an instructor in one course. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

195

Digital Promise, Washington, DC Digital Promise is an independent, nonprofit organization authorized by the U.S. Congress to incentivize innovation in education. •

Karen Cator, President and CEO, Digital Promise. Cator previously worked as Director of the Office of Educational Technology at the U.S. Department of Education. She has been actively following the MOOC phenomenon due to her interest in lifelong learning and educational innovation using technology.

Harrison Central School District, NY The school district is not producing or using MOOCs at present. •

Lawrence Paska, Director of Social Studies, Harrison Central School District, Harrison, NY; former State Education Agency Technology Policy Coordinator. In his previous role as Technology Policy Coordinator for New York State, Paska developed policies and initiatives to foster and implement educational technology across learning environments statewide.

Sir John Daniel, Canada • From 1990-2001, Sir John Daniel was Vice-Chancellor of The Open University in the United Kingdom, which has recently created FutureLearn, a for-profit company, to deliver MOOCs. He was President and CEO of the Commonwealth of Learning from 2004 to 2012 and headed UNESCO’s Education Sector from 2001-2004. He is currently an advisor to Academic Partnerships in Texas. Daniel has written extensively on the topics of technology in education, including online learning and MOOCs. Technology Crossover Ventures (TCV), CA This venture capital firm invests in the online education space broadly, including online education providers and “enablers,” curriculum-oriented products, schools, and software companies that support the education industry. For example, TCV was an early investor in Embanet, which helps universities take their programs online, and sold the firm to Pearson in 2012. TCV is not currently investing in MOOC-related companies as they do not fulfill TCV’s investment criterion of a minimum of $30 million in revenues. •

Jake Reynolds, General Partner, Technology Crossover Ventures. Reynolds is a board member of K-12 Inc., the largest for-profit provider of K-12 online education in the United States. Previously Reynolds served as a board member of Capella University.

William and Flora Hewlett Foundation, CA The foundation has supported many innovative, technology-related projects in education. For example, the Open Educational Resource movement and Peer 2 Peer University (P2PU), a nonprofit, online, open learning community which allows users to develop and deliver their own courses and to participate in courses developed by others. •

Victor Vuchic, Program Officer Education, William and Flora Hewlett Foundation. Vuchic has been a strong proponent and supporter of Open Educational Resources for many years. Before “MOOC” became a buzzword, he was working with groups providing massive open online courses through P2PU. In March 2014, Vuchic founded V3 Consulting. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

196

Appendix IV: Costs of Online Video Production Video production costs for MOOCs and other online courses vary dramatically. At one extreme an instructor may use a desktop camera and video-recording software such as Camtasia or ScreenFlow to record him/herself. At the other extreme, an external professional production team may be hired to video-tape live lectures or recording sessions, and to incorporate custom-made animations and graphics into the edited product. The former strategy usually results in “low production value” video and the latter in “high production value” video. (See Box 12 for an explanation of these terms). In between these extremes, an in-house production team may work with course instructors to capture and edit the video. Decisions as to which strategy to adopt are usually made based on time and cost considerations, availability of a dedicated team to work within the required deadline for course launch, and capacity to meet desired production values. An internal production team at an institution that is simultaneously developing many online courses must juggle many priorities. Costs of engaging an internal production team can be hard to isolate for a specific project unless detailed time logs are maintained. The video production costs shown in Table A.2 are based on the work being executed by a video production company partnering with a large R1 university in the development and delivery of multiple online courses over a one-year period. The courses are primarily targeted at the university’s enrolled students who pay fees and receive credit for successful course completion. However, like MOOCs, the courses are open to any member of the public to participate in for free and without receiving credit. The production company indicated that the university’s decision-making process with respect to course production was primarily driven by considerations of quality of teaching and outcomes, consistent with the university’s core values, as opposed to price sensitivity. The mandate was to create high production value videos. For each 16-week course completed to date, a total of 5-14 hours of finished video has been created. Each weekly unit of a course typically includes five 8-minute clips of video. Table A.2 provides estimated costs for a course that involves the production of 10 hours of finished video. The total cost is $43,344 and the major cost drivers are camera operators and equipment, video-editing, the project manager, and the production manager. While these costs may be approximated to around $4,300 per hour of finished video, it is likely that an external team will discount rates as expected engagement time increases. Readers may substitute their own prices and estimated hours of video and labor to calculate their own local costs. If the video content can be re-used on multiple occasions, the costs can be spread across these instances.

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

197

Table A.2 Production Costs for an Online Course with 10 hours of Finished Video Ingredient

Cost per unit

Cost assumptions for one course, 10 hours finished video

Estimated cost for one course, 10 hours of finished video

Management Project manager

$90,000/year incl. benefits

Annual salary + benefits spread over multiple courses

$5,000

Production manager

$90,000/year incl. benefits

Annual salary + benefits spread over multiple courses

$5,000

Video capture Camera operator with equipment

$800$1,200/day

2 people, 5 days each, $1,000/day

$10,000

Audio technician with equipment

$500/day

1 person, 5 days

$2,500

Make-up artist

$350/day

1 person, 5 days

$1,750

Set components

Highly variable

$250 per course

$250

Labor for set assembly

Variable

2 interns, 2 days at $12/hour

$384

Video facility/location rental

$100/hour

Rental for 5 days

$4,000

Tele-prompter

$200/day

5 days of use

$1,000

Video editing and production Video-editor

$100/hour

88 hours of editing

Intern to obtain/verify copyright permissions

$15/hour

40 hours

$600

Graphic designer to create custom graphics/animation

$100/hour

16 hours

$1,600

Intern to locate graphics, images/animations

$15/hour

50 hours

$750

Graduate Assistants to review video for content errors

$20/hour

20 hours

$400

Video transcripts

$130/hour

10 hours finished video

GRAND TOTAL

$8,800

$1,300 $43,344

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

198

Assumptions: A day is 8 hours long. While “efficiency” of shooting varies depending on factors including instructor’s preparedness and confidence, it is assumed that to obtain 1 hour of usable video requires 4 hours of shooting time. The rule of thumb adopted here for editing time is twice the shoot time plus 1 day extra. Note: Travel and accommodation for a “destination” shoot or for engagement of a team that is not locally based would be additional. For 8 hours travel time and overnight accommodation for a team of 5, this could amount to $4,350, assuming one production manager and two camera operators each at $1,000/day, one audio technician and one make-up artist, with hotel and per diem at $100 per person for one night.

Box 12: High Production vs. Low Production Values for Video Low production value video: image is not sharp, color is dull, audio is tinny and unclear; background is static and unchanging. The lowest production values generally arise when an un-manned camera is placed in the back of a classroom to record a lecture. A step up involves the instructor speaking into a web-cam on his/her computer screen with the occasional PowerPoint slide included. High production value video: aims for substantial visual variety and maximal viewer engagement. Two or more cameras are used, switching back and forth to provide different perspectives. Cameras may be placed on sliders to allow movement. One camera is likely to have a wide-angled lens to capture the larger picture while another provides close-ups so that the instructor appears to be talking directly to the viewer. The edited video incorporates high quality graphics, animations, supplementary audio and images to provide visual cues that help explain or enhance the verbal message being delivered. Roles of production team members Project manager: co-ordinates across all courses, liaising with and scheduling instructors and production team, ensuring textual resources and images are verified for copyright. Production manager: co-ordinates the video production team and directs the video collection, editing, and post-production. Camera operators and equipment: provide and operate camera equipment; arrange and control lighting. A wide-angled camera is used to capture a group, 1-2 other cameras are focused on individuals. If the shoot is complex with a dynamic set up requiring lights to be moved around, a separate lighting technician may be required. Audio-technician: sets up the audio equipment and microphones (portable, lapel, or boom); monitors the recording equipment while the shoot is under way; helps synchronize audio with the video material. Make-up artist: helps instructor get “camera-ready,” including coaching on clothes to wear. Video-editors: review hundreds of gigabytes of digital content from multiple takes, eliminate repetitions and errors in elicitation, and distill the message to be delivered. Incorporate “eye-candy:” images, graphics, and animations to help explain the content. Submit “proof draft” to instructor or graduate assistants to check for content errors. Source: U.S. video production company. Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

199

Appendix V: Sample Pre- and Post-MOOC Participant Surveys Penn State University [This survey was provided courtesy of Penn State University]. ______________________________________________________________________________________________________ NOTE: We are assuming that Coursera will gather basic demographic and user analytics information on the MOOC participants. Pre-course Survey 1. Please indicate your previous online learning experience by selecting all that apply. a. I have taken other MOOCs b. I have taken other online learning courses for academic credit c. I have taken other online learning courses but not for academic credit d. None of the above 2. I am taking the course out of general interest, curiosity, or enjoyment. a. not at all important b. moderately important c. important d. very important e. absolutely critical 3. I intend to watch ____ of the lectures for this course. a. none b. some c. most

d. all

4. I intend to complete ____ of the assessments for this course. a. none b. some c. most d. all 5. I intend to earn a Statement of Accomplishment (or Verified Certificate) for this course. a. strongly disagree b. somewhat disagree c. neutral d. somewhat agree e. strongly agree How important were the following considerations in your decision to enroll in this course? 6. I am interested in taking a course from this particular institution. a. not at all important b. moderately important c. important d. very important e. absolutely critical 7. I am interested in taking a course from this particular professor. a. not at all important b. moderately important c. important d. very important e. absolutely critical 8. I am interested in earning a credential. a. not at all important b. moderately important d. very important e. absolutely critical

c. important

9. I am interested in connecting with other students interested in this topic. a. not at all important b. moderately important c. important d. very important e. absolutely critical 10. I have friends taking this course. a. not at all important b. moderately important d. very important e. absolutely critical 11. The course relates to my current academic program. a. not at all important b. moderately important

c. important

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

200

d. very important

e. absolutely critical

12. The course relates to my current job responsibilities or company's line of business. a. not at all important b. moderately important c. important d. very important e. absolutely critical 13. The skills from this course may be useful for obtaining a new job. a. not at all important b. moderately important c. important d. very important e. absolutely critical 14. Which of the following best describes your previous experience in the subject area of this course? a. I am mostly new to this subject. b. I have explored this subject on my own. c. I have completed some coursework or have some work experience in this field. d. I have a degree or significant work experience in this field. e. [+ custom questions from instructor] 15. Due to the large enrollments in a MOOC, there is little to no interactions between the course instructor and participants. Does this model meet your expectations? a. YES, I understand there may be no or limited interactions with the course instructor b. NO, I expect to have at least some interactions with the course instructor c. I do not have expectations either way _______________________________________________________________________________________________________ Post-course survey (Administered by instructor at the end of course) Please rate each of the following: Course content 1. Factual accuracy 2. Depth of content 3. Real-world applicability

poor poor poor

fair fair fair

good good good

very good very good very good

excellent excellent excellent

Course materials 4. Lecture videos 5. In-video quizzes 6. Assessments 7. Discussion

poor poor poor poor

fair fair fair fair

good good good good

very good very good very good very good

excellent excellent excellent excellent

Instructor/teaching staff 8. Clarity 9. Knowledge 10. Presentation 11. Responsiveness

poor poor poor poor

fair fair fair fair

good good good good

very good very good very good very good

excellent excellent excellent excellent

Overall Experience 1. This MOOC met my expectations. a. strongly disagree b. somewhat disagree d. somewhat agree e. strongly agree

c. neutral

2. The promotional materials for this MOOC were accurate a. Yes b. No

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

201

12. Overall course experience poor

fair

good

very good

excellent

13. I feel like I achieved my personal goals for this course. a. strongly disagree b. somewhat disagree c. neutral d. somewhat agree e. strongly agree If you did not achieve your goals for this course, which of the following factors would have helped? 14. More interesting content a. strongly disagree b. somewhat disagree d. somewhat agree e. strongly agree

c. neutral f. N/A

15. More practically relevant content a. strongly disagree b. somewhat disagree d. somewhat agree e. strongly agree

c. neutral f. N/A

16. Fewer technical issues a. strongly disagree d. somewhat agree

b. somewhat disagree e. strongly agree

c. neutral f. N/A

17. Reduced scope and length of course a. strongly disagree b. somewhat disagree d. somewhat agree e. strongly agree

c. neutral f. N/A

18. Content spread over longer time period a. strongly disagree b. somewhat disagree d. somewhat agree e. strongly agree

c. neutral f. N/A

Please rate each of the following: 19. Course workload a. too light d. somewhat heavy

b. somewhat light e. too heavy

c. just right

20. Course difficulty a. too easy d. somewhat difficult

b. somewhat easy e. too difficult

c. just right

21. Course pacing a. too slow d. somewhat fast

b. somewhat slow e. too fast

c. just right

To what extent do you agree or disagree with each of the following statements? 22. I found the material interesting a. strongly disagree b. somewhat disagree d. somewhat agree e. strongly agree

c. neutral f. N/A

23. The course ran smoothly, with few technical difficulties a. strongly disagree b. somewhat disagree c. neutral d. somewhat agree e. strongly agree f. N/A

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

202

24. Students who complete this course should be awarded college credit. a. strongly disagree b. somewhat disagree c. neutral d. somewhat agree e. strongly agree f. N/A 25. How many hours per week did you spend on this course? a. 0-5 hours b. 5-10 hours c. 10-15 hours d. 15-20 hours e. 20+ hours How likely would you be to: 26. Revisit the course materials in the future? a. very unlikely b. somewhat unlikely d. somewhat likely e. very likely

c. neutral

27. Take this course again? a. very unlikely d. somewhat likely

c. neutral

b. somewhat unlikely e. very likely

28. Take another course by the same instructors? a. very unlikely b. somewhat unlikely d. somewhat likely e. very likely

c. neutral

29. Recommend this course to a friend? a. very unlikely b. somewhat unlikely d. somewhat likely e. very likely

c. neutral

In a previous online learning experience I valued . . . (Select all that apply) 30. Communications with the other course participants a. strongly disagree b. somewhat disagree d. somewhat agree e. strongly agree

c. neutral f. N/A

31. Communications with the course instructor a. strongly disagree b. somewhat disagree d. somewhat agree e. strongly agree

c. neutral f. N/A

32. Access to the course content a. strongly disagree b. somewhat disagree d. somewhat agree e. strongly agree

c. neutral f. N/A

33. Opportunities for team work a. strongly disagree b. somewhat disagree d. somewhat agree e. strongly agree

c. neutral f. N/A

34. What did you like best about this course?

[open text field]

35. What did you like least about this course?

[open text field]

36. Other suggestions/comments

[open text field]

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

203

American Museum of Natural History Pre-Course Survey [This survey was provided courtesy of American Museum of Natural History] 1.

What is your professional role? a. educator b. student

c. retired

d. other (please specify)

If you are an educator, what is your role? (select all that apply) a. I am a preschool teacher. b. I am an elementary school teacher. c. I am a middle school teacher. d. I am a high school teacher. e. I am a higher education (college/university) instructor. f. I teach in a public school. g. I teach in a private school. h. I am a school administrator. i. I am a museum/informal educator. j. I am an after-school educator. k. I am a summer program educator. l. I am a media specialist/librarian. m. I am a staff development coordinator. n. Does not apply.

If you are a student, what kind of school do you attend? (select all that apply) a. public b. private c. home school e. college f. graduate school d. continuing education

What subject/s do you teach? (select all that apply) a. chemistry b. biology d. physics e. general science g. Special Education h. English Language Learners (ELL) j. does not apply

For how long have you been a teacher? a. pre-service b. 1-2 years c. 3-5 years d. 6-9 years e. 10-19 years f. 20+ years g. Does not apply.

Gender: a. Male

b. Female

c. Other

d. high school f. does not apply.

c. Earth science f. environmental science i. other (please specify)

d. Decline to answer.

With which ethnic group do you identify? a. American Indian or Alaskan Native

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

204

b. c. d. e. f. g. h. i. j.

Asian Black or African American Hispanics of any race Native Hawaiian or Other Pacific Islander White Race and Ethnicity unknown Two or more races Other (Include text box to self-identify.) Decline to answer.

In what country do you live? (Drop-down option)

If you live in the United States, please enter your zip code. (Open-ended text box)

10. What is the highest level of academic study you have completed? a. High School Diploma / U.S. GED b. Associate’s Degree c. Bachelor’s Degree d. Master’s Degree e. Professional or Doctoral Degree (Ph.D., Ed.D., J.D., M.D.) f. Does not apply. 11. On average, how many hours do you plan to spend on this course each week? a. Less than 1 hour b. 1-2 hours c. 3-5 hours d. 6-8 hours e. 8+ hours 12. Do you intend to earn a Statement of Accomplishment or Verified Certificate for this course? a. yes b. no c. I don’t know 13. Please select the level of importance that the following factors had in your decision to enroll in this course. (Options: Not important, Somewhat important, Important, Very important, N/A) a. My district is providing professional development credit for this course. b. To deepen my current knowledge of the subject area. c. To learn skills that will help with my career. d. To gain access to educational materials. e. To connect with other teachers, become part of an online community, or expand network. 14. How did you learn about this course? (Select all that apply) a. AMNH website b. Coursera website c. search engine d. district or school e. friend or colleague f. online social network

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

205

g. h.

blog/news articles/press coverage other (please specify)

15. What is your prior experience with this subject? (Select all that apply) a. previous coursework/degree b. I teach this subject. c. science enthusiast d. personal experience e. I work in a related field. f. No prior academic, personal, or professional experience in this subject area. 16. What is your experience with the American Museum of Natural History? (select all that apply) a. I have used resources from the Museumâ&#x20AC;&#x2122;s website b. I have participated in an AMNH online course. c. I have participated in face-to-face teacher professional development at AMNH. d. I have visited the Museum. e. I have no prior experience with the Museum. _____________________________________________________________________________________

American Museum of Natural History Post-Course Survey 1.

On average, how many hours did you spend on this course each week? a. less than 1 hour b. 1-2 hours c. 3-5 hours d. 6-8 hours e. 8+ hours

How appropriate was the amount of work in this course? a. too much work b. about right

c. not enough work

This course helped me: (Likert scale: Strongly Disagree, Disagree, No Opinion, Agree, Strongly Agree) a. learn important concepts b. learn to analyze and critically evaluate ideas, arguments, and points of view about the topics covered c. increase my capacity to teach science d. deepen my understanding of the connection between science and society e. gain a broader understanding of the process of scientific discovery

Which of the following course materials contributed to your learning? (Likert scale: Strongly Disagree, Disagree, No Opinion, Agree, Strongly Agree) a. essays b. videos c. discussion forums d. quizzes e. assignment

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

206

I plan to use or adapt the following course materials for my own teaching: a. essays b. videos c. assignment d. classroom resources

Please rate how much you agree or disagree with each of the following statements: (Likert scale: Strongly Disagree, Disagree, No Opinion, Agree, Strongly Agree) a. Participating in discussions was helpful in understanding weekly content. b. Participating in discussions increased my engagement with the course. c. I felt comfortable sharing ideas and viewpoints in the online discussions.

Did you receive professional development credit from your district for taking this course? a. yes b. no c. I donâ&#x20AC;&#x2122;t know d. does not apply

How accessible was the level of the content given your background in science? a. much too difficult b. too difficult c. about right d. too easy e. much too easy

Technical issues were addressed by Coursera staff in a timely manner. a. yes b. no c. does not apply

10. Would you recommend this course to a friend or colleague? a. yes b. no c. maybe 11. Would you take another AMNH course offered through Coursera? a. yes b. no c. maybe 12. Would you take an AMNH online science course that is eligible for graduate credit? (For more information, please visit our Seminars on Science site.) a. yes b. no c. maybe 13. Would you be willing to speak to an AMNH representative about your course experience? a. yes b. no 14. Are you interested in face-to-face professional development offered by AMNH? a. yes b. no c. maybe 15. Would you like to be added to the AMNH Education Newsletter list? a. yes b. no c. maybe 16. What is your overall rating of this course? a. excellent b. above average

c. average

d. below average

e. poor

17. Please tell us what you feel were the best parts of this course. (open-ended) 18. Please tell us how the course might be improved. (open-ended)

Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

207

Appendix VI: List of Interviewees (in alphabetical order by last name) • • • • • • • • • • • • • • • • • • • • • • • • • • • •

Lawrence S. Bacow, President Emeritus, Tufts University, MA Judy Baker, Ph.D., Dean, Foothill College, CA Ryan Baker, Associate Professor, Teachers College, Columbia University, NY Dr. Tony Bates, President/CEO, Tony Bates Associates Ltd., Vancouver, BC, Canada David BenDaniel, Berens Professor of Entrepreneurship, Johnson School, Cornell University MJ Bishop, Director, Center for Innovation and Excellence in Learning and Teaching, University System of Maryland Karen Cator, President and CEO, Digital Promise Michael Cennamo, Educational Technologist, Columbia Center for New Media Teaching and Learning, Columbia University Catheryn Cheal, Ph.D., Associate Vice President and Senior Academic Technology Officer, San José State University, CA Professor Michael J. Cima, MIT Amy Collier, Director of Digital Learning Initiatives, Stanford University Sir John Daniel Chris Dede, Wirth Professor in Learning Technology, Harvard University Ted Dodds, Chief Information Officer and Vice President for Information Technologies, Cornell University Stephen Downes, National Research Council Dr. Justyna Dymerska, Executive Vice President, Academic Partnerships Dr. Li Fengliang, Associate Professor, Institute of Education, Tsinghua University, China Douglas H. Fisher, Director of the Vanderbilt Institute for Digital Learning, Vanderbilt University Armando Fox, Professor, Electrical Engineering and Computer Science Department, Faculty Advisor to the MOOCLab, UC Berkeley, CA Rebecca Griffiths, Program Director for Online Learning, Ithaka S+R Kevin Guthrie, President, ITHAKA Nick Hathaway, Vice President for Administration and Finance, University of Oklahoma Kyle Harper, Senior Vice Provost, University of Oklahoma Jessica Heppen, Managing Researcher, American Institutes for Research Dr. Douglas Hersh, formerly Dean, Educational Programs, Santa Barbara City College, CA. Currently Dean, Learning and Technology Resources, Grossmont College, CA Shanna Jaggars, Assistant Director, Community College Research Center, Teachers College, Columbia University Patricia James, Dean of Distance Education, Mt. San Jacinto Community College, CA Laura Kalbaugh, Dean, Academic Success and Transition Resources, Wake Technical Community College, Raleigh, NC Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

208

• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •

Deborah Keyek-Franssen, (Ph.D.), Associate Vice President for Digital Education and Engagement, University of Colorado System Dr. Ro Kinzler, Senior Director, Science Education, American Museum of Natural History, NY Ann Kirschner, University Dean, Macaulay Honors College at the City University of New York Dr. Glenn Kleiman, Executive Director, Friday Institute for Educational Innovation, North Carolina State University. Ken Koedinger, Professor of Human-Computer Interaction and Psychology, Carnegie Mellon University Pang Wei Koh, Director of Course Operations, Coursera John E. Kolb, VP of Information Technology and Services and CIO, Rensselaer Polytechnic Institute Stephen Laster, Chief Digital Officer, McGraw-Hill Education Mark Lester, Head of UK Partnerships, FutureLearn Minghua Li, Professor, School of Public Administration, East China Normal University Melissa Loble, formerly Associate Dean for Distance Learning, University of California, Irvine Extension; currently Senior Director, Canvas Network Howard Lurie, formerly Vice President for External Affairs, edX; currently Managing Director, Education Partnerships, Acrobatiq - A Carnegie Mellon University Venture Matthew Martin, Professor of Communication Studies, West Virginia University Barbara Means, Center for Technology in Learning, SRI International Belinda Miles, Provost and Executive Vice President of Access, Learning, & Success, Cuyahoga Community College, OH Steven Mintz, Executive Director, Institute for Transformational Learning, University of Texas System Christopher Newfield, Professor of Literature and American Studies, University of California, Santa Barbara David Niemi, VP, Measurement and Research, Kaplan Inc. Ken Parker, CEO, NextThought Lawrence Paska, Director of Social Studies, Harrison Central School District, Harrison, NY; former State Education Agency Technology Policy Coordinator Dr. Paul S. Peercy, Dean Emeritus, College of Engineering, University of Wisconsin-Madison Rebecca Petersen, Director, Research and Educational Initiatives, edX Dave Pritchard, Cecil and Ida Greene Professor of Physics, MIT Vincent Racaniello, Professor, Columbia University Lawrence C. Ragan, Director, The Center for Online Innovation in Learning, Penn State World Campus Timothy Renick, Vice Provost and Chief Enrollment Officer, Georgia State University Jake Reynolds, General Partner, Technology Crossover Ventures Saad Rizvi, Senior Vice President, Efficacy, Pearson Christina Royal, Ph.D., Chief Academic Officer/Provost, Inver Hills Community College Cathy Sandeen, Vice President, Education Attainment and Innovation, American Council on Education Professor Sanjay Sarma, Director of Digital Learning, MIT Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

209

• • • • • • • • • • • • • • • • • • • • • • • •

Bror Saxberg, Chief Learning Officer, Kaplan Inc. Emily Schneider, Student Director, Lytics Lab, Stanford University Paul Sechrist, President, Oklahoma City Community College George Siemens, formerly Professor, Centre for Distance Education, Athabasca University, Canada. Currently Director of the Learning Innovation and Networked Knowledge Research Lab, University of Texas at Austin. Stan Silverman, Professor and Director, Technology Based Learning, New York Institute of Technology Beth Smith, President, Academic Senate for California Community Colleges Burck Smith, CEO and Founder, Straighterline Sree Sreenivasan, formerly Chief Digital Officer and Professor, Columbia University. Currently Chief Digital Officer at the Metropolitan Museum of Art, NY Dr. Robert Steiner, Director, Online Teacher Education Programs, American Museum of Natural History Sasha Thackaberry, District Director, eLearning Technologies, Cuyahoga Community College, OH Candace Thille, Assistant Professor, Director of Open Learning Initiative, Senior Research Fellow, Office of the Vice Provost for Online Learning, Stanford University Joseph Ugoretz, Associate Dean of Teaching, Learning and Technology, Macaulay Honors College (CUNY) Karen Vignare, Associate Provost, Center for Innovation in Learning, University of Maryland University College (UMUC) Victor Vuchic, formerly Program Officer Education, William and Flora Hewlett Foundation. Currently founder of V3 Consulting. Yuan "Elle" Wang, MOOC Researcher, Teachers College, Columbia University David White, Assistant Dean for Academic Programs and Executive Director of the Online M.S. in Computer Science, College of Computing, Georgia Tech Dr. John Whitmer, Program Manager, Academic Technology & Analytics Office of the Chancellor, California State University Joseph Jay Williams, Research Fellow, Lytics Lab, Graduate School of Education, Office of the Vice Provost of Online Learning, Stanford University Anonymous Development Officer, Highly Selective University, U.S. Online Learning Coordinator, Large California Community College Professor, East Coast University, U.S. Senior Administrator, Leading UK University Senior Administrator, East Coast University, U.S.



Hollands & Tirthali: MOOCs: Expectations and Reality May 2014

210

Communications to the authors may be directed to: Fiona Hollands at fmh7@tc.columbia.edu or Devayani Tirthali at dgt2103@columbia.edu.

Front page and back page image adapted from: 3D Social Networking licensed under CC BY 2.0